[GH-ISSUE #764] Dual 3080 Memory Error: OOM #552

New issue

Closed

opened 2026-10-29 14:41:07 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:41:07 +00:00

Owner

Originally created by @cestney on GitHub (Mar 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/764

Dual 3080 Memory Error: OOM

Specific Configuration:
epyc 7642x1 256G-RAM 3080-10G x2 cuda-12.6 python3.11 (ktransformers build from source. [0.22rc1 0.21])
Model: Q2_K_XS

Command Line:
python -m ktransformers.local_chat --gguf_path /home/master/workspace/model --model_path /home/master/workspace/model/token --cpu_infer 12 --max_new_tokens 1200 --optimize_rule_path /home/master/workspace/kt/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml
Tried: DeepSeek-V3-Chat-multi.yaml didn't work either.

Installed two versions, 0.21 and 0.22rc, both encountered OOM when loading the model into the GPU.
In theory, the combined 10G of the two cards should exceed 20G, which should be sufficient for deployment. So, what could be the issue? Moreover, the 3080 supports the marlin operator.

The specific error messages vary slightly between versions (0.21 and 0.22rc, and different optimize parameters), but they all occur after the following code finishes loading:
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
...

0.22rc1-multi-gpu-marlin:

0.21-multi-gpu-marlin:

0.21-multi-gpu:

Originally created by @cestney on GitHub (Mar 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/764 Dual 3080 Memory Error: OOM Specific Configuration: epyc 7642x1 256G-RAM 3080-10G x2 cuda-12.6 python3.11 (ktransformers build from source. [0.22rc1 0.21]) Model: Q2_K_XS Command Line: python -m ktransformers.local_chat --gguf_path /home/master/workspace/model --model_path /home/master/workspace/model/token --cpu_infer 12 --max_new_tokens 1200 --optimize_rule_path /home/master/workspace/kt/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml Tried: DeepSeek-V3-Chat-multi.yaml didn't work either. Installed two versions, 0.21 and 0.22rc, both encountered OOM when loading the model into the GPU. In theory, the combined 10G of the two cards should exceed 20G, which should be sufficient for deployment. So, what could be the issue? Moreover, the 3080 supports the marlin operator. The specific error messages vary slightly between versions (0.21 and 0.22rc, and different optimize parameters), but they all occur after the following code finishes loading: Injecting model.layers.60.mlp.shared_experts.act_fn as default Injecting model.layers.60.input_layernorm as default Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as default ... 0.22rc1-multi-gpu-marlin: ![Image](https://github.com/user-attachments/assets/a6674396-db15-485c-aeea-299480c33612) 0.21-multi-gpu-marlin: ![Image](https://github.com/user-attachments/assets/7ff83f84-a327-46f6-b903-ec07c1da1cd5) 0.21-multi-gpu: ![Image](https://github.com/user-attachments/assets/659e259e-21d9-4446-9509-16de57ca1023)

bsakaguchi

2026-10-29 14:41:07 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:41:07 +00:00

Author

Owner

@ovowei commented on GitHub (Mar 4, 2025):

Try replacing --optimize_rule_path with --optimize_config_path

@ovowei commented on GitHub (Mar 4, 2025): Try replacing `--optimize_rule_path` with `--optimize_config_path`

bsakaguchi commented

2026-10-29 14:41:08 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！