[GH-ISSUE #960] [Bug] DeepSeek R1 Q4_K_M failed on dual 3080ti with version v0.2.3 due to CUDA OOM

bsakaguchi commented

2026-10-29 14:42:49 +00:00

Owner

Originally created by @windr00 on GitHub (Mar 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/960

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

cpu: single AMD EPYC 7b13
ram: 512G DDR4 3200MHz
gpu: 2x 3080ti 12G

version: 0.2.3 built from source
command line: ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002

customization to source code: no, also no change to optimzie rules

issue: cuda oom

I happened to run it successfully on an older version, maybe v0.2.0 with short context. But failed on v0.2.3

by the way, I got 9.2 tps in decode with the same hardware with RTX 4090(non D variant)

cpu: 单路AMD EPYC 7b13
ram: 512G DDR4 3200MHz
gpu: 2x 3080ti 12G

版本: 0.2.3 源码编译
命令行: ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002

修改了源码吗: 没有, 并且没有修改optimize_rule

问题: cuda oom

我使用老版本（可能是v0.2.0，记不清了）成功运行过，但是现在用v0.2.3就失败了

换用4090（非D），在同样的硬件上可以在decode阶段得到9.2tps

Reproduction

ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002

Environment

OS: Ubuntu 22.04.5
CPU: AMD Epyc 7B13
GPU: 2x RTX 3080ti 12G

Originally created by @windr00 on GitHub (Mar 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/960 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug **cpu**: single AMD EPYC 7b13 **ram**: 512G DDR4 3200MHz **gpu**: 2x 3080ti 12G **version**: 0.2.3 built from source **command line**: ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002 **customization to source code**: no, also no change to optimzie rules **issue**: cuda oom I happened to run it successfully on an older version, maybe v0.2.0 with short context. But failed on v0.2.3 by the way, I got 9.2 tps in decode with the same hardware with RTX 4090(non D variant) **cpu**: 单路AMD EPYC 7b13 **ram**: 512G DDR4 3200MHz **gpu**: 2x 3080ti 12G **版本**: 0.2.3 源码编译 **命令行**: ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002 **修改了源码吗**: 没有, 并且没有修改optimize_rule **问题**: cuda oom 我使用老版本（可能是v0.2.0，记不清了）成功运行过，但是现在用v0.2.3就失败了换用4090（非D），在同样的硬件上可以在decode阶段得到9.2tps ### Reproduction ktransformers --model_path ./DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --cpu_infer 60 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 100 --port 10002 ### Environment OS: Ubuntu 22.04.5 CPU: AMD Epyc 7B13 GPU: 2x RTX 3080ti 12G

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@windr00 commented on GitHub (Mar 23, 2025):

When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to nvtop, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs?

当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？

@windr00 commented on GitHub (Mar 23, 2025): When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to **nvtop**, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs? 当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@keenfox commented on GitHub (Mar 24, 2025):

When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to nvtop, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs?

当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？

Yes, you guessed it right~
没错，你猜的很对~

@keenfox commented on GitHub (Mar 24, 2025): > When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to **nvtop**, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs? > > 当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？ Yes, you guessed it right~ 没错，你猜的很对~

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@Ksdb104 commented on GitHub (Mar 26, 2025):

那么问题来了，这个要怎么解决呢。。。我现在也打算搞两张12G显存的显卡。。。

@Ksdb104 commented on GitHub (Mar 26, 2025): 那么问题来了，这个要怎么解决呢。。。我现在也打算搞两张12G显存的显卡。。。

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@windr00 commented on GitHub (Mar 27, 2025):

目前还没找到解决方法

那么问题来了，这个要怎么解决呢。。。我现在也打算搞两张12G显存的显卡。。。

@windr00 commented on GitHub (Mar 27, 2025): 目前还没找到解决方法 > 那么问题来了，这个要怎么解决呢。。。我现在也打算搞两张12G显存的显卡。。。

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@windr00 commented on GitHub (Mar 27, 2025):

When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to nvtop, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs?
当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？

Yes, you guessed it right~ 没错，你猜的很对~

所以这是正常的吗？

@windr00 commented on GitHub (Mar 27, 2025): > > When I run with RTX 4090, I noticed a vram spike when loading the weights in the begining, consuming about 75% of VRAM according to **nvtop**, then it drops down to 6GB when loading the first few layers. Is this vram spike causing the CUDA OOM on dual 12GB GPUs? > > 当我用4090运行时，我注意到模型刚开始加载时，使用nvtop发现显存会激增至75%左右，然后当开始加载每一层时，使用量又降低至6GB，直至占用11G左右。是不是这个开始时的占用导致双12G显卡报OOM？ > > Yes, you guessed it right~ 没错，你猜的很对~ 所以这是正常的吗？

bsakaguchi commented

2026-10-29 14:42:50 +00:00

Author

Owner

@Leechy-Litchi commented on GitHub (Apr 8, 2025):

同样的问题，使用了0.2.3和0.2.4但都出现，加载DeepSeek V3/R1的模型都是一样的报错，Qwen2-57B-Q4_K_M能够正常使用，显卡是双卡RTX A3000 laptop 12G

@Leechy-Litchi commented on GitHub (Apr 8, 2025): 同样的问题，使用了0.2.3和0.2.4但都出现，加载DeepSeek V3/R1的模型都是一样的报错，Qwen2-57B-Q4_K_M能够正常使用，显卡是双卡RTX A3000 laptop 12G

bsakaguchi commented

2026-10-29 14:42:51 +00:00

Author

Owner

@Ksdb104 commented on GitHub (Apr 10, 2025):

同样的问题，使用了0.2.3和0.2.4但都出现，加载DeepSeek V3/R1的模型都是一样的报错，Qwen2-57B-Q4_K_M能够正常使用，显卡是双卡RTX A3000 laptop 12G

以我的经验看，单卡最好＞=14G，这块应该也不会优化了

@Ksdb104 commented on GitHub (Apr 10, 2025): > 同样的问题，使用了0.2.3和0.2.4但都出现，加载DeepSeek V3/R1的模型都是一样的报错，Qwen2-57B-Q4_K_M能够正常使用，显卡是双卡RTX A3000 laptop 12G 以我的经验看，单卡最好＞=14G，这块应该也不会优化了

Rows
Columns

[GH-ISSUE #960] [Bug] DeepSeek R1 Q4_K_M failed on dual 3080ti with version v0.2.3 due to CUDA OOM #683

Checklist

Describe the bug

Reproduction

Environment