kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1651] kt-kernel 0.4.2 Kimi-K2 Thinking: torch.OutOfMemoryError: CUDA out of memory. #1063

New issue
Open
opened 2026-10-29 14:48:07 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 14:48:07 +00:00
Owner
Copy link

Originally created by @mrgaolei on GitHub (Nov 28, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1651

Originally assigned to: @chenht2022, @ovowei on GitHub.

Reminder

  • I have read the above rules and searched the existing issues.

System Info

Ubuntu 24.04.3
kt-kernel 0.4.2
Kimi-K2 Thinking

Dual-6416h CPU
1T RAM
RTX 3090 24G

Reproduction

python -m sglang.launch_server   --host 0.0.0.0   --port 60000   --model /opt/ai-models/Kimi-K2-Thinking/   --kt-weight-path /opt/ai-models/Kimi-K2-Instruct-CPU-weight/   --kt-cpuinfer 36   --kt-threadpool-count 2   --kt-num-gpu-experts 16   --kt-method AMXINT4   --attention-backend flashinfer   --trust-remote-code   --mem-fraction-static 0.98   --chunked-prefill-size 4096   --max-running-requests 37
   --max-total-tokens 3700   --enable-mixed-chunk   --tensor-parallel-size 1   --enable-p2p-check   --disable-shared-experts-fusion```

WorkerPool[0x2a5a6f30] 2 subpools, [numa:threads][0:18] [1:18]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 18 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 18 threads
[2025-11-28 11:19:49] Scheduler hit an exception: Traceback (most recent call last):
File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 2630, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 309, in init
self.tp_worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in init
self._model_runner = ModelRunner(
^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 338, in init
self.initialize(min_per_gpu_memory)
File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 442, in initialize
self.load_model()
File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 806, in load_model
self.model = get_model(
^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/model_loader/init.py", line 28, in get_model
return loader.load_model(
^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 594, in load_model
model = _initialize_model(
^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 262, in _initialize_model
return model_class(**kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3394, in init
self.model = DeepseekV2Model(
^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3172, in init
self.model = DeepseekV2Model(
File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 591, in make_layers
+ get_offloader().wrap_modules(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules
return list(all_modules_generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 593, in
layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3174, in
lambda idx, prefix: DeepseekV2DecoderLayer(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2895, in init
self.self_attn = DeepseekV2AttentionMLA(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1305, in init
self.o_proj = RowParallelLinear(
^^^^^^^^^^^^^^^^^^
File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 1255, in init
self.quant_method.create_weights(
File "/home/aigao/sglang/python/sglang/srt/layers/quantization/unquant.py", line 108, in create_weights
torch.empty(
File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in torch_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 109.50 MiB is free. Processtorch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 109.50 MiB is free. Process 4308 has 360.00 MiB memory in use. Including non-PyTorch memory, this process has 23.07 GiB memory in use. Of the allocated memory 22.72 GiB is alloc 4308 has 360.00 MiB memory in use. Including non-PyTorch memory, this process has 23.07 GiB memory in use. Of the allocated memory 22.72 GiB is alloca


### Others

Can kt-kernel support 24G DRAM? like kt 0.3.2?
If not, is mean only 1 n090 card can only use kt 0.3.2?
Originally created by @mrgaolei on GitHub (Nov 28, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1651 Originally assigned to: @chenht2022, @ovowei on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Ubuntu 24.04.3 kt-kernel 0.4.2 Kimi-K2 Thinking Dual-6416h CPU 1T RAM RTX 3090 24G ### Reproduction ``` python -m sglang.launch_server --host 0.0.0.0 --port 60000 --model /opt/ai-models/Kimi-K2-Thinking/ --kt-weight-path /opt/ai-models/Kimi-K2-Instruct-CPU-weight/ --kt-cpuinfer 36 --kt-threadpool-count 2 --kt-num-gpu-experts 16 --kt-method AMXINT4 --attention-backend flashinfer --trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 4096 --max-running-requests 37 --max-total-tokens 3700 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion``` ``` WorkerPool[0x2a5a6f30] 2 subpools, [numa:threads][0:18] [1:18] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 18 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 18 threads [2025-11-28 11:19:49] Scheduler hit an exception: Traceback (most recent call last): File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 2630, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 309, in __init__ self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 338, in __init__ self.initialize(min_per_gpu_memory) File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 442, in initialize self.load_model() File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 806, in load_model self.model = get_model( ^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/__init__.py", line 28, in get_model return loader.load_model( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 594, in load_model model = _initialize_model( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 262, in _initialize_model return model_class(**kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3394, in __init__ self.model = DeepseekV2Model( ^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3172, in __init__ self.model = DeepseekV2Model( File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 591, in make_layers + get_offloader().wrap_modules( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules return list(all_modules_generator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 593, in <genexpr> layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3174, in <lambda> lambda idx, prefix: DeepseekV2DecoderLayer( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2895, in __init__ self.self_attn = DeepseekV2AttentionMLA( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1305, in __init__ self.o_proj = RowParallelLinear( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 1255, in __init__ self.quant_method.create_weights( File "/home/aigao/sglang/python/sglang/srt/layers/quantization/unquant.py", line 108, in create_weights torch.empty( File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in __torch_function__ return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 109.50 MiB is free. Processtorch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 109.50 MiB is free. Process 4308 has 360.00 MiB memory in use. Including non-PyTorch memory, this process has 23.07 GiB memory in use. Of the allocated memory 22.72 GiB is alloc 4308 has 360.00 MiB memory in use. Including non-PyTorch memory, this process has 23.07 GiB memory in use. Of the allocated memory 22.72 GiB is alloca ``` ### Others Can kt-kernel support 24G DRAM? like kt 0.3.2? If not, is mean only 1 n090 card can only use kt 0.3.2?
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@chenht2022 commented on GitHub (Nov 28, 2025):

The issue is that you used --kt-num-gpu-experts 16, which specifies that each layer has 16 experts on the GPU. The 24GB VRAM can’t handle that, so try lowering this parameter.

<!-- gh-comment-id:3588130691 --> @chenht2022 commented on GitHub (Nov 28, 2025): The issue is that you used `--kt-num-gpu-experts 16`, which specifies that each layer has 16 experts on the GPU. The 24GB VRAM can’t handle that, so try lowering this parameter.
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@mrgaolei commented on GitHub (Nov 29, 2025):

The issue is that you used --kt-num-gpu-experts 16, which specifies that each layer has 16 experts on the GPU. The 24GB VRAM can’t handle that, so try lowering this parameter.

I tried:

--kt-num-gpu-experts [1-4]

this crash also happen.
Who tested 24G VRAM and 1T RAM how to start Kimi-K2 via kt 0.4.2?

<!-- gh-comment-id:3591011428 --> @mrgaolei commented on GitHub (Nov 29, 2025): > The issue is that you used `--kt-num-gpu-experts 16`, which specifies that each layer has 16 experts on the GPU. The 24GB VRAM can’t handle that, so try lowering this parameter. I tried: ``` --kt-num-gpu-experts [1-4] ``` this crash also happen. Who tested 24G VRAM and 1T RAM how to start Kimi-K2 via kt 0.4.2?
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@chenht2022 commented on GitHub (Dec 1, 2025):

I see. The native Kimi-K2-Thinking model uses BF16-precision (non-expert) weights on the GPU side, so it consumes more VRAM than DeepSeek-V3/R1. A 24 GB GPU isn’t sufficient. You may consider using a quantized version.

<!-- gh-comment-id:3596137993 --> @chenht2022 commented on GitHub (Dec 1, 2025): I see. The native Kimi-K2-Thinking model uses BF16-precision (non-expert) weights on the GPU side, so it consumes more VRAM than DeepSeek-V3/R1. A 24 GB GPU isn’t sufficient. You may consider using a quantized version.
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@shrould8888 commented on GitHub (Dec 1, 2025):

you need at least 27-28GB VRAM even --kt-num-gpu-experts 0

<!-- gh-comment-id:3598996109 --> @shrould8888 commented on GitHub (Dec 1, 2025): you need at least 27-28GB VRAM even --kt-num-gpu-experts 0
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@mrgaolei commented on GitHub (Dec 2, 2025):

I see. The native Kimi-K2-Thinking model uses BF16-precision (non-expert) weights on the GPU side, so it consumes more VRAM than DeepSeek-V3/R1. A 24 GB GPU isn’t sufficient. You may consider using a quantized version.

Does kt 0.4.2 support Kimi-K2 Thinking Q4 model?

<!-- gh-comment-id:3600014485 --> @mrgaolei commented on GitHub (Dec 2, 2025): > I see. The native Kimi-K2-Thinking model uses BF16-precision (non-expert) weights on the GPU side, so it consumes more VRAM than DeepSeek-V3/R1. A 24 GB GPU isn’t sufficient. You may consider using a quantized version. Does kt 0.4.2 support Kimi-K2 Thinking Q4 model?
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@chenht2022 commented on GitHub (Dec 2, 2025):

We provide the scripts to perform quantization. But by default, we quantize expert weights but not non-expert weights. However, you currently only need to quantize the non-expert weights. I guess you may customize which weights get quantized by hacking the ignore_patterns in convert_gpu_weights.py.

Is it right? @ovowei

<!-- gh-comment-id:3600104844 --> @chenht2022 commented on GitHub (Dec 2, 2025): We provide the [scripts](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts) to perform quantization. But by default, we quantize expert weights but not non-expert weights. However, you currently only need to quantize the non-expert weights. I guess you may customize which weights get quantized by hacking the `ignore_patterns` in `convert_gpu_weights.py`. Is it right? @ovowei
bsakaguchi commented 2026-10-29 14:48:08 +00:00
Author
Owner
Copy link

@shrould8888 commented on GitHub (Dec 2, 2025):

you just need one more 24GB 3090.

I run this model with 4090 x 2 without any problems

I recommend that you should have at least 48GB VRAM in order to have decent context length

<!-- gh-comment-id:3600335884 --> @shrould8888 commented on GitHub (Dec 2, 2025): you just need one more 24GB 3090. I run this model with 4090 x 2 without any problems I recommend that you should have at least 48GB VRAM in order to have decent context length
bsakaguchi commented 2026-10-29 14:48:09 +00:00
Author
Owner
Copy link

@mrgaolei commented on GitHub (Dec 5, 2025):

you just need one more 24GB 3090.

I run this model with 4090 x 2 without any problems

I recommend that you should have at least 48GB VRAM in order to have decent context length

Can I run this with 3090 + 4060Ti 16G?

<!-- gh-comment-id:3615018733 --> @mrgaolei commented on GitHub (Dec 5, 2025): > you just need one more 24GB 3090. > > I run this model with 4090 x 2 without any problems > > I recommend that you should have at least 48GB VRAM in order to have decent context length Can I run this with 3090 + 4060Ti 16G?
bsakaguchi commented 2026-10-29 14:48:09 +00:00
Author
Owner
Copy link

@ovowei commented on GitHub (Dec 5, 2025):

I haven’t tried running on mixed GPUs like a 3090 and a 4060Ti 16G. It might cause some issues, but you can give it a try and see if it works.

<!-- gh-comment-id:3615155426 --> @ovowei commented on GitHub (Dec 5, 2025): I haven’t tried running on mixed GPUs like a 3090 and a 4060Ti 16G. It might cause some issues, but you can give it a try and see if it works.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1063
No description provided.