[GH-ISSUE #1681] How to use 2 16G GPUs to run kt-kernel 0.4.x #1078

New issue

Open

opened 2026-10-29 14:48:17 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:48:17 +00:00

Owner

Originally created by @mrgaolei on GitHub (Dec 8, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1681

Originally assigned to: @SkqLiao on GitHub.

Reminder

I have read the above rules and searched the existing issues.

System Info

Intel 6416H * 1 CPU
768G RAM
4060Ti 16G *2 GPU

Reproduction

  File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 311, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in __init__
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize
    self.load_model()
  File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/model_loader/__init__.py", line 28, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 594, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 262, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3034, in __init__
    self.model = DeepseekV2Model(
                 ^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2830, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 576, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 578, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2832, in <lambda>
    lambda idx, prefix: DeepseekV2DecoderLayer(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2588, in __init__
    self.self_attn = DeepseekV2AttentionMLA(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1203, in __init__
    self.o_proj = RowParallelLinear(
                  ^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 1255, in __init__
    self.quant_method.create_weights(
  File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8.py", line 293, in create_weights
    data=torch.empty(
         ^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 15.57 GiB of which 4.69 MiB is free. Including non-PyTorch memory, this process has 15.55 GiB memory in use. Of the allocated memory 15.19 GiB is allocated by PyTorch, and 208.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[2025-12-08 16:43:01] Received sigquit from a child process. It usually means the child failed.
./ktdsv32.sh: 第 20 行： 626563 已杀死               python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model /opt/ai-models/v3-2 --kt-weight-path /opt/ai-models/v3-2-CPU --kt-cpuinfer 36 --kt-threadpool-count 1 --kt-num-gpu-experts 0 --attention-backend triton -
-trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 2048 --max-running-requests 16 --max-total-tokens 20000 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion --kt-method AMXINT4```

Others

I know that kt-kernel 0.4.x need more VDRAM to run new models, for example: DeepSeek V3.2, Kimi Thinking.
But is there a way to use cheap GPU to run it?

I have 2 4060Ti 16G, total 32G, can it run DeepSeek V3.2?
Because in document, DeepSeek V3.2 with kt-kernel 0.4.3 only need 27G VDRAM.

Therer is my startup script:

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model /opt/ai-models/v3-2 \
  --kt-weight-path /opt/ai-models/v3-2-CPU \
  --kt-cpuinfer 36 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 0 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 2048 \
  --max-running-requests 16 \
  --max-total-tokens 20000 \
  --served-model-name DeepSeek-V3.2 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --kt-method AMXINT4

I tried adjust tensor-parallel-size to 1 or 2, and kt-num-gpu-experts to 0 or 1.

Originally created by @mrgaolei on GitHub (Dec 8, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1681 Originally assigned to: @SkqLiao on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Intel 6416H * 1 CPU 768G RAM 4060Ti 16G *2 GPU ### Reproduction ``` File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/managers/scheduler.py", line 311, in __init__ self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in __init__ self.initialize(min_per_gpu_memory) File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize self.load_model() File "/home/aigao/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model self.model = get_model( ^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/__init__.py", line 28, in get_model return loader.load_model( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 594, in load_model model = _initialize_model( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/model_loader/loader.py", line 262, in _initialize_model return model_class(**kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 3034, in __init__ self.model = DeepseekV2Model( ^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2830, in __init__ self.layers, self.start_layer, self.end_layer = make_layers( ^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 576, in make_layers + get_offloader().wrap_modules( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules return list(all_modules_generator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/utils/common.py", line 578, in <genexpr> layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2832, in <lambda> lambda idx, prefix: DeepseekV2DecoderLayer( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2588, in __init__ self.self_attn = DeepseekV2AttentionMLA( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1203, in __init__ self.o_proj = RowParallelLinear( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 1255, in __init__ self.quant_method.create_weights( File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8.py", line 293, in create_weights data=torch.empty( ^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in __torch_function__ return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 15.57 GiB of which 4.69 MiB is free. Including non-PyTorch memory, this process has 15.55 GiB memory in use. Of the allocated memory 15.19 GiB is allocated by PyTorch, and 208.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) [2025-12-08 16:43:01] Received sigquit from a child process. It usually means the child failed. ./ktdsv32.sh: 第 20 行： 626563 已杀死 python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model /opt/ai-models/v3-2 --kt-weight-path /opt/ai-models/v3-2-CPU --kt-cpuinfer 36 --kt-threadpool-count 1 --kt-num-gpu-experts 0 --attention-backend triton - -trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 2048 --max-running-requests 16 --max-total-tokens 20000 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion --kt-method AMXINT4``` ``` ### Others I know that kt-kernel 0.4.x need more VDRAM to run new models, for example: DeepSeek V3.2, Kimi Thinking. But is there a way to use cheap GPU to run it? I have 2 4060Ti 16G, total 32G, can it run DeepSeek V3.2? Because in document, DeepSeek V3.2 with kt-kernel 0.4.3 only need 27G VDRAM. Therer is my startup script: ```bash python -m sglang.launch_server \ --host 0.0.0.0 \ --port 30000 \ --model /opt/ai-models/v3-2 \ --kt-weight-path /opt/ai-models/v3-2-CPU \ --kt-cpuinfer 36 \ --kt-threadpool-count 1 \ --kt-num-gpu-experts 0 \ --attention-backend triton \ --trust-remote-code \ --mem-fraction-static 0.98 \ --chunked-prefill-size 2048 \ --max-running-requests 16 \ --max-total-tokens 20000 \ --served-model-name DeepSeek-V3.2 \ --enable-mixed-chunk \ --tensor-parallel-size 1 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --kt-method AMXINT4 ``` I tried adjust `tensor-parallel-size` to 1 or 2, and `kt-num-gpu-experts` to 0 or 1.

bsakaguchi added the

bug

label

2026-10-29 14:48:17 +00:00

bsakaguchi commented

2026-10-29 14:48:18 +00:00

Author

Owner

@SkqLiao commented on GitHub (Dec 9, 2025):

Try reduce --max-total-tokens 20000? It needs to reserve enough VRAM for KV Cache.

@SkqLiao commented on GitHub (Dec 9, 2025): Try reduce `--max-total-tokens 20000`? It needs to reserve enough VRAM for KV Cache.

bsakaguchi commented

2026-10-29 14:48:18 +00:00

Author

Owner

@SkqLiao commented on GitHub (Dec 9, 2025):

Hi, I tried to reproduce your issue on 2x4090(24G) GPUs. I set the exact same parameters and it successfully started.
I tested it on DeepSeek-R1, which has the same parameters as DeepSeek-V3.2.
After the startup was successful, the two GPUs used 13302MiB / 24564MiB of VRAM, indicating that even 2x16GB cards should be able to run it.
Here's the server log from SGLang:

// After loading weights:
Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=12.29 GB, mem usage=10.63 GB.
// After allocating KV cache
[2025-12-09 10:05:11 TP0] Using KV cache dtype: torch.bfloat16
[2025-12-09 10:05:11 TP1] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB
[2025-12-09 10:05:11 TP0] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB
[2025-12-09 10:05:11 TP1] Memory pool end. avail mem=10.97 GB
[2025-12-09 10:05:11 TP0] Memory pool end. avail mem=10.97 GB
// After capturing CUDA graph
[2025-12-09 10:05:21 TP0] Capture cuda graph end. Time elapsed: 10.02 s. mem usage=0.39 GB. avail mem=10.56 GB.

@SkqLiao commented on GitHub (Dec 9, 2025): Hi, I tried to reproduce your issue on 2x4090(24G) GPUs. I set the exact same parameters and it successfully started. I tested it on DeepSeek-R1, which has the same parameters as DeepSeek-V3.2. After the startup was successful, the two GPUs used 13302MiB / 24564MiB of VRAM, indicating that even 2x16GB cards should be able to run it. Here's the server log from SGLang: ``` // After loading weights: Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=12.29 GB, mem usage=10.63 GB. // After allocating KV cache [2025-12-09 10:05:11 TP0] Using KV cache dtype: torch.bfloat16 [2025-12-09 10:05:11 TP1] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB [2025-12-09 10:05:11 TP0] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB [2025-12-09 10:05:11 TP1] Memory pool end. avail mem=10.97 GB [2025-12-09 10:05:11 TP0] Memory pool end. avail mem=10.97 GB // After capturing CUDA graph [2025-12-09 10:05:21 TP0] Capture cuda graph end. Time elapsed: 10.02 s. mem usage=0.39 GB. avail mem=10.56 GB. ```

bsakaguchi commented

2026-10-29 14:48:18 +00:00

Author

Owner

@mrgaolei commented on GitHub (Dec 10, 2025):

Hi, I tried to reproduce your issue on 2x4090(24G) GPUs. I set the exact same parameters and it successfully started. I tested it on DeepSeek-R1, which has the same parameters as DeepSeek-V3.2. After the startup was successful, the two GPUs used 13302MiB / 24564MiB of VRAM, indicating that even 2x16GB cards should be able to run it. Here's the server log from SGLang:
// After loading weights:
Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=12.29 GB, mem usage=10.63 GB.
// After allocating KV cache
[2025-12-09 10:05:11 TP0] Using KV cache dtype: torch.bfloat16
[2025-12-09 10:05:11 TP1] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB
[2025-12-09 10:05:11 TP0] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB
[2025-12-09 10:05:11 TP1] Memory pool end. avail mem=10.97 GB
[2025-12-09 10:05:11 TP0] Memory pool end. avail mem=10.97 GB
// After capturing CUDA graph
[2025-12-09 10:05:21 TP0] Capture cuda graph end. Time elapsed: 10.02 s. mem usage=0.39 GB. avail mem=10.56 GB.

Can you share me your startup script?
I try to reduce --max-total-tokens 1000 also CUDA oom.

@mrgaolei commented on GitHub (Dec 10, 2025): > Hi, I tried to reproduce your issue on 2x4090(24G) GPUs. I set the exact same parameters and it successfully started. I tested it on DeepSeek-R1, which has the same parameters as DeepSeek-V3.2. After the startup was successful, the two GPUs used 13302MiB / 24564MiB of VRAM, indicating that even 2x16GB cards should be able to run it. Here's the server log from SGLang: > > ``` > // After loading weights: > Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=12.29 GB, mem usage=10.63 GB. > // After allocating KV cache > [2025-12-09 10:05:11 TP0] Using KV cache dtype: torch.bfloat16 > [2025-12-09 10:05:11 TP1] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB > [2025-12-09 10:05:11 TP0] KV Cache is allocated. #tokens: 20000, KV size: 1.31 GB > [2025-12-09 10:05:11 TP1] Memory pool end. avail mem=10.97 GB > [2025-12-09 10:05:11 TP0] Memory pool end. avail mem=10.97 GB > // After capturing CUDA graph > [2025-12-09 10:05:21 TP0] Capture cuda graph end. Time elapsed: 10.02 s. mem usage=0.39 GB. avail mem=10.56 GB. > ``` Can you share me your startup script? I try to reduce `--max-total-tokens 1000` also CUDA oom.