[GH-ISSUE #21] Add support to switch main GPU

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Owner

Originally created by @firmanmm on GitHub (Aug 4, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/21

Hi, wanted to ask if there is a way to swap GPU to use the second GPU because currently it will keep using the main GPU, unfortunately for me, my main GPU is 16GB while my second GPU is 24GB. I already tried replacing any mention of cuda:0 to cuda:1 but it still load in GPU 0 which trigger torch.OutOfMemoryError: CUDA out of memory. . Thank you for this awesome project.

Originally created by @firmanmm on GitHub (Aug 4, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/21 Hi, wanted to ask if there is a way to swap GPU to use the second GPU because currently it will keep using the main GPU, unfortunately for me, my main GPU is 16GB while my second GPU is 24GB. I already tried replacing any mention of `cuda:0` to `cuda:1` but it still load in GPU 0 which trigger `torch.OutOfMemoryError: CUDA out of memory.` . Thank you for this awesome project.

bsakaguchi closed this issue

2026-10-29 14:32:40 +00:00

bsakaguchi commented

2026-10-29 14:32:40 +00:00

Author

Owner

@firmanmm commented on GitHub (Aug 4, 2024):

I seems to manage to get it running by adding CUDA_VISIBLE_DEVICES=1 before running it and manage to start it but somehow I didn't get the advertised speed.
I only manage to get this result :
CPU: 7950X3D
GPU: 4080 Super, 3090
Ram: 192GB, speed limited to 3600
Model: DeepSeek-Coder-V2-Instruct

prompt eval count:    14 token(s)
prompt eval duration: 1.2786610126495361s
prompt eval rate:     10.948953523647642 tokens/s
eval count:           160 token(s)
eval duration:        24.879518270492554s
eval rate:            6.430992684844793 tokens/s

However, I did notice that my ram is not getting filled :

@firmanmm commented on GitHub (Aug 4, 2024): I seems to manage to get it running by adding `CUDA_VISIBLE_DEVICES=1` before running it and manage to start it but somehow I didn't get the advertised speed. I only manage to get this result : CPU: 7950X3D GPU: 4080 Super, 3090 Ram: 192GB, speed limited to 3600 Model: DeepSeek-Coder-V2-Instruct ``` prompt eval count: 14 token(s) prompt eval duration: 1.2786610126495361s prompt eval rate: 10.948953523647642 tokens/s eval count: 160 token(s) eval duration: 24.879518270492554s eval rate: 6.430992684844793 tokens/s ``` However, I did notice that my ram is not getting filled : ![image](https://github.com/user-attachments/assets/b316cef3-2329-49b8-99b4-68a03f4d1469)

bsakaguchi commented

2026-10-29 14:32:40 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 5, 2024):

Hi, for now you can try export CUDA_VISIBLE_DEVICE=1 (your second gpu idx) in your bash when you start your program, and leave everything else unchanged.

@Azure-Tang commented on GitHub (Aug 5, 2024): Hi, for now you can try `export CUDA_VISIBLE_DEVICE=1` (your second gpu idx) in your bash when you start your program, and leave everything else unchanged.

bsakaguchi commented

2026-10-29 14:32:40 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 5, 2024):

I seems to manage to get it running by adding CUDA_VISIBLE_DEVICES=1 before running it and manage to start it but somehow I didn't get the advertised speed. I only manage to get this result : CPU: 7950X3D GPU: 4080 Super, 3090 Ram: 192GB, speed limited to 3600 Model: DeepSeek-Coder-V2-Instruct
prompt eval count:    14 token(s)
prompt eval duration: 1.2786610126495361s
prompt eval rate:     10.948953523647642 tokens/s
eval count:           160 token(s)
eval duration:        24.879518270492554s
eval rate:            6.430992684844793 tokens/s
However, I did notice that my ram is not getting filled :

Your RAM seems too low, which may cause extra time to load weights during inference. However, this shouldn’t happen because you have enough RAM for the weights to be loaded into it during the warm-up phase. Can you provide your start command and optimise yaml?

Any idea？
@chenht2022

@Azure-Tang commented on GitHub (Aug 5, 2024): > I seems to manage to get it running by adding `CUDA_VISIBLE_DEVICES=1` before running it and manage to start it but somehow I didn't get the advertised speed. I only manage to get this result : CPU: 7950X3D GPU: 4080 Super, 3090 Ram: 192GB, speed limited to 3600 Model: DeepSeek-Coder-V2-Instruct > > ``` > prompt eval count: 14 token(s) > prompt eval duration: 1.2786610126495361s > prompt eval rate: 10.948953523647642 tokens/s > eval count: 160 token(s) > eval duration: 24.879518270492554s > eval rate: 6.430992684844793 tokens/s > ``` > > However, I did notice that my ram is not getting filled : ![image](https://private-user-images.githubusercontent.com/20027378/354915556-b316cef3-2329-49b8-99b4-68a03f4d1469.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI4Mzk1MTksIm5iZiI6MTcyMjgzOTIxOSwicGF0aCI6Ii8yMDAyNzM3OC8zNTQ5MTU1NTYtYjMxNmNlZjMtMjMyOS00OWI4LTk5YjQtNjhhMDNmNGQxNDY5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODA1VDA2MjY1OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWMxNjFjZjNlNTRkODM0NDc3Y2NjZGMyNDdmMzUzZTlkMzQ0Yzc0ZGMxNzEzZjBmNWIzYjAwNWE5ZDQ1MjE2NGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Aq9k7Exyo8xRU926KY9dSvh7m3HqWZi17woDYeXu2NI) Your RAM seems too low, which may cause extra time to load weights during inference. However, this shouldn’t happen because you have enough RAM for the weights to be loaded into it during the warm-up phase. Can you provide your start command and optimise yaml? Any idea？ @chenht2022

bsakaguchi commented

2026-10-29 14:32:41 +00:00

Author

Owner

@chenht2022 commented on GitHub (Aug 5, 2024):

In htop, green represents Kernel buffers, and orange represents Memory mapped.
The weights of experts in KTransformers are mmap-ed, so this portion of memory usage usually appears as orange. In your screenshot, it accounts for at least 120GB, which is as expected.

Perhaps you can improve performance by increasing parallelism. Try using --cpu_infer 24.

@chenht2022 commented on GitHub (Aug 5, 2024): In `htop`, green represents Kernel buffers, and orange represents Memory mapped. The weights of experts in KTransformers are mmap-ed, so this portion of memory usage usually appears as orange. In your screenshot, it accounts for at least 120GB, which is as expected. Perhaps you can improve performance by increasing parallelism. Try using `--cpu_infer 24`.

bsakaguchi commented

2026-10-29 14:32:41 +00:00

Author

Owner

@firmanmm commented on GitHub (Aug 5, 2024):

Hi @chenht2022 , thanks for the suggestion but it seems setting it to 24 seems to reduce the eval speed a bit :

prompt eval count:    14 token(s)
prompt eval duration: 1.4017484188079834s
prompt eval rate:     9.987526871551813 tokens/s
eval count:           160 token(s)
eval duration:        26.098639011383057s
eval rate:            6.130587879705726 tokens/s

Maybe for extra context I'm running it from the main branch. With below command :

CUDA_VISIBLE_DEVICES=1 python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-Coder-V2-Instruct --gguf_path ../newmodel --cpu_infer 24

@firmanmm commented on GitHub (Aug 5, 2024): Hi @chenht2022 , thanks for the suggestion but it seems setting it to 24 seems to reduce the eval speed a bit : ``` prompt eval count: 14 token(s) prompt eval duration: 1.4017484188079834s prompt eval rate: 9.987526871551813 tokens/s eval count: 160 token(s) eval duration: 26.098639011383057s eval rate: 6.130587879705726 tokens/s ``` Maybe for extra context I'm running it from the main branch. With below command : ``` CUDA_VISIBLE_DEVICES=1 python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-Coder-V2-Instruct --gguf_path ../newmodel --cpu_infer 24 ```

bsakaguchi commented

2026-10-29 14:32:41 +00:00

Author

Owner

@chenht2022 commented on GitHub (Aug 6, 2024):

Since the MoE layer is offloaded to the CPU, its performance bottleneck during the generation phase is the memory bandwidth for reading weights. The comparison with llama.cpp shown in the video was conducted on a machine with an Intel 4th generation Xeon processor, equipped with 8 memory channels, each channel with 4800MHz. Achieving this level of performance places stringent requirements on the hardware.
As a reference, we also conducted tests on another setup configured with an Intel 14900KF + 4090D GPU + 192GB of memory, 2 memory channels, each channel with 4000MHz. The prefill performance for ktransformers and llama.cpp were 65.6 tokens/s and 8.93 tokens/s, respectively, the generate performance were 6.07 tokens/s and 1.23 tokens/s, respectively.

p.s.

You can use the following formula to calculate the equivalent memory bandwidth in generate phase. This value will be lower than the theoretical upper limit of the hardware due to scheduling overhead and waiting for some GPU calculations.

Bandwidth = num_hidden_layers * hidden_size * moe_intermediate_size * num_experts_per_tok * (bytes_per_elem_up + bytes_per_elem_gate + bytes_per_elem_down) * tokens_per_second / 10^9
= 60 * 5120 * 1536 * 6 * (0.562500 + 0.562500 + 0.820312) * 6.43 / 10^9
= 35.4 GB/s

Due to the sparsity of DeepSeek's MoE layer, prefill transitions from being memory bandwidth-bound to computation-bound when the prompt is long enough, so that there is a significant speedup compared to the generation phase.

@chenht2022 commented on GitHub (Aug 6, 2024): Since the MoE layer is offloaded to the CPU, its performance bottleneck during the generation phase is the memory bandwidth for reading weights. The comparison with llama.cpp shown in the video was conducted on a machine with an Intel 4th generation Xeon processor, equipped with 8 memory channels, each channel with 4800MHz. Achieving this level of performance places stringent requirements on the hardware. As a reference, we also conducted tests on another setup configured with an Intel 14900KF + 4090D GPU + 192GB of memory, 2 memory channels, each channel with 4000MHz. The prefill performance for ktransformers and llama.cpp were 65.6 tokens/s and 8.93 tokens/s, respectively, the generate performance were 6.07 tokens/s and 1.23 tokens/s, respectively. p.s. - You can use the following formula to calculate the equivalent memory bandwidth in generate phase. This value will be lower than the theoretical upper limit of the hardware due to scheduling overhead and waiting for some GPU calculations. ``` Bandwidth = num_hidden_layers * hidden_size * moe_intermediate_size * num_experts_per_tok * (bytes_per_elem_up + bytes_per_elem_gate + bytes_per_elem_down) * tokens_per_second / 10^9 = 60 * 5120 * 1536 * 6 * (0.562500 + 0.562500 + 0.820312) * 6.43 / 10^9 = 35.4 GB/s ``` - Due to the sparsity of DeepSeek's MoE layer, prefill transitions from being memory bandwidth-bound to computation-bound when the prompt is long enough, so that there is a significant speedup compared to the generation phase.

bsakaguchi commented

2026-10-29 14:32:41 +00:00

Author

Owner

@firmanmm commented on GitHub (Aug 6, 2024):

Ah, I see. I thought the video being shared in the README actually just shows a standard consumer desktop machine, which rarely have more than 2 channels. Anyway, thank you for the explanations.

@firmanmm commented on GitHub (Aug 6, 2024): Ah, I see. I thought the video being shared in the README actually just shows a standard consumer desktop machine, which rarely have more than 2 channels. Anyway, thank you for the explanations.

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871