[GH-ISSUE #8] Two GPU configuration? #1

New issue

Closed

opened 2026-10-29 14:32:32 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:32:32 +00:00

Owner

Originally created by @krzysiekpodk on GitHub (Jul 28, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/8

I tried to modify YAML for deepseek to utilize more vram across two GPUs, any suggestions how to modify it - i.e. which modules could potentially also weight around 24gb?

Originally created by @krzysiekpodk on GitHub (Jul 28, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/8 I tried to modify YAML for deepseek to utilize more vram across two GPUs, any suggestions how to modify it - i.e. which modules could potentially also weight around 24gb?

bsakaguchi closed this issue

2026-10-29 14:32:32 +00:00

bsakaguchi commented

2026-10-29 14:32:33 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Jul 29, 2024):

We have designed our operators to support different devices, which potentially allowing pipelines to utilize two devices simultaneously. But by default, all weights are loaded onto cuda:0 for convenience, considering that our project primarily serves local llama users. To enable pipeline processing on a single compute node, we need to implement a device map to manage the device allocation for operations. This approach could also potentially extend to multi-node pipelines. I have added this feature to my to-do list and plan to introduce it soon.

@Azure-Tang commented on GitHub (Jul 29, 2024): We have designed our operators to support different devices, which potentially allowing pipelines to utilize two devices simultaneously. But by default, all weights are loaded onto cuda:0 for convenience, considering that our project primarily serves local llama users. To enable pipeline processing on a single compute node, we need to implement a device map to manage the device allocation for operations. This approach could also potentially extend to multi-node pipelines. I have added this feature to my to-do list and plan to introduce it soon.

bsakaguchi commented

2026-10-29 14:32:33 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 16, 2024):

Hi,

We’ve updated our multi-GPU configuration. You can find the tutorial here. Additionally, we’ve provided yaml examples for 2-GPU setups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules.

If you encounter any issues with the two-GPU configuration, don’t hesitate to reach out.

@Azure-Tang commented on GitHub (Aug 16, 2024): Hi, We’ve updated our multi-GPU configuration. You can find the tutorial [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/injection_tutorial.md). Additionally, we’ve provided yaml examples for 2-GPU setups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules. If you encounter any issues with the two-GPU configuration, don’t hesitate to reach out.

bsakaguchi commented

2026-10-29 14:32:33 +00:00

Author

Owner

@Yongbozzz commented on GitHub (Feb 21, 2025):

etups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules

Hi , thank you so much , I tried the 2 GPUs version and it works, but the performance is not better than 1 GPU? I see the total GPU utils is not changing compared with single GPU(2 GPU: 17%+16% ,1 GPU:30+%)

@Yongbozzz commented on GitHub (Feb 21, 2025): > etups using deepseek, qwen. These examples are available in the following directory: ktransformers/optimize/optimize_rules Hi , thank you so much , I tried the 2 GPUs version and it works, but the performance is not better than 1 GPU? I see the total GPU utils is not changing compared with single GPU(2 GPU: 17%+16% ,1 GPU:30+%)

bsakaguchi referenced this issue

2026-10-29 14:37:20 +00:00

[GH-ISSUE #420] deepseek r1 using 4090 nvidia running, meet segmentation fault(core dumped) problem #288

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:38:09 +00:00

[GH-ISSUE #484] Docker run local chat raise error, please confirm whether AMD CPU can run ktransformers #342

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646