[GH-ISSUE #131] Multi-GPU Deployment Best Practices Request #71

New issue

Closed

opened 2026-10-29 14:33:35 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 14:33:35 +00:00

Owner

Originally created by @ChuangLee on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/131

I have several machines equipped with 8×4090 GPUs, and I believe many users are in a similar situation. While I was able to successfully deploy deepseek-v2 on a single GPU, my experience with multi-GPU deployment has been challenging.

Could you please provide a guide or best practices for deploying deepseek-v2 on multi-GPU systems? Additionally, if there are any performance benchmarks or reports available, that information would be very helpful.

Thanks for your support!

Best regards

Originally created by @ChuangLee on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/131 I have several machines equipped with 8×4090 GPUs, and I believe many users are in a similar situation. While I was able to successfully deploy deepseek-v2 on a single GPU, my experience with multi-GPU deployment has been challenging. Could you please provide a guide or best practices for deploying deepseek-v2 on multi-GPU systems? Additionally, if there are any performance benchmarks or reports available, that information would be very helpful. Thanks for your support! Best regards

bsakaguchi

2026-10-29 14:33:35 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:33:36 +00:00

Author

Owner

@Atream commented on GitHub (Feb 11, 2025):

You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask.

Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU.

@Atream commented on GitHub (Feb 11, 2025): You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask. Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU.

bsakaguchi commented

2026-10-29 14:33:36 +00:00

Author

Owner

@ArYuZzz commented on GitHub (Feb 12, 2025):

You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask.

Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU.

Could you please post the link of this multi-GPU yaml example?
Additionally, my server has two A100 without NVLink, does simply place more experts on GPU accelerate the inference since the bandwidth of PCIe is pretty low?
Or can I use two A100 to place KV-Cache to get longer context?
It seems hard to fully utilize two GPU without NVLink.

@ArYuZzz commented on GitHub (Feb 12, 2025): > You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask. > > Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU. Could you please post the link of this `multi-GPU yaml example`? Additionally, my server has two A100 without NVLink, does simply place more experts on GPU accelerate the inference since the bandwidth of PCIe is pretty low? Or can I use two A100 to place KV-Cache to get longer context? It seems hard to fully utilize two GPU without NVLink.

bsakaguchi commented

2026-10-29 14:33:36 +00:00

Author

Owner

@Atream commented on GitHub (Feb 12, 2025):

You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask.
Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU.

Could you please post the link of this multi-GPU yaml example? Additionally, my server has two A100 without NVLink, does simply place more experts on GPU accelerate the inference since the bandwidth of PCIe is pretty low? Or can I use two A100 to place KV-Cache to get longer context? It seems hard to fully utilize two GPU without NVLink.

Currently, we do not support TP, and the communication volume is very low. Whether there is an NVLink between the two GPUs is not important. We will soon provide a YAML example for using multiple GPUs. Please wait for a moment.

@Atream commented on GitHub (Feb 12, 2025): > > You can try to place more experts on the GPU using KExpertsMarlin. You can refer to the multi-GPU yaml example we provided for configuration. If you encounter any issues, feel free to ask. > > Currently, KTransformers does not yet support batch serving, and it also lacks support for Tensor Parallel. As a result, using multiple GPUs may not offer significant advantages compared to using a single GPU with larger VRAM. This is because without Tensor Parallel, the model cannot effectively distribute its computations across multiple GPUs to achieve better performance. Instead, the model may still be limited by the memory and computational capacity of a single GPU. > > Could you please post the link of this `multi-GPU yaml example`? Additionally, my server has two A100 without NVLink, does simply place more experts on GPU accelerate the inference since the bandwidth of PCIe is pretty low? Or can I use two A100 to place KV-Cache to get longer context? It seems hard to fully utilize two GPU without NVLink. Currently, we do not support TP, and the communication volume is very low. Whether there is an NVLink between the two GPUs is not important. We will soon provide a YAML example for using multiple GPUs. Please wait for a moment.

bsakaguchi commented

2026-10-29 14:33:36 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

bsakaguchi referenced this issue

2026-10-29 14:51:34 +00:00

[PR #71] [MERGED] [fix] Fix qlen > chunk_size mask is none error #1342