kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #345] 为啥我用8卡4090跑的，速度还没有单卡的快呢？ #225

New issue

Closed

opened 2026-10-29 14:36:04 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 14:36:04 +00:00

Owner

Originally created by @zky001 on GitHub (Feb 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/345

如题，我修改了推理优化的配置，启用了8卡来跑，但是实际测试下来，速度和我用单卡跑的差不多，我用的是256G的内存，跑的是自己2bit量化的版本

Originally created by @zky001 on GitHub (Feb 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/345 如题，我修改了推理优化的配置，启用了8卡来跑，但是实际测试下来，速度和我用单卡跑的差不多，我用的是256G的内存，跑的是自己2bit量化的版本 ![Image](https://github.com/user-attachments/assets/43614eeb-b45c-4ebe-897f-8fb370a73959)

bsakaguchi

2026-10-29 14:36:04 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:36:05 +00:00

Author

Owner

@zinccat commented on GitHub (Feb 16, 2025):

我个人猜测可能存在的问题包括2bit kernel支持差, cpu并不适合跑2bit, 以及offload没有针对性优化

@zinccat commented on GitHub (Feb 16, 2025): 我个人猜测可能存在的问题包括2bit kernel支持差, cpu并不适合跑2bit, 以及offload没有针对性优化

bsakaguchi commented

2026-10-29 14:36:05 +00:00

Author

Owner

@zky001 commented on GitHub (Feb 16, 2025):

其实如果能利用大内存+多GPU来做计算，估计效果会很赞，这种情况下如果用上5080的8卡机，支持4bit的模型可能就速度快很多

@zky001 commented on GitHub (Feb 16, 2025): 其实如果能利用大内存+多GPU来做计算，估计效果会很赞，这种情况下如果用上5080的8卡机，支持4bit的模型可能就速度快很多

bsakaguchi commented

2026-10-29 14:36:05 +00:00

Author

Owner

@yaoxue1993 commented on GitHub (Feb 16, 2025):

因为计算瓶颈在cpu。而且8卡流水线并行只会增加通信开销

@yaoxue1993 commented on GitHub (Feb 16, 2025): 因为计算瓶颈在cpu。而且8卡流水线并行只会增加通信开销

bsakaguchi commented

2026-10-29 14:36:05 +00:00

Author

Owner

@exumeOAO commented on GitHub (Feb 16, 2025):

因为4090不支持nvlink

@exumeOAO commented on GitHub (Feb 16, 2025): 因为4090不支持nvlink

bsakaguchi commented

2026-10-29 14:36:06 +00:00

Author

Owner

@zinccat commented on GitHub (Feb 16, 2025):

其实8卡 2bit 已经可以全部放下了，那llamacpp就应该很快

@zinccat commented on GitHub (Feb 16, 2025): 其实8卡 2bit 已经可以全部放下了，那llamacpp就应该很快

bsakaguchi commented

2026-10-29 14:36:06 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！