kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #538] 内存占用很少速度极其慢 500G内存，单卡4090 #388

New issue

Closed

opened 2026-10-29 14:38:57 +00:00 by bsakaguchi · 11 comments

bsakaguchi commented

2026-10-29 14:38:57 +00:00

Owner

Originally created by @seanzhang-zhichen on GitHub (Feb 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/538

到底怎么才能复现作者的速度

Originally created by @seanzhang-zhichen on GitHub (Feb 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/538 ![Image](https://github.com/user-attachments/assets/4b8f4a07-59c1-41d5-99ff-37b07f5dbd7a) 到底怎么才能复现作者的速度

bsakaguchi

2026-10-29 14:38:57 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@codedoves commented on GitHub (Feb 20, 2025):

你的硬件配置和运行命令是什么

@codedoves commented on GitHub (Feb 20, 2025): 你的硬件配置和运行命令是什么

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@seanzhang-zhichen commented on GitHub (Feb 20, 2025):

是因为我得内存带宽太小了吗？但是都没咋用内存啊

@seanzhang-zhichen commented on GitHub (Feb 20, 2025): ![Image](https://github.com/user-attachments/assets/ec333a8e-7133-4313-b171-ccd9551f1e28) 是因为我得内存带宽太小了吗？但是都没咋用内存啊

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@yansiyu550 commented on GitHub (Feb 20, 2025):

我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6%

sysbench 测试内存带宽是单线程的，我测出来 22GB

@yansiyu550 commented on GitHub (Feb 20, 2025): 我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6% sysbench 测试内存带宽是单线程的，我测出来 22GB

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@seanzhang-zhichen commented on GitHub (Feb 20, 2025):

我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6%

sysbench 测试内存带宽是单线程的，我测出来 22GB

我开了32个线程测，也没到这么大的内存带宽

另一个服务器，带宽有10G/s 但推理速度还是 0.3token/s 不知道什么原因了

@seanzhang-zhichen commented on GitHub (Feb 20, 2025): > 我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6% > > sysbench 测试内存带宽是单线程的，我测出来 22GB 我开了32个线程测，也没到这么大的内存带宽另一个服务器，带宽有10G/s 但推理速度还是 0.3token/s 不知道什么原因了

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@seanzhang-zhichen commented on GitHub (Feb 20, 2025):

到底怎么才能复现作者的速度

看了下，虽然占用少，但是cache中占了挺多内存，内存应该还是用上了的

@seanzhang-zhichen commented on GitHub (Feb 20, 2025): > ![Image](https://github.com/user-attachments/assets/4b8f4a07-59c1-41d5-99ff-37b07f5dbd7a) > > 到底怎么才能复现作者的速度看了下，虽然占用少，但是cache中占了挺多内存，内存应该还是用上了的

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@seanzhang-zhichen commented on GitHub (Feb 20, 2025):

这推理速度实在是一言难尽啊

@seanzhang-zhichen commented on GitHub (Feb 20, 2025): 这推理速度实在是一言难尽啊 ![Image](https://github.com/user-attachments/assets/b1f20c54-e213-4526-9daa-511626903658)

bsakaguchi commented

2026-10-29 14:38:58 +00:00

Author

Owner

@yansiyu550 commented on GitHub (Feb 20, 2025):

这推理速度实在是一言难尽啊

CPU支持AMX吗

@yansiyu550 commented on GitHub (Feb 20, 2025): > 这推理速度实在是一言难尽啊 CPU支持AMX吗

bsakaguchi commented

2026-10-29 14:38:59 +00:00

Author

Owner

@yileld commented on GitHub (Feb 20, 2025):

我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6%
sysbench 测试内存带宽是单线程的，我测出来 22GB

我开了32个线程测，也没到这么大的内存带宽

另一个服务器，带宽有10G/s 但推理速度还是 0.3token/s 不知道什么原因了

正常的，官方的带宽600G/s呢

@yileld commented on GitHub (Feb 20, 2025): > > 我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6% > > sysbench 测试内存带宽是单线程的，我测出来 22GB > > 我开了32个线程测，也没到这么大的内存带宽 > > 另一个服务器，带宽有10G/s 但推理速度还是 0.3token/s 不知道什么原因了正常的，官方的带宽600G/s呢

bsakaguchi commented

2026-10-29 14:38:59 +00:00

Author

Owner

@yangzetao commented on GitHub (Feb 21, 2025):

我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6%

sysbench 测试内存带宽是单线程的，我测出来 22GB

你的设备好牛。
另外，你的这个结果是AMX加速的，对么？
如果使用AVX512优化版本，你的Decode能达到多少？我很想知道在ktransformer中AMX和 AVX512的性能有多大的差距。 @yansiyu550

@yangzetao commented on GitHub (Feb 21, 2025): > 我测了预填充 50.53 token/s，Decode 10.11 token/s，内存带宽实测352 GB/s，内存：810.5G/944G = 85.9%，显存：15375MiB / 24564MiB = 15.01G/24G = 62.6% > > sysbench 测试内存带宽是单线程的，我测出来 22GB 你的设备好牛。另外，你的这个结果是AMX加速的，对么？如果使用AVX512优化版本，你的Decode能达到多少？我很想知道在ktransformer中AMX和 AVX512的性能有多大的差距。 @yansiyu550

bsakaguchi commented

2026-10-29 14:38:59 +00:00

Author

Owner

@xieyongshuai commented on GitHub (Feb 26, 2025):

同样的问题，内存用不到并且速度极慢

@xieyongshuai commented on GitHub (Feb 26, 2025): 同样的问题，内存用不到并且速度极慢

bsakaguchi commented

2026-10-29 14:38:59 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！