kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #797] [Report] EPYC 91352 , 2448G DDR5 6400, 4090, SUCCESSFUL #578

New issue

Open

opened 2026-10-29 14:41:25 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:41:25 +00:00

Owner

Originally created by @opernal on GitHub (Mar 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/797

EPYC 5th gen 9135 * 2 (16 cores * 2)
24*48G DDR5 6400, 24 channels
4090 24G

compile with USE_NUMA=1

bios NUMA auto
hyperthreading off

--cpu_infer 31
R1-Q4KM
evalscope result
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| 10% | 0.002 | 0.056 | 39.4701 | 12 | 683 | 17.2 |
| 25% | 0.0022 | 0.0563 | 50.4104 | 16 | 876 | 17.268 |
| 50% | 0.0023 | 0.0568 | 64.9231 | 21 | 1122 | 17.3042 |
| 66% | 0.0026 | 0.0571 | 67.6473 | 21 | 1176 | 17.3774 |
| 75% | 0.0028 | 0.0574 | 72.3129 | 24 | 1257 | 17.3833 |
| 80% | 0.0028 | 0.0575 | 74.2269 | 25 | 1291 | 17.3843 |
| 90% | 0.003 | 0.0583 | 81.1205 | 31 | 1398 | 17.3926 |
| 95% | 0.0038 | 0.0594 | 90.3616 | 31 | 1572 | 17.3968 |
| 98% | 0.0038 | 0.0678 | 90.3616 | 31 | 1572 | 17.3968 |
| 99% | 0.0038 | 0.0773 | 90.3616 | 31 | 1572 | 17.3968 |
+------------+----------+----------+-------------+--------------+---------------+----------------------+

CPU usage reach to near 100%, GPU around 40%

Can better CPUs score more ? plz advise.

thanks.

Originally created by @opernal on GitHub (Mar 5, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/797 EPYC 5th gen 9135 * 2 (16 cores * 2) 24*48G DDR5 6400, 24 channels 4090 24G compile with USE_NUMA=1 bios NUMA auto hyperthreading off --cpu_infer 31 R1-Q4KM evalscope result +------------+----------+----------+-------------+--------------+---------------+----------------------+ | Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) | +------------+----------+----------+-------------+--------------+---------------+----------------------+ | 10% | 0.002 | 0.056 | 39.4701 | 12 | 683 | 17.2 | | 25% | 0.0022 | 0.0563 | 50.4104 | 16 | 876 | 17.268 | | 50% | 0.0023 | 0.0568 | 64.9231 | 21 | 1122 | 17.3042 | | 66% | 0.0026 | 0.0571 | 67.6473 | 21 | 1176 | 17.3774 | | 75% | 0.0028 | 0.0574 | 72.3129 | 24 | 1257 | 17.3833 | | 80% | 0.0028 | 0.0575 | 74.2269 | 25 | 1291 | 17.3843 | | 90% | 0.003 | 0.0583 | 81.1205 | 31 | 1398 | 17.3926 | | 95% | 0.0038 | 0.0594 | 90.3616 | 31 | 1572 | 17.3968 | | 98% | 0.0038 | 0.0678 | 90.3616 | 31 | 1572 | 17.3968 | | 99% | 0.0038 | 0.0773 | 90.3616 | 31 | 1572 | 17.3968 | +------------+----------+----------+-------------+--------------+---------------+----------------------+ CPU usage reach to near 100%, GPU around 40% Can better CPUs score more ? plz advise. thanks.

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@wait0423 commented on GitHub (Mar 5, 2025):

根据其他作者的案例，我认为你的CPU导致性能瓶颈了，目前来看输出token的速度和内存带宽成正比。我看到其他作者使用12通道4800频率的内存，带宽为480GB/s，输出token速度为12-14tks，你的配置带宽近1228GB/s，如果为线性关系，理论上输出速度应该大于24tks，所以我认为你的CPU造成性能瓶颈。你可以查看已使用的内存带宽来确认是否跑满。
以上为个人拙见，若有大佬还请多指教。

@wait0423 commented on GitHub (Mar 5, 2025): 根据其他作者的案例，我认为你的CPU导致性能瓶颈了，目前来看输出token的速度和内存带宽成正比。我看到其他作者使用12通道4800频率的内存，带宽为480GB/s，输出token速度为12-14tks，你的配置带宽近1228GB/s，如果为线性关系，理论上输出速度应该大于24tks，所以我认为你的CPU造成性能瓶颈。你可以查看已使用的内存带宽来确认是否跑满。以上为个人拙见，若有大佬还请多指教。

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@gomingge commented on GitHub (Mar 6, 2025):

CPU核心不够，32核才能满足，建议至少上EPYC 5th 9335。之后大于32核，提升也不会太高了

@gomingge commented on GitHub (Mar 6, 2025): CPU核心不够，32核才能满足，建议至少上EPYC 5th 9335。之后大于32核，提升也不会太高了

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@supermy commented on GitHub (Mar 8, 2025):

桌面版 RTX 4090：
显存类型：GDDR6X
显存位宽：384 位
显存带宽：1.01 TB/s。

移动版 RTX 4090：
显存类型：GDDR6
显存位宽：256 位
显存带宽：576.0 GB/s

PCIe 5.0 x16：
每通道传输速率为 32 GT/s，单向带宽为 64 GB/s
PCIe 4.0 x16：
每通道传输速率为 16 GT/s，单向带宽为 32 GB/s

PCIE 接口是不是瓶颈？

@supermy commented on GitHub (Mar 8, 2025): 桌面版 RTX 4090：显存类型：GDDR6X 显存位宽：384 位显存带宽：1.01 TB/s。移动版 RTX 4090：显存类型：GDDR6 显存位宽：256 位显存带宽：576.0 GB/s PCIe 5.0 x16：每通道传输速率为 32 GT/s，单向带宽为 64 GB/s PCIe 4.0 x16：每通道传输速率为 16 GT/s，单向带宽为 32 GB/s PCIE 接口是不是瓶颈？

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@ubergarm commented on GitHub (Mar 10, 2025):

@wait0423

如何计算理论性能的数学公式？

公式

TG_{max} = \frac{R_b}{P_a \times \left(\frac{bpw}{8}\right)}

定义

$`TG_{max}`$: 理论最大令牌生成速度 (令牌/秒)
$`R_b`$: 内存带宽 (GB/s)
$`P_a`$: 激活的MoE参数量 (十亿参数，B) —— 例如R1模型为37B
$`bpw`$: 量化位数/权重 —— 例如Q4=4，Q2=2.51bpw（通过公式中的常数8转换为字节）

示例计算
当总内存带宽为1228 GB/s，采用Q4@4.0每权重比特量化时：

\frac{1228}{\left(\frac{4}{8} \times 37\right)} \approx 66 \text{ 令牌/秒}

原帖提到"CPU占用率接近100%"，因此我认同一项罕见情况：内存带宽可能未成为瓶颈。DDR5-6400内存质量上乘，但具体取决于NUMA配置是否正确，以确保数据并行操作正常。

@gomingge

的确，这似乎是CPU成为性能瓶颈的罕见案例。

@supermy

你可通过查看 nvtop 右上角显示的当前活动的PCIe代及速率来验证。我倾向于认为这不是问题根源。需注意空闲时数值会变化，例如：

Device 1 [NVIDIA RTX A6000] PCIe GEN1@16x RX: 400.0 KiB/s TX: 500.0 KiB/s

@opernal

这速度极佳！是我所听过的最佳案例之一。能否提供 numactl --hardware --cpu-compress 的输出结果，以便明确 bios NUMA auto的具体含义？若您正在使用 USE_NUMA=1，建议在BIOS中显式设置为 NPS1 —— 此配置将呈现两个NUMA节点，此时 ktransformers 会为数据并行操作加载双份模型权重。

您可以尝试定制混合量化方案，例如结合 fp8/Q8_0 权重以最大化输出质量，或进一步调整速度与质量平衡。该团队在HuggingFace KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid 提供了一个现成方案，可能最便于测试。

若 NUMA auto 显示恰好两个NUMA节点（每个CPU插槽对应一个），则很可能CPU已达极限 —— 这在多数机器受内存带宽制约（且NUMA配置加剧瓶颈）的背景下实属罕见。

目前llama.cpp 的实验分支及优化分支ik_llama.cpp 尚未实现数据并行功能（据我所知），因此当前性能可能略逊于 ktransformers。

祝顺利！请持续分享您的测试结果！感谢关于 EvalScope 的提示，期待了解您用于基准测试的具体命令！

wait0423

Based on cases from other authors, I believe your CPU is causing a performance bottleneck. Currently, the token output speed appears to be proportional to memory bandwidth. I’ve observed other authors using 12-channel 4800 MHz memory with a bandwidth of 480 GB/s, achieving 12–14 tokens per second (tks). Your configuration has a bandwidth of ~1228 GB/s. If this relationship is linear, the theoretical output speed should exceed 24 tks. Thus, I suspect your CPU is the bottleneck. You can check the utilized memory bandwidth to confirm if it’s saturated.

These are just my personal thoughts. If any experts have insights, please enlighten me.

What is the math to calculate theoretical performance?

Formula

TG_{max} = \frac{R_b}{P_a \times \left(\frac{bpw}{8}\right)}

Definitions

$`TG_{max}`$: Theoretical Maximum Token Generation Speed (tokens/second)
$`R_b`$: RAM Bandwidth (GB/s)
$`P_a`$: MoE Parameters Activated (Billions (B)) – e.g., 37B for R1
$`bpw`$: Quantization, bits per weight – e.g., Q4=4, Q2=2.51bpw. (converted to bytes using constant 8 in equation)

Example Calculation
For 1228 GB/s aggregate RAM bandwidth running Q4@4.0 bits-per-weight quantization:

\frac{1228}{\left(\frac{4}{8} \times 37\right)} \approx 66 \text{ tokens/sec}

The OP says "CPU usage reach to near 100%" so yes I agree this is a rare case where the RAM bandwidth may not be the bottle-neck. DDR5-6400 is very good quality. It depends on the exact NUMA configuration though to confirm proper data parallel operation.

gomingge

Insufficient CPU cores. A 32-core configuration is required. It’s recommended to use at least an EPYC 5th Gen 9335. Beyond32 cores, the improvement won’t be significant.

Yes, it does seem like this is the rare case where CPU is the bottle-neck.

supermy

Desktop RTX 4090:

VRAM Type: GDDR6X
Memory Bus Width: 384-bit
Memory Bandwidth: 1.01 TB/s

Mobile RTX 4090:

VRAM Type: GDDR6
Memory Bus Width: 256-bit
Memory Bandwidth: 576.0 GB/s

PCIe 5.0 x16:

Per-lane transfer rate: 32 GT/s
Unidirectional bandwidth: 64 GB/s

PCIe 4.0 x16:

Per-lane transfer rate: 16 GT/s
Unidirectional bandwidth: 32 GB/s

Is the PCIe interface a bottleneck?

You can check this looking at the top right corner of nvtop to see the active PCIe GEN and rate. I doubt that this is the issue. Keep in mind it will change when idle like so:

Device 1 [NVIDIA RTX A6000] PCIe GEN 1@16x RX: 400.0 KiB/s TX: 500.0 KiB/s

opernal

Heya that is great speed! One of the best I have heard reported. Can you give the output of numactl --hardware --cpu-compress to clarify what bios NUMA auto means? If you are using USE_NUMA=1 I would suggest setting explicitly BIOS to NPS1 to present two NUMA nodes which ktransformers will load the model weights twice for data parallel operation.

You might want to try rolling your own hybrid fp8 quant, e.g. mixed fp8 / Q8_0 weights for max quality output or even more custom for a balance of speed and quality. This team provided one on huggingface KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid that might be the easiest to test.

Assuming NUMA auto shows exactly two NUMA nodes (one for each CPU socket) then you are probably maxing out CPU which is rare given most machines are bottle-necked on RAM bandwidth which is exacerbated by NUMA node challenges.

There are also some experimental branches of llama.cpp and a new optimized fork called ik_llama.cpp but neither implements data parallel yet afaict so probably will be slightly worse than ktransformers today.

Good luck and keep us posted with your results! Thanks for the tip on EvalScope, would be curious to see the command you run to get that benchmark!

@ubergarm commented on GitHub (Mar 10, 2025): @wait0423 如何计算理论性能的数学公式？ **公式** ```math TG_{max} = \frac{R_b}{P_a \times \left(\frac{bpw}{8}\right)} ``` **定义** - $ $`TG_{max}`$ $: 理论最大令牌生成速度 (令牌/秒) - $ $`R_b`$ $: 内存带宽 (GB/s) - $ $`P_a`$ $: 激活的MoE参数量 (十亿参数，B) —— 例如R1模型为37B - $ $`bpw`$ $: 量化位数/权重 —— 例如Q4=4，Q2=2.51bpw（通过公式中的常数8转换为字节） **示例计算** 当总内存带宽为1228 GB/s，采用Q4@4.0每权重比特量化时： ```math \frac{1228}{\left(\frac{4}{8} \times 37\right)} \approx 66 \text{ 令牌/秒} ``` 原帖提到"CPU占用率接近100%"，因此我认同一项罕见情况：内存带宽可能未成为瓶颈。DDR5-6400内存质量上乘，但具体取决于NUMA配置是否正确，以确保数据并行操作正常。 @gomingge 的确，这似乎是CPU成为性能瓶颈的罕见案例。 @supermy 你可通过查看 `nvtop` 右上角显示的当前活动的PCIe代及速率来验证。我倾向于认为这不是问题根源。需注意空闲时数值会变化，例如： ``` Device 1 [NVIDIA RTX A6000] PCIe GEN1@16x RX: 400.0 KiB/s TX: 500.0 KiB/s ``` @opernal 这速度*极佳*！是我所听过的最佳案例之一。能否提供 `numactl --hardware --cpu-compress` 的输出结果，以便明确 `bios NUMA auto`的具体含义？若您正在使用 `USE_NUMA=1`，建议在BIOS中显式设置为 `NPS1` —— 此配置将呈现两个NUMA节点，此时 ktransformers 会为数据并行操作加载*双份*模型权重。您可以尝试定制混合量化方案，例如结合 `fp8`/`Q8_0` 权重以最大化输出质量，或进一步调整速度与质量平衡。该团队在[HuggingFace KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid](https://huggingface.co/KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid/tree/main/DeepSeek-R1-q4km-fp8) 提供了一个现成方案，可能最便于测试。若 `NUMA auto` 显示恰好两个NUMA节点（每个CPU插槽对应一个），则很可能CPU已达极限 —— 这在多数机器受内存带宽制约（且NUMA配置加剧瓶颈）的背景下实属罕见。目前[llama.cpp](https://github.com/ggml-org/llama.cpp/pull/11397#issuecomment-2702482878) 的实验分支及优化分支[ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) 尚未实现数据并行功能（据我所知），因此当前性能可能略逊于 ktransformers。祝顺利！请持续分享您的测试结果！感谢关于 `EvalScope` 的提示，期待了解您用于基准测试的具体命令！ --- wait0423 > Based on cases from other authors, I believe your CPU is causing a performance bottleneck. Currently, the token output speed appears to be proportional to memory bandwidth. I’ve observed other authors using 12-channel 4800 MHz memory with a bandwidth of 480 GB/s, achieving 12–14 tokens per second (tks). Your configuration has a bandwidth of ~1228 GB/s. If this relationship is linear, the theoretical output speed should exceed 24 tks. Thus, I suspect your CPU is the bottleneck. You can check the utilized memory bandwidth to confirm if it’s saturated. > These are just my personal thoughts. If any experts have insights, please enlighten me. What is the math to calculate theoretical performance? **Formula** ```math TG_{max} = \frac{R_b}{P_a \times \left(\frac{bpw}{8}\right)} ``` **Definitions** - $ $`TG_{max}`$ $: Theoretical Maximum Token Generation Speed (tokens/second) - $ $`R_b`$ $: RAM Bandwidth (GB/s) - $ $`P_a`$ $: MoE Parameters Activated (Billions (B)) – e.g., 37B for R1 - $ $`bpw`$ $: Quantization, bits per weight – e.g., Q4=4, Q2=2.51bpw. (converted to bytes using constant 8 in equation) **Example Calculation** For 1228 GB/s aggregate RAM bandwidth running Q4@4.0 bits-per-weight quantization: ```math \frac{1228}{\left(\frac{4}{8} \times 37\right)} \approx 66 \text{ tokens/sec} ``` The OP says "CPU usage reach to near 100%" so yes I agree this is a rare case where the RAM bandwidth may not be the bottle-neck. DDR5-6400 is very good quality. It depends on the exact NUMA configuration though to confirm proper data parallel operation. gomingge > Insufficient CPU cores. A 32-core configuration is required. It’s recommended to use at least an EPYC 5th Gen 9335. Beyond32 cores, the improvement won’t be significant. Yes, it does seem like this is the rare case where CPU is the bottle-neck. supermy > **Desktop RTX 4090**: - VRAM Type: GDDR6X - Memory Bus Width: 384-bit - Memory Bandwidth: 1.01 TB/s > **Mobile RTX 4090**: - VRAM Type: GDDR6 - Memory Bus Width: 256-bit - Memory Bandwidth: 576.0 GB/s > **PCIe 5.0 x16**: - Per-lane transfer rate: 32 GT/s - Unidirectional bandwidth: 64 GB/s >**PCIe 4.0 x16**: - Per-lane transfer rate: 16 GT/s - Unidirectional bandwidth: 32 GB/s > Is the PCIe interface a bottleneck? You can check this looking at the top right corner of `nvtop` to see the active PCIe GEN and rate. I doubt that this is the issue. Keep in mind it will change when idle like so: ``` Device 1 [NVIDIA RTX A6000] PCIe GEN 1@16x RX: 400.0 KiB/s TX: 500.0 KiB/s ``` opernal Heya that is *great* speed! One of the best I have heard reported. Can you give the output of `numactl --hardware --cpu-compress` to clarify what `bios NUMA auto` means? If you are using `USE_NUMA=1` I would suggest setting explicitly BIOS to `NPS1` to present two NUMA nodes which ktransformers will load the model weights *twice* for data parallel operation. You might want to try rolling your own hybrid `fp8` quant, e.g. mixed `fp8` / `Q8_0` weights for max quality output or even more custom for a balance of speed and quality. This team provided one on [huggingface KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid](https://huggingface.co/KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid/tree/main/DeepSeek-R1-q4km-fp8) that might be the easiest to test. Assuming `NUMA auto` shows exactly two NUMA nodes (one for each CPU socket) then you are probably maxing out CPU which is rare given most machines are bottle-necked on RAM bandwidth which is exacerbated by NUMA node challenges. There are also some experimental branches of [llama.cpp](https://github.com/ggml-org/llama.cpp/pull/11397#issuecomment-2702482878) and a new optimized fork called [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) but neither implements data parallel yet afaict so probably will be slightly worse than ktransformers today. Good luck and keep us posted with your results! Thanks for the tip on `EvalScope`, would be curious to see the command you run to get that benchmark!

bsakaguchi commented

2026-10-29 14:41:26 +00:00

Author

Owner

@opernal commented on GitHub (Mar 14, 2025):

@ubergarm

Hey there. Plz check info below. hope it helps.

Numa settings:
numactl --hardware --cpu-compress
available: 2 nodes (0-1)
node 0 cpus: 0-15 (16)
node 0 size: 580088 MB
node 0 free: 314825 MB
node 1 cpus: 16-31 (16)
node 1 size: 580505 MB
node 1 free: 276529 MB
node distances:
node 0 1
0: 10 32
1: 32 10

EvalScope command:
evalscope perf --url "http://127.0.0.1:10002/v1/chat/completions" --parallel 1 --model DeepSeek-Coder-V2-Instruct --number 15 --api openai --dataset openqa --stream --tokenizer-path "./ktransformers/DeepSeek-R1"

@wait0423 @gomingge
CPU is the key factor of this setup. Cores' not enough. Trying to figure out ways to upgrade CPUs. Quite expensive for now.... hmmm

@opernal commented on GitHub (Mar 14, 2025): @ubergarm Hey there. Plz check info below. hope it helps. Numa settings: numactl --hardware --cpu-compress available: 2 nodes (0-1) node 0 cpus: 0-15 (16) node 0 size: 580088 MB node 0 free: 314825 MB node 1 cpus: 16-31 (16) node 1 size: 580505 MB node 1 free: 276529 MB node distances: node 0 1 0: 10 32 1: 32 10 EvalScope command: evalscope perf --url "http://127.0.0.1:10002/v1/chat/completions" --parallel 1 --model DeepSeek-Coder-V2-Instruct --number 15 --api openai --dataset openqa --stream --tokenizer-path "./ktransformers/DeepSeek-R1" @wait0423 @gomingge CPU is the key factor of this setup. Cores' not enough. Trying to figure out ways to upgrade CPUs. Quite expensive for now.... hmmm