[GH-ISSUE #1478] 长输入就是会慢吗,claudecode #985

New issue

Closed

opened 2026-10-29 14:47:11 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:47:11 +00:00

Owner

Originally created by @wlz987 on GitHub (Aug 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1478

/workspace/LLM/Qwen/Qwen3-Coder-30B-A3B-Instruct.gguf

prefill_batch_i: 127,
prefill_batch_i: 127,
padded_batch_size 54 capture_padded_batch_size 54
Model execution time (GPU): 11286.153 ms, 0.089 tokens/s
198
53661
prefill_batch_i: 127,
prefill_batch_i: 127,
padded_batch_size 54 capture_padded_batch_size 54
Model execution time (GPU): 11390.851 ms, 0.088 tokens/s
12
21324
prefill_batch_i: 127,
prefill_batch_i: 127,
padded_batch_size 54 capture_padded_batch_size 54

我仅仅说你好就是 10+token/s;
A100 80g + 1TB 内存

Originally created by @wlz987 on GitHub (Aug 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1478 /workspace/LLM/Qwen/Qwen3-Coder-30B-A3B-Instruct.gguf prefill_batch_i: 127, prefill_batch_i: 127, padded_batch_size 54 capture_padded_batch_size 54 Model execution time (GPU): 11286.153 ms, 0.089 tokens/s 198 53661 prefill_batch_i: 127, prefill_batch_i: 127, padded_batch_size 54 capture_padded_batch_size 54 Model execution time (GPU): 11390.851 ms, 0.088 tokens/s 12 21324 prefill_batch_i: 127, prefill_batch_i: 127, padded_batch_size 54 capture_padded_batch_size 54 我仅仅说你好就是 10+token/s; A100 80g + 1TB 内存

bsakaguchi closed this issue

2026-10-29 14:47:11 +00:00

bsakaguchi commented

2026-10-29 14:47:12 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 20, 2025):

prefill阶段取决于算力。因为kt异构的特性，在prefill阶段天然受到cpu算力的限制。如果想体验更高的prefill速度可以尝试使用带有amx的cpu

@Azure-Tang commented on GitHub (Aug 20, 2025): prefill阶段取决于算力。因为kt异构的特性，在prefill阶段天然受到cpu算力的限制。如果想体验更高的prefill速度可以尝试使用带有amx的cpu