kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #62] Why is ITL's first token so long? #36

New issue

Closed

opened 2026-10-29 21:08:14 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:08:14 +00:00

Owner

Originally created by @sunshenao on GitHub (Jan 3, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/62

model: Qwen/Qwen2.5-72B-Instruct-GPTQ-Int4
GPU: H20 * 8
P: 4H20
D: 4H20
input:1024,output : 6

sudo sh disagg_performance_benchmark.sh

This is what I get when qps=10.

The average ITL is much larger for separated than for non-separated, but the median ITL for separated is much smaller.
I see this in the result file generated

The first token of the decode part always takes a long time.
This phenomenon does not occur at lower qps, e.g., when qps = 2

But when the qps is small, the ITL increase is not very obvious. but the TTFT increases a lot

May I ask why this is so, is there any way to reduce the time of the first token of decode?
I look forward to your answer. think you

Originally created by @sunshenao on GitHub (Jan 3, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/62 model: Qwen/Qwen2.5-72B-Instruct-GPTQ-Int4 GPU: H20 * 8 P: 4*H20 D: 4*H20 input:1024,output : 6 sudo sh disagg_performance_benchmark.sh This is what I get when qps=10. ![mean_itl_ms](https://github.com/user-attachments/assets/2659e7de-7702-47b9-a9e4-a0cbeb9f0d48) ![median_itl_ms](https://github.com/user-attachments/assets/29ad5ba6-1bd1-48b2-ad30-aea6e28a0e49) The average ITL is much larger for separated than for non-separated, but the median ITL for separated is much smaller. I see this in the result file generated ![image](https://github.com/user-attachments/assets/5531400d-96db-4beb-ab59-443253f81e51) The first token of the decode part always takes a long time. This phenomenon does not occur at lower qps, e.g., when qps = 2 ![image](https://github.com/user-attachments/assets/160cb0f0-3479-467b-a2dc-72c563ddbbe1) But when the qps is small, the ITL increase is not very obvious. but the TTFT increases a lot ![image](https://github.com/user-attachments/assets/d2b79876-85a3-4627-9b80-4016ff630d19) May I ask why this is so, is there any way to reduce the time of the first token of decode? I look forward to your answer. think you

bsakaguchi

2026-10-29 21:08:14 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:15 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jan 7, 2025):

In disaggregated prefilling scenarios, the first token of the decode (i.e., TTFT) consists of the prefill stage overhead, KVCache transfer cost, and the first run overhead of the decode stage. Since the layer-wise KVCache transfer is not ready yet, the unsatisfactory performance of TTFT is as expected temporarily.

Also, the implementation of KVCache transfer is not in a zero-copy fashion currently. And there is a buffer_lock in the implementation of simple_buffer.py, which might cause troubles when QPS is large.

Please refer to the roadmap of the disaggregated prefilling feature of vLLM (https://github.com/vllm-project/vllm/issues/10818) and Mooncake (#44), there remains much work to do before we make this feature production-level ready.

BTW, how many GPUs do you use for the set of chunked prefill experiments?

@ShangmingCai commented on GitHub (Jan 7, 2025): In disaggregated prefilling scenarios, the first token of the decode (i.e., TTFT) consists of the prefill stage overhead, KVCache transfer cost, and the first run overhead of the decode stage. Since the layer-wise KVCache transfer is not ready yet, the unsatisfactory performance of TTFT is as expected temporarily. Also, the implementation of KVCache transfer is not in a zero-copy fashion currently. And there is a `buffer_lock` in the implementation of `simple_buffer.py`, which might cause troubles when QPS is large. Please refer to the roadmap of the disaggregated prefilling feature of vLLM (https://github.com/vllm-project/vllm/issues/10818) and Mooncake (#44), there remains much work to do before we make this feature production-level ready. BTW, how many GPUs do you use for the set of chunked prefill experiments?

bsakaguchi commented

2026-10-29 21:08:15 +00:00

Author

Owner

@sunshenao commented on GitHub (Jan 7, 2025):

Thanks for your answer, I started 2 chunked prefill instances with 4*h20 each, just like the disagg_performance_benchmark.sh configuration.
The other question I have is, is that so far the results of my tests don't show any increase in performance compared to the chunked prefill instances, what is the reason for this?

For example, my results running on a30

gpu: A30
prefill: 1A30
decode: 1 A30
model : Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
mooncake.json: {
"prefill_url": "localhost:13003",
"decode_url": "localhost:13103",
"metadata_server": "localhost:2379",
"metadata_backend": "etcd",
"protocol": "tcp",
"device_name": ""
}
input_len: 256, output_len: 6
sudo sh disagg_performance_benchmark.sh

@sunshenao commented on GitHub (Jan 7, 2025): Thanks for your answer, I started 2 chunked prefill instances with 4*h20 each, just like the disagg_performance_benchmark.sh configuration. The other question I have is, is that so far the results of my tests don't show any increase in performance compared to the chunked prefill instances, what is the reason for this? For example, my results running on a30 gpu: A30 prefill: 1*A30 decode: 1* A30 model : Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 mooncake.json: { "prefill_url": "localhost:13003", "decode_url": "localhost:13103", "metadata_server": "localhost:2379", "metadata_backend": "etcd", "protocol": "tcp", "device_name": "" } input_len: 256, output_len: 6 sudo sh disagg_performance_benchmark.sh ![image](https://github.com/user-attachments/assets/8485f414-eb93-4f8b-aba6-114ea3156825) ![image](https://github.com/user-attachments/assets/5bf08613-2373-4ba9-9cc7-a84c168a7c7a) ![image](https://github.com/user-attachments/assets/1bc88877-9ab7-47e4-80cb-610c3168322e) ![image](https://github.com/user-attachments/assets/4d3b8e10-be86-4084-84e0-2a5a8db8a52a)

bsakaguchi commented

2026-10-29 21:08:15 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jan 7, 2025):

Thanks for your answer, I started 2 chunked prefill instances with 4*h20 each, just like the disagg_performance_benchmark.sh configuration. The other question I have is, is that so far the results of my tests don't show any increase in performance compared to the chunked prefill instances, what is the reason for this?

Yes. Since PD separation is similar to pipeline processing at the current stage, even with high QPS, there exist computing bubbles on both nodes. However, for the two chunked prefill implementations, the resources of each GPU can be fully utilized, and because it does not involve KVCache transmission between nodes, the performance will not be limited by the network.

Therefore, 1P1D tests will not show any increase in performance compared to 2 chunked prefill instances. To better evaluate the practicality of PD separation, we need the implementation of XpYd, and also a heterogeneous GPU environment and high-speed network.

@ShangmingCai commented on GitHub (Jan 7, 2025): > Thanks for your answer, I started 2 chunked prefill instances with 4*h20 each, just like the disagg_performance_benchmark.sh configuration. The other question I have is, is that so far the results of my tests don't show any increase in performance compared to the chunked prefill instances, what is the reason for this? > Yes. Since PD separation is similar to pipeline processing at the current stage, even with high QPS, there exist computing bubbles on both nodes. However, for the two chunked prefill implementations, the resources of each GPU can be fully utilized, and because it does not involve KVCache transmission between nodes, the performance will not be limited by the network. Therefore, 1P1D tests will not show any increase in performance compared to 2 chunked prefill instances. To better evaluate the practicality of PD separation, we need the implementation of XpYd, and also a heterogeneous GPU environment and high-speed network.

bsakaguchi commented

2026-10-29 21:08:15 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:15 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 30, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 30, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.