[GH-ISSUE #178] TTFTs increases with query processing #84

New issue

Closed

opened 2026-10-29 21:08:54 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:08:54 +00:00

Owner

Originally created by @zhenwei-intel on GitHub (Mar 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/178

Hello, thank you for this amazing project!

I’m encountering performance issues when running a 1P1D:

Baseline: 2 instances server w/o chunked prefill.
1P1D: 1 prefill & 1 decode, with tcp Mooncakestoreconnector.

Observed Behavior:
TTFT remains stable even with query processing in baseline. while TTFT increases dramatically with query processing in 1P1D.

Benchmark Script: python ../benchmark_serving.py --backend vllm --model meta-llama/Meta-Llama-3.1-8B-Instruct --dataset-name sonnet --dataset-path ../sonnet_4x.txt --sonnet-input-len 1024 --sonnet-output-len 6 --sonnet-prefix-len 50 --num-prompts 100 --port 8000 --host localhost --request-rate 2

Originally created by @zhenwei-intel on GitHub (Mar 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/178 Hello, thank you for this amazing project! I’m encountering performance issues when running a 1P1D: - Baseline: 2 instances server w/o chunked prefill. - 1P1D: 1 prefill & 1 decode, with tcp Mooncakestoreconnector. **Observed Behavior**: **TTFT remains stable even with query processing in baseline. while TTFT increases dramatically with query processing in 1P1D.** Benchmark Script: python ../benchmark_serving.py --backend vllm --model meta-llama/Meta-Llama-3.1-8B-Instruct --dataset-name sonnet --dataset-path ../sonnet_4x.txt --sonnet-input-len 1024 --sonnet-output-len 6 --sonnet-prefix-len 50 --num-prompts 100 --port 8000 --host localhost --request-rate 2 ![Image](https://github.com/user-attachments/assets/6292f2c8-26c1-4026-8cf7-66ab81bd2768) ![Image](https://github.com/user-attachments/assets/bf4bc7ad-770d-4748-9efd-f2bf95c45437)

bsakaguchi closed this issue

2026-10-29 21:08:55 +00:00

bsakaguchi commented

2026-10-29 21:08:55 +00:00

Author

Owner

@onepick commented on GitHub (May 11, 2025):

@zhenwei-intel I hit the same issue, did you have any solution?

@onepick commented on GitHub (May 11, 2025): @zhenwei-intel I hit the same issue, did you have any solution?

bsakaguchi referenced this issue

2026-10-29 21:26:19 +00:00

[PR #84] [MERGED] fix: when detect async event, stop the outstanding requests & prevent cq burn out leading severe error #1317