[GH-ISSUE #522] [Bug] the Mooncake Transfer Engine Troughput suddenly drops to half，When the sglang server receives thousands of requests #183

New issue

Closed

opened 2026-10-29 21:10:35 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:10:35 +00:00

Owner

Originally created by @zhaoerfang on GitHub (Jun 19, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/522

Describe the bug

[Bug] the Mooncake Transfer Engine Troughput suddenly drops to half，When the sglang server receives thousands of requests。I used two machines to start prefill and decode services, using Mooncake transfer engine. I used sglang.bench_serving to test twice, each time with 1000 requests, and each request output length was 512. In the first test, Mooncake Transfer Engine Troughput was about 130MB/s, P99 TTFT was about 2 seconds, and in the second test, Transfer Engine Troughput was about 70MB/s, and P99 TTFT was increased to 12s. However, there is no problem when the length of bench_serving is small (e.g. 5). Has anyone else encountered similar situations? I hope to get your help.
Reproduction
prefill

NCCL_IB_HCA=mlx5_bond_0 (200G*2 RDMA) NCCL_SOCKET_IFNAME=mlx5_bond_0_ip python3 -m sglang.launch_server --model-path DeepSeek-R1-Distill-Qwen-32B --disaggregation-mode prefill --port xx --host mlx5_bond_0_ip --tp-size 8 --disaggregation-ib-device mlx5_bond_0
Environment

Sglang version v0.4.7
Used Model DeepSeek-R1-Distill-Qwen-32B
GPU L208
network Roce 200G2

Originally created by @zhaoerfang on GitHub (Jun 19, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/522 Describe the bug [Bug] the Mooncake Transfer Engine Troughput suddenly drops to half，When the sglang server receives thousands of requests。I used two machines to start prefill and decode services, using Mooncake transfer engine. I used sglang.bench_serving to test twice, each time with 1000 requests, and each request output length was 512. In the first test, Mooncake Transfer Engine Troughput was about 130MB/s, P99 TTFT was about 2 seconds, and in the second test, Transfer Engine Troughput was about 70MB/s, and P99 TTFT was increased to 12s. However, there is no problem when the length of bench_serving is small (e.g. 5). Has anyone else encountered similar situations? I hope to get your help. Reproduction prefill NCCL_IB_HCA=mlx5_bond_0 (200G*2 RDMA) NCCL_SOCKET_IFNAME=mlx5_bond_0_ip python3 -m sglang.launch_server --model-path DeepSeek-R1-Distill-Qwen-32B --disaggregation-mode prefill --port xx --host mlx5_bond_0_ip --tp-size 8 --disaggregation-ib-device mlx5_bond_0 Environment Sglang version v0.4.7 Used Model DeepSeek-R1-Distill-Qwen-32B GPU L208 network Roce 200G2

bsakaguchi closed this issue

2026-10-29 21:10:35 +00:00

bsakaguchi commented

2026-10-29 21:10:36 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jun 19, 2025):

Try configuring --page-size 64.

@ShangmingCai commented on GitHub (Jun 19, 2025): Try configuring `--page-size 64`.

bsakaguchi commented

2026-10-29 21:10:36 +00:00

Author