[GH-ISSUE #1091] [Performance]: P/D separation is slower than 3 identical vllm instances node #358

New issue

Closed

opened 2026-10-29 21:13:20 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:13:20 +00:00

Owner

Originally created by @BilyZ98 on GitHub (Nov 21, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1091

Describe your performance question

I used mooncake master as transfer engine for P/D separation with RDMA enabled. The results show that TTFT and TPOT is both higher than that of without P/D separation. This is not a good case to use mooncake master.
Is this because that I did not start mooncake store nodes ?
I also tried p2p nccl as kv connector which shows that TTFT is higher but TPOT is lower which shows the advantage of P/D separation somehow.
But for mooncake I expect that both TTFT and TPOT is lower since it has a gloa cache buffer so there is less prefill work for prefill nodes and decode nodes can get kv cache faster. The results I get for now is not expected.

There is only 12.5 GB for this master service , how can I confugre to user for storage space for master serivce ?

Do I need to start a mooncake store for each vllm instances at the same time ?

Here's benchmark configuration and resutls.
Benchmark path: Mooncake/benchmarks/xypd_benchmarks/vllm-benchmarks

mooncake.json

{
"local_hostname": "localhost",
"metadata_server": "http://localhost:8081/metadata",
"protocol": "rdma",
"device_name": "mlx5_0,mlx5_1",
"master_server_address": "localhost:10001"
}

Configration:

VLLM_SRC_PATH=${VLLM_SRC_PATH:-"vllm-src"}
MOONCAKE_CONFIG_PATH=${MOONCAKE_CONFIG_PATH:-"mooncake.json"}
MODEL=${MODEL:-"/HOME/nsccgz_qylin/nsccgz_qylinxy_1/HDD_POOL/zzt/Qwen2-7B-Instruct"}
DEMO_PATH=${DEMO_PATH:-"../proxy_demo.py"}
NUM_PREFILL=${NUM_PREFILL:-"2"}
NUM_DECODE=${NUM_DECODE:-"2"}
PREFIX_LEN=${PREFIX_LEN:-"512"}
NUM_FOLDS=${NUM_FOLDS:-"20"}
MASTER_PORT=${MASTER_PORT:-"10001"}
PREFILL_PORT_BASE=${PREFILL_PORT_BASE:-"8100"}
DECODE_PORT_BASE=${DECODE_PORT_BASE:-"8200"}
PROXY_PORT=${PROXY_PORT:-"8500"}
logs_root=${LOG_ROOT:-"logs"}
results_root=${RESULT_ROOT:-"results"}

Input length: 1024, output length: 1024, concurrency: 8
Benchmark results

3 identical gpus/nodes without p/d

Successful requests:                     160       
Benchmark duration (s):                  147.17    
Total input tokens:                      245521    
Total generated tokens:                  163840    
Request throughput (req/s):              1.09      
Output token throughput (tok/s):         1113.28   
Total Token throughput (tok/s):          2781.58   
---------------Time to First Token----------------
Mean TTFT (ms):                          67.13     
Median TTFT (ms):                        62.79     
P99 TTFT (ms):                           150.34    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.09      
Median TPOT (ms):                        7.11      
P99 TPOT (ms):                           7.19      
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.09      
Median ITL (ms):                         7.03      
P99 ITL (ms):                            7.86      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          7317.74   
Median E2EL (ms):                        7340.87   
P99 E2EL (ms):                           7473.99   
==================================================

with p/d, 2 prefill nodes, 1 decode nodes

============ Serving Benchmark Result ============
Successful requests:                     160       
Benchmark duration (s):                  187.21    
Total input tokens:                      245521    
Total generated tokens:                  163840    
Request throughput (req/s):              0.85      
Output token throughput (tok/s):         875.15    
Total Token throughput (tok/s):          2186.59   
---------------Time to First Token----------------
Mean TTFT (ms):                          1631.30   
Median TTFT (ms):                        1616.03   
P99 TTFT (ms):                           1866.69   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.55      
Median TPOT (ms):                        7.50      
P99 TPOT (ms):                           7.83      
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.55      
Median ITL (ms):                         7.48      
P99 ITL (ms):                            8.20      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          9352.89   
Median E2EL (ms):                        9335.43   
P99 E2EL (ms):                           9550.55   
==================================================

Master logs

Master Metrics: Storage: 11.72 GB / 12.50 GB (93.8%) | Keys: 95

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @BilyZ98 on GitHub (Nov 21, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1091 ### Describe your performance question I used mooncake master as transfer engine for P/D separation with RDMA enabled. The results show that TTFT and TPOT is both higher than that of without P/D separation. This is not a good case to use mooncake master. Is this because that I did not start mooncake store nodes ? I also tried p2p nccl as kv connector which shows that TTFT is higher but TPOT is lower which shows the advantage of P/D separation somehow. But for mooncake I expect that both TTFT and TPOT is lower since it has a gloa cache buffer so there is less prefill work for prefill nodes and decode nodes can get kv cache faster. The results I get for now is not expected. There is only 12.5 GB for this master service , how can I confugre to user for storage space for master serivce ? Do I need to start a mooncake store for each vllm instances at the same time ? Here's benchmark configuration and resutls. Benchmark path: `Mooncake/benchmarks/xypd_benchmarks/vllm-benchmarks` mooncake.json ``` { "local_hostname": "localhost", "metadata_server": "http://localhost:8081/metadata", "protocol": "rdma", "device_name": "mlx5_0,mlx5_1", "master_server_address": "localhost:10001" } ``` Configration: ``` VLLM_SRC_PATH=${VLLM_SRC_PATH:-"vllm-src"} MOONCAKE_CONFIG_PATH=${MOONCAKE_CONFIG_PATH:-"mooncake.json"} MODEL=${MODEL:-"/HOME/nsccgz_qylin/nsccgz_qylinxy_1/HDD_POOL/zzt/Qwen2-7B-Instruct"} DEMO_PATH=${DEMO_PATH:-"../proxy_demo.py"} NUM_PREFILL=${NUM_PREFILL:-"2"} NUM_DECODE=${NUM_DECODE:-"2"} PREFIX_LEN=${PREFIX_LEN:-"512"} NUM_FOLDS=${NUM_FOLDS:-"20"} MASTER_PORT=${MASTER_PORT:-"10001"} PREFILL_PORT_BASE=${PREFILL_PORT_BASE:-"8100"} DECODE_PORT_BASE=${DECODE_PORT_BASE:-"8200"} PROXY_PORT=${PROXY_PORT:-"8500"} logs_root=${LOG_ROOT:-"logs"} results_root=${RESULT_ROOT:-"results"} ``` Input length: 1024, output length: 1024, concurrency: 8 Benchmark results 3 identical gpus/nodes without p/d ``` Successful requests: 160 Benchmark duration (s): 147.17 Total input tokens: 245521 Total generated tokens: 163840 Request throughput (req/s): 1.09 Output token throughput (tok/s): 1113.28 Total Token throughput (tok/s): 2781.58 ---------------Time to First Token---------------- Mean TTFT (ms): 67.13 Median TTFT (ms): 62.79 P99 TTFT (ms): 150.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.09 Median TPOT (ms): 7.11 P99 TPOT (ms): 7.19 ---------------Inter-token Latency---------------- Mean ITL (ms): 7.09 Median ITL (ms): 7.03 P99 ITL (ms): 7.86 ----------------End-to-end Latency---------------- Mean E2EL (ms): 7317.74 Median E2EL (ms): 7340.87 P99 E2EL (ms): 7473.99 ================================================== ``` with p/d, 2 prefill nodes, 1 decode nodes ``` ============ Serving Benchmark Result ============ Successful requests: 160 Benchmark duration (s): 187.21 Total input tokens: 245521 Total generated tokens: 163840 Request throughput (req/s): 0.85 Output token throughput (tok/s): 875.15 Total Token throughput (tok/s): 2186.59 ---------------Time to First Token---------------- Mean TTFT (ms): 1631.30 Median TTFT (ms): 1616.03 P99 TTFT (ms): 1866.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.55 Median TPOT (ms): 7.50 P99 TPOT (ms): 7.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 7.55 Median ITL (ms): 7.48 P99 ITL (ms): 8.20 ----------------End-to-end Latency---------------- Mean E2EL (ms): 9352.89 Median E2EL (ms): 9335.43 P99 E2EL (ms): 9550.55 ================================================== ``` Master logs ``` Master Metrics: Storage: 11.72 GB / 12.50 GB (93.8%) | Keys: 95 ``` ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:13:20 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:13:21 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 18, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 18, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:13:21 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 25, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jun 25, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:28:03 +00:00

[PR #358] [CLOSED] feat(store): enhance logging with environment-configured levels #1509