kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #5] benchmark performance #5

New issue

Closed

opened 2026-10-29 21:07:44 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:07:44 +00:00

Owner

Originally created by @BaiStone2017 on GitHub (Nov 28, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5

In https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm_benchmark_results.md。

The performance of "Non-disaggregated", use 2 A10?

Originally created by @BaiStone2017 on GitHub (Nov 28, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5 In https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm_benchmark_results.md。 The performance of "Non-disaggregated", use 2 A10?

bsakaguchi

2026-10-29 21:07:44 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:07:45 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Nov 29, 2024):

In https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm_benchmark_results.md。

The performance of "Non-disaggregated", use 2 A10?

Currently, it is conducted on 1 A10 to test and compare the TTFT latency and verify the feasibility of inter-node disaggregated designs. To fairly compare the total throughput of non-disaggregated and disaggregated designs, we need to conduct experiments under specific prefill/decode workloads to utilize the prefill node fully. However, we have not found a good way to conduct a fair comparison of 2 non-disaggregated instances and 1 prefill + 1 decode without OOM.
According to the author of PR 8498,

"for disagg prefill it will have lower throughput compared to chunked prefill if the prefill workload / decode workload doesn’t match # of prefill GPUs / # of decode GPUs. In my current implementation, the # of prefill GPU / # of decode GPU is 1:1, but the prefill workload / decode workload is typically a really small number (roughly 0.1 IIRC)."

After we solve the TP problem, we will conduct a series of experiments with different GPU ratios. If you are interested, you can also join vllm's slack channel about prefill disaggregation to get the latest updates.

@ShangmingCai commented on GitHub (Nov 29, 2024): > In [https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm_benchmark_results.md。](https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm_benchmark_results.md%E3%80%82) > > The performance of "Non-disaggregated", use 2 A10? Currently, it is conducted on 1 A10 to test and compare the TTFT latency and verify the feasibility of inter-node disaggregated designs. To fairly compare the total throughput of non-disaggregated and disaggregated designs, we need to conduct experiments under specific prefill/decode workloads to utilize the prefill node fully. However, we have not found a good way to conduct a fair comparison of 2 non-disaggregated instances and 1 prefill + 1 decode without OOM. According to the author of PR 8498, > "for disagg prefill it will have lower throughput compared to chunked prefill if the prefill workload / decode workload doesn’t match # of prefill GPUs / # of decode GPUs. In my current implementation, the # of prefill GPU / # of decode GPU is 1:1, but the prefill workload / decode workload is typically a really small number (roughly 0.1 IIRC)." After we solve the TP problem, we will conduct a series of experiments with different GPU ratios. If you are interested, you can also join vllm's slack channel about prefill disaggregation to get the latest updates.

bsakaguchi commented

2026-10-29 21:07:45 +00:00

Author

Owner

@Edenzzzz commented on GitHub (Nov 29, 2024):

Are there benchmark comparisons against NCCL?

@Edenzzzz commented on GitHub (Nov 29, 2024): Are there benchmark comparisons against NCCL?

bsakaguchi commented

2026-10-29 21:07:45 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Nov 29, 2024):

Are there benchmark comparisons against NCCL?

We are unable to obtain inter-node disaggregated results with NCCL based on PR 8498 currently due to its parallel_state initialization process of disagg_group in conflict with vllm's process_group. This could be fixed with the help of PR 10072, which has already been merged. More results will be provided once we finish the integration of mooncake_transfer_engine with PR 10502.

@ShangmingCai commented on GitHub (Nov 29, 2024): > Are there benchmark comparisons against NCCL? We are unable to obtain inter-node disaggregated results with NCCL based on PR 8498 currently due to its `parallel_state` initialization process of `disagg_group` in conflict with vllm's `process_group`. This could be fixed with the help of PR 10072, which has already been merged. More results will be provided once we finish the integration of `mooncake_transfer_engine` with PR 10502.

bsakaguchi commented

2026-10-29 21:07:45 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 25, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 25, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:07:45 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 2, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Aug 2, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:17:35 +00:00

[GH-ISSUE #2440] [Bug]: Execute mooncake+hixl, final destruction phase core #646