kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #184] Does Mooncake support TP parallelism across multiple machines? #86

New issue

Closed

opened 2026-10-29 21:08:57 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:08:57 +00:00

Owner

Originally created by @William-188 on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/184

I started a prefilling process on two machines using vLLM and started a decoding process on another two machines.

Prefilling nodes ：
First, start Ray on each nodes，
And start vllm like this：
python3 -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-R1 --host 0.0.0.0 --port 8100 --trust-remote-code -tp 16 --max-model-len 32000 --enable_chunked_prefill=False --max-seq-len-to-capture 32000 --max-num-batched-tokens 32000 --gpu-memory-utilization 0.8 --max-num-seqs 32 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

Decode nodes：
First, start Ray on each nodes，
And start vllm like this：
python3 -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-R1 --host 0.0.0.0 --port 8100 --trust-remote-code -tp 16 --max-model-len 32000 --enable_chunked_prefill=False --max-seq-len-to-capture 32000 --max-num-batched-tokens 32000 --gpu-memory-utilization 0.8 --max-num-seqs 32 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'

node1:
start etcd:
/usr/bin/etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379

start mooncake_master:
mooncake_master -max_threads 128 --port 50001

mooncake.json:
{
"local_hostname": "node1 ip",
"metadata_server": "etcd://node1 ip:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "node1 ip:50001"
}

node2:
mooncake.json:
{
"local_hostname": "node2 ip",
"metadata_server": "etcd://node1 ip:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "node1 ip:50001"
}

node3:
mooncake.json:
{
"local_hostname": "node3 ip",
"metadata_server": "etcd://node1 ip:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "node1 ip:50001"
}

node4:
mooncake.json:
{
"local_hostname": "node4 ip",
"metadata_server": "etcd://node1 ip:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "node1 ip:50001"
}

The execution runs without errors, but when the decode nodes receive a new request, the GPU utilization drops to 0 before rising again, significantly impacting performance.

g0021_kv_producer_prefilling_16g_0330_3.log
g0023_run_kv_consumer_decode_16g_0330_3.log

Originally created by @William-188 on GitHub (Mar 31, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/184 I started a prefilling process on two machines using vLLM and started a decoding process on another two machines. Prefilling nodes ： First, start Ray on each nodes， And start vllm like this： python3 -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-R1 --host 0.0.0.0 --port 8100 --trust-remote-code -tp 16 --max-model-len 32000 --enable_chunked_prefill=False --max-seq-len-to-capture 32000 --max-num-batched-tokens 32000 --gpu-memory-utilization 0.8 --max-num-seqs 32 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' Decode nodes： First, start Ray on each nodes， And start vllm like this： python3 -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-R1 --host 0.0.0.0 --port 8100 --trust-remote-code -tp 16 --max-model-len 32000 --enable_chunked_prefill=False --max-seq-len-to-capture 32000 --max-num-batched-tokens 32000 --gpu-memory-utilization 0.8 --max-num-seqs 32 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}' node1: start etcd: /usr/bin/etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 start mooncake_master: mooncake_master -max_threads 128 --port 50001 mooncake.json: { "local_hostname": "node1 ip", "metadata_server": "etcd://node1 ip:2379", "protocol": "tcp", "device_name": "", "master_server_address": "node1 ip:50001" } node2: mooncake.json: { "local_hostname": "node2 ip", "metadata_server": "etcd://node1 ip:2379", "protocol": "tcp", "device_name": "", "master_server_address": "node1 ip:50001" } node3: mooncake.json: { "local_hostname": "node3 ip", "metadata_server": "etcd://node1 ip:2379", "protocol": "tcp", "device_name": "", "master_server_address": "node1 ip:50001" } node4: mooncake.json: { "local_hostname": "node4 ip", "metadata_server": "etcd://node1 ip:2379", "protocol": "tcp", "device_name": "", "master_server_address": "node1 ip:50001" } The execution runs without errors, but when the decode nodes receive a new request, the GPU utilization drops to 0 before rising again, significantly impacting performance. [g0021_kv_producer_prefilling_16g_0330_3.log](https://github.com/user-attachments/files/19531843/g0021_kv_producer_prefilling_16g_0330_3.log) [g0023_run_kv_consumer_decode_16g_0330_3.log](https://github.com/user-attachments/files/19531842/g0023_run_kv_consumer_decode_16g_0330_3.log)

bsakaguchi

2026-10-29 21:08:57 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:58 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

TP is supported, but PP is not supported. So, when you run PP and TP at the same time, errors will occur. Currently, all vllm disaggregated connectors do not support PP, because the entire disaggregated framework needs to make some additional modifications for PP.

I see the log and find out that you are using ray to support tp larger than 8, and the model is deepseek. MLA and deepseek are not supported yet. But someone is working on it. You can refer to https://github.com/vllm-project/vllm/pull/14824 if you want a quick workaround.

Apart from this, we haven't verified that the configuration logic of MooncakeStore is compatible with ray yet. Maybe you can try tp = 8 with quantization = fp8?

@ShangmingCai commented on GitHub (Mar 31, 2025): ~~TP is supported, but PP is not supported. So, when you run PP and TP at the same time, errors will occur. Currently, all vllm disaggregated connectors do not support PP, because the entire disaggregated framework needs to make some additional modifications for PP.~~ I see the log and find out that you are using ray to support tp larger than 8, and the model is deepseek. MLA and deepseek are not supported yet. But someone is working on it. You can refer to https://github.com/vllm-project/vllm/pull/14824 if you want a quick workaround. Apart from this, we haven't verified that the configuration logic of MooncakeStore is compatible with ray yet. Maybe you can try tp = 8 with quantization = fp8?

bsakaguchi commented

2026-10-29 21:08:58 +00:00

Author

Owner

@William-188 commented on GitHub (Apr 2, 2025):

TP is supported, but PP is not supported. So, when you run PP and TP at the same time, errors will occur. Currently, all vllm disaggregated connectors do not support PP, because the entire disaggregated framework needs to make some additional modifications for PP.

I see the log and find out that you are using ray to support tp larger than 8, and the model is deepseek. MLA and deepseek are not supported yet. But someone is working on it. You can refer to vllm-project/vllm#14824 if you want a quick workaround.

Apart from this, we haven't verified that the configuration logic of MooncakeStore is compatible with ray yet. Maybe you can try tp = 8 with quantization = fp8?

I have referred to vllm-project/vllm#14824 and edited mooncake_store_connector.py. It seems that MLA is now supported. Are there any other ways to support multi-machine parallelism? 8 GPUs with 8 TP do not have enough GPU memory to run properly.

@William-188 commented on GitHub (Apr 2, 2025): > ~TP is supported, but PP is not supported. So, when you run PP and TP at the same time, errors will occur. Currently, all vllm disaggregated connectors do not support PP, because the entire disaggregated framework needs to make some additional modifications for PP.~ > > I see the log and find out that you are using ray to support tp larger than 8, and the model is deepseek. MLA and deepseek are not supported yet. But someone is working on it. You can refer to [vllm-project/vllm#14824](https://github.com/vllm-project/vllm/pull/14824) if you want a quick workaround. > > Apart from this, we haven't verified that the configuration logic of MooncakeStore is compatible with ray yet. Maybe you can try tp = 8 with quantization = fp8? I have referred to vllm-project/vllm#14824 and edited mooncake_store_connector.py. It seems that MLA is now supported. Are there any other ways to support multi-machine parallelism? 8 GPUs with 8 TP do not have enough GPU memory to run properly.

bsakaguchi commented

2026-10-29 21:08:58 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Apr 2, 2025):

@William-188 Sure. We will work to support tp > 8 with Ray once we finish some high priority jobs first.

@ShangmingCai commented on GitHub (Apr 2, 2025): @William-188 Sure. We will work to support tp > 8 with Ray once we finish some high priority jobs first.

bsakaguchi commented

2026-10-29 21:08:58 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 18, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 18, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:58 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 25, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 25, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:19 +00:00

[PR #86] [MERGED] [TransferEngine] feature: registerLocalMemory support the "*" location. #1316