kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #90] What is the communication protocol for two GPUs within the same node? #50

New issue

Closed

opened 2026-10-29 21:08:25 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:08:25 +00:00

Owner

Originally created by @Rookie-Kai on GitHub (Jan 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/90

In your provided example, it's mentioned that communication can be handled using TCP or RDMA. However, in my case, the two GPUs reside within the same node. For GPUs on the same node, RDMA is typically not necessary for communication. In this scenario, to achieve optimal performance, should I still use TCP for communication? If not, what communication method should I configure to achieve the best results?

Additionally, I have another issue. My communication protocol is TCP. After deploying using Mooncake for separate prefill and decode stages with vLLM, I conducted a stress test. Initially, it functioned correctly, but after a very short period, the entire service hung. The decode side displayed the following metrics:
Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%

However, when using the same configuration with only vLLM (without Mooncake's separation), this problem does not occur. Could this issue be caused by my use of TCP for communication?

my environments：
vllm: 0.6.6.post1
gcc: 11.4.0
cmake: 3.29.0
GPU: 4090D *2 (same node)
model_size: 7B

Thank you very much for your open-source contribution. I understand you are very busy, but I would be extremely grateful if you could provide some insights into this issue. Thank you!

Originally created by @Rookie-Kai on GitHub (Jan 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/90 In your provided example, it's mentioned that communication can be handled using TCP or RDMA. However, in my case, the two GPUs reside within the same node. For GPUs on the same node, RDMA is typically not necessary for communication. In this scenario, to achieve optimal performance, should I still use TCP for communication? If not, what communication method should I configure to achieve the best results? Additionally, I have another issue. My communication protocol is TCP. After deploying using Mooncake for separate prefill and decode stages with vLLM, I conducted a stress test. Initially, it functioned correctly, but after a very short period, the entire service hung. The decode side displayed the following metrics: `Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%` However, when using the same configuration with only vLLM (without Mooncake's separation), this problem does not occur. Could this issue be caused by my use of TCP for communication? **my environments：** **vllm:** 0.6.6.post1 **gcc:** 11.4.0 **cmake:** 3.29.0 **GPU:** 4090D *2 (same node) **model_size:** 7B Thank you very much for your open-source contribution. I understand you are very busy, but I would be extremely grateful if you could provide some insights into this issue. Thank you!

bsakaguchi

2026-10-29 21:08:25 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:26 +00:00

Author

Owner

@alogfans commented on GitHub (Jan 23, 2025):

Currently Mooncake supports copying data using cudaMalloc/malloc if the destination is in the same process. We are investigating how to optimize the transfer between two processes in the same machine.
It seems the problem is caused by error of Tcp transport. Did you find some error messages from Transfer Engine?

@alogfans commented on GitHub (Jan 23, 2025): 1) Currently Mooncake supports copying data using cudaMalloc/malloc if the destination is in the same process. We are investigating how to optimize the transfer between two processes in the same machine. 2) It seems the problem is caused by error of Tcp transport. Did you find some error messages from Transfer Engine?

bsakaguchi commented

2026-10-29 21:08:26 +00:00

Author

Owner

@Rookie-Kai commented on GitHub (Jan 23, 2025):

目前 Mooncake 支持使用 cudaMalloc/malloc 复制数据（如果目标位于同一进程中）。我们正在研究如何优化同一台机器中两个进程之间的传输。

问题似乎是由 Tcp 传输错误引起的。您是否发现 Transfer Engine 的一些错误消息？

Thank you for your response. Unfortunately, I did not find any error messages in the Transfer Engine. I ran the sample proxy.py file, and normally, when a request is successful, the proxy_server returns a 200 status. However, when the service hangs, each request shows the following --

Additionally, I would like to confirm something. I have two 4090D GPUs on the same node, connected via PCIe, and internal communication can be done through the intranet. Currently, when using Mooncake + vLLM, is TCP the only option? For intra-node communication, do we need to wait for the next update?

@Rookie-Kai commented on GitHub (Jan 23, 2025): > 1. 目前 Mooncake 支持使用 cudaMalloc/malloc 复制数据（如果目标位于同一进程中）。我们正在研究如何优化同一台机器中两个进程之间的传输。 > 2. 问题似乎是由 Tcp 传输错误引起的。您是否发现 Transfer Engine 的一些错误消息？ Thank you for your response. Unfortunately, I did not find any error messages in the Transfer Engine. I ran the sample proxy.py file, and normally, when a request is successful, the proxy_server returns a 200 status. However, when the service hangs, each request shows the following `--` Additionally, I would like to confirm something. I have two 4090D GPUs on the same node, connected via PCIe, and internal communication can be done through the intranet. Currently, when using Mooncake + vLLM, is TCP the only option? For intra-node communication, do we need to wait for the next update?

bsakaguchi commented

2026-10-29 21:08:26 +00:00

Author

Owner

@stmatengss commented on GitHub (Jan 23, 2025):

目前 Mooncake 支持使用 cudaMalloc/malloc 复制数据（如果目标位于同一进程中）。我们正在研究如何优化同一台机器中两个进程之间的传输。

问题似乎是由 Tcp 传输错误引起的。您是否发现 Transfer Engine 的一些错误消息？

Thank you for your response. Unfortunately, I did not find any error messages in the Transfer Engine. I ran the sample proxy.py file, and normally, when a request is successful, the proxy_server returns a 200 status. However, when the service hangs, each request shows the following --

Additionally, I would like to confirm something. I have two 4090D GPUs on the same node, connected via PCIe, and internal communication can be done through the intranet. Currently, when using Mooncake + vLLM, is TCP the only option? For intra-node communication, do we need to wait for the next update?

#77 is a PR trying to support intra-node communication with shared memory. We will merge it into the main after finishing enough tests.

@stmatengss commented on GitHub (Jan 23, 2025): > > 1. 目前 Mooncake 支持使用 cudaMalloc/malloc 复制数据（如果目标位于同一进程中）。我们正在研究如何优化同一台机器中两个进程之间的传输。 > > 2. 问题似乎是由 Tcp 传输错误引起的。您是否发现 Transfer Engine 的一些错误消息？ > > Thank you for your response. Unfortunately, I did not find any error messages in the Transfer Engine. I ran the sample proxy.py file, and normally, when a request is successful, the proxy_server returns a 200 status. However, when the service hangs, each request shows the following `--` > > Additionally, I would like to confirm something. I have two 4090D GPUs on the same node, connected via PCIe, and internal communication can be done through the intranet. Currently, when using Mooncake + vLLM, is TCP the only option? For intra-node communication, do we need to wait for the next update? #77 is a PR trying to support intra-node communication with shared memory. We will merge it into the main after finishing enough tests.

bsakaguchi commented

2026-10-29 21:08:26 +00:00

Author

Owner

@hasB4K commented on GitHub (Feb 5, 2025):

If I understand correctly, you guys do not support NVLink as of now? Shared memory through mmap seems sub-optimal.

@hasB4K commented on GitHub (Feb 5, 2025): If I understand correctly, you guys do not support NVLink as of now? Shared memory through mmap seems sub-optimal.

bsakaguchi commented

2026-10-29 21:08:27 +00:00

Author

Owner

@alogfans commented on GitHub (Feb 6, 2025):

If I understand correctly, you guys do not support NVLink as of now? Shared memory through mmap seems sub-optimal.

Right, we are developing Transfer Engine to enable mmap and nvlink.

@alogfans commented on GitHub (Feb 6, 2025): > If I understand correctly, you guys do not support NVLink as of now? Shared memory through mmap seems sub-optimal. Right, we are developing Transfer Engine to enable mmap and nvlink.

bsakaguchi commented

2026-10-29 21:08:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 30, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 30, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.