[GH-ISSUE #20] Does mooncake work with one machine? #13

New issue

Closed

opened 2026-10-29 21:07:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:07:53 +00:00

Owner

Originally created by @wxsms on GitHub (Dec 5, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/20

Thank you for the awesome work in advance! I have a small question here, as the doc stat:

Local memcpy: If the target Segment is actually in the local DRAM/VRAM, direct data copy interfaces such as memcpy, cudaMemcpy are used.

But I can not found any cudaMemcpy usage in the source code. When I try vLLM with Mooncake plus tcp protocol on 2 GPU connected by NVLink, the inference is quite slow on high QPS. Was this expected or how can I correctly configure this on single machine?

Originally created by @wxsms on GitHub (Dec 5, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/20 Thank you for the awesome work in advance! I have a small question here, as the doc stat: > Local memcpy: If the target Segment is actually in the local DRAM/VRAM, direct data copy interfaces such as memcpy, cudaMemcpy are used. But I can not found any `cudaMemcpy` usage in the source code. When I try vLLM with Mooncake plus tcp protocol on 2 GPU connected by NVLink, the inference is quite slow on high QPS. Was this expected or how can I correctly configure this on single machine?

bsakaguchi closed this issue

2026-10-29 21:07:53 +00:00

bsakaguchi commented

2026-10-29 21:07:54 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 6, 2024):

The local memcpy feature in the doc is available if you use the RDMA protocol. (The TCP protocol does not implement this feature.) See mooncake-transfer-engine/src/transport/rdma_transport/worker_pool.cpp:199. We are now testing VRAM-direct KVCache transfer and will release later.

@alogfans commented on GitHub (Dec 6, 2024): The `local memcpy` feature in the doc is available if you use the RDMA protocol. (The TCP protocol does not implement this feature.) See [`mooncake-transfer-engine/src/transport/rdma_transport/worker_pool.cpp:199`](https://github.com/kvcache-ai/Mooncake/blob/e1479903367e484c4d547b3251b5f56df91c7c8e/mooncake-transfer-engine/src/transport/rdma_transport/worker_pool.cpp#L199). We are now testing VRAM-direct KVCache transfer and will release later.

bsakaguchi referenced this issue

2026-10-29 21:17:35 +00:00

[GH-ISSUE #2440] [Bug]: Execute mooncake+hixl, final destruction phase core #646