kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3031] [Feature Request]: Support remote CPU memory destinations over MNNVL/NVLink #779

New issue

Closed

opened 2026-10-29 21:19:26 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:19:26 +00:00

Owner

Originally created by @thincal on GitHub (Jul 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3031

Describe your feature request

Problem

When use Mooncake Transfer Engine for Prefill/Decode KV transfer on GB200/GB300 NVL systems. SGLang HiSparse's direct-to-host path registers a pinned CPU KV pool on the Decode side as the transfer destination.

This path works over RDMA, but fails when the target segment uses MNNVL/NVLink:

... nvlink_transport.cpp:497] Requested address ... not found!

Observed with Mooncake 0.3.11. From the current transport behavior:

NvlinkTransport::registerLocalMemory() publishes remote-accessible metadata for CUDA allocations backed by cuMemCreate / Fabric handles, but a pinned CPU buffer is not exported as a remote NVLink target.
When the target segment protocol is nvlink, transport selection stays on NVLink and does not fall back to RDMA for a host-memory destination.
Registration may appear to succeed, while the transfer fails later in relocateSharedMemoryAddress() because the destination range is absent from the remote metadata.

As a result, CPU-backed KV-cache paths such as HiSparse cannot use the inter-node NVLink fabric on GB systems. They must use RDMA or add GPU staging, so the GB NVLink interconnect cannot be fully utilized for this workload.

Requested capability

Please support remote CPU-memory destinations with MNNVL/NVLink. Possible designs include:

Provide a Mooncake allocation/registration API for CUDA VMM HOST_NUMA / EGM memory backed by Fabric-exportable handles, so frameworks such as SGLang can use it as their CPU KV pool.
Alternatively, support capability-aware dual registration/routing: use MNNVL for HBM destinations and automatically route CPU destinations through RDMA.
Detect unsupported pinned-host registrations early and return a clear error, instead of failing at transfer time with Requested address ... not found!.

Expected behavior

On hardware with EGM/Fabric-backed host memory support: GPU HBM -> remote CPU memory can transfer through MNNVL.
Otherwise: Mooncake deterministically falls back to a compatible transport or reports the incompatibility during registration.

#2914 proposes a Store-managed EGM pool. This request is specifically about Transfer Engine support for application-registered destination buffers, which is needed by SGLang HiSparse's direct-to-host KV-transfer path.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @thincal on GitHub (Jul 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3031 ### Describe your feature request ## Problem When use Mooncake Transfer Engine for Prefill/Decode KV transfer on GB200/GB300 NVL systems. SGLang HiSparse's direct-to-host path registers a pinned CPU KV pool on the Decode side as the transfer destination. This path works over RDMA, but fails when the target segment uses MNNVL/NVLink: ``` ... nvlink_transport.cpp:497] Requested address ... not found! ``` Observed with Mooncake 0.3.11. From the current transport behavior: - `NvlinkTransport::registerLocalMemory()` publishes remote-accessible metadata for CUDA allocations backed by `cuMemCreate` / Fabric handles, but a pinned CPU buffer is not exported as a remote NVLink target. - When the target segment protocol is `nvlink`, transport selection stays on NVLink and does not fall back to RDMA for a host-memory destination. - Registration may appear to succeed, while the transfer fails later in `relocateSharedMemoryAddress()` because the destination range is absent from the remote metadata. As a result, CPU-backed KV-cache paths such as HiSparse cannot use the inter-node NVLink fabric on GB systems. They must use RDMA or add GPU staging, so the GB NVLink interconnect cannot be fully utilized for this workload. ## Requested capability Please support remote CPU-memory destinations with MNNVL/NVLink. Possible designs include: 1. Provide a Mooncake allocation/registration API for CUDA VMM `HOST_NUMA` / EGM memory backed by Fabric-exportable handles, so frameworks such as SGLang can use it as their CPU KV pool. 2. Alternatively, support capability-aware dual registration/routing: use MNNVL for HBM destinations and automatically route CPU destinations through RDMA. 3. Detect unsupported pinned-host registrations early and return a clear error, instead of failing at transfer time with `Requested address ... not found!`. ## Expected behavior - On hardware with EGM/Fabric-backed host memory support: GPU HBM -> remote CPU memory can transfer through MNNVL. - Otherwise: Mooncake deterministically falls back to a compatible transport or reports the incompatibility during registration. ## Related issue #2914 proposes a Store-managed EGM pool. This request is specifically about Transfer Engine support for application-registered destination buffers, which is needed by SGLang HiSparse's direct-to-host KV-transfer path. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:19:26 +00:00

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 21, 2026):

Thanks for opening this issue, @thincal!

Field	Value
Issue	#3031
GitHub user ID	`1206509`
Reporter	@thincal

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 21, 2026): Thanks for opening this issue, @thincal! | Field | Value | |-------|-------| | **Issue** | #3031 | | **GitHub user ID** | `1206509` | | **Reporter** | @thincal | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@alogfans commented on GitHub (Jul 23, 2026):

This requires staging/proxy mechanism. You can try to use TENT, which supports host-to-host transfer via MNNVL/NVLink transparently.

@alogfans commented on GitHub (Jul 23, 2026): This requires staging/proxy mechanism. You can try to use TENT, which supports host-to-host transfer via MNNVL/NVLink transparently.

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@thincal commented on GitHub (Jul 25, 2026):

This requires staging/proxy mechanism. You can try to use TENT, which supports host-to-host transfer via MNNVL/NVLink transparently.

@alogfans thanks for your information, is there any official released whl package for TENT ?

@thincal commented on GitHub (Jul 25, 2026): > This requires staging/proxy mechanism. You can try to use TENT, which supports host-to-host transfer via MNNVL/NVLink transparently. @alogfans thanks for your information, is there any official released whl package for TENT ?

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@thincal commented on GitHub (Jul 26, 2026):

TENT, which supports host-to-host transfer via MNNVL/NVLink

@alogfans But does it also support gpu-to-host transfer via. MNNVL ?

@thincal commented on GitHub (Jul 26, 2026): > TENT, which supports host-to-host transfer via MNNVL/NVLink @alogfans But does it also support gpu-to-host transfer via. MNNVL ?

bsakaguchi referenced this issue

2026-10-29 21:30:35 +00:00

[PR #779] [MERGED] [TE] Fix notifs problems in C wrapper #1816