kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2518] [RFC]: AscendRdmaTransport — NPU↔RDMA Protocol Bridge #667

New issue

Closed

opened 2026-10-29 21:17:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:17:53 +00:00

Owner

Originally created by @tianlang-wq on GitHub (Jun 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2518

Originally assigned to: @tianlang-wq on GitHub.

Changes proposed

Problem

A Mooncake cluster may contain both NPU (Ascend) and DCU (Hygon) compute nodes that share a pool of Mooncake-Store nodes for KVCache data.

Mooncake-Store (registered as "rdma" or "ascend")
        │
 ┌──────┴──────────┐
 ▼                 ▼
NPU Client       DCU Client
(speaks ascend)  (speaks rdma)

The store registers a single protocol to Mooncake-Metadata. This creates a split:

Store protocol	NPU client	DCU client
`"rdma"`	❌ protocol mismatch	✅ works
`"ascend"`	✅ works	❌ protocol mismatch

At the hardware level, NPU device memory cannot be directly addressed by the RDMA NIC, so a standard RdmaTransport cannot transfer NPU data. Conversely, the AscendDirectTransport uses Ascend HCCL which DCU does not support.

Design: AscendRdmaTransport

AscendRdmaTransport is a protocol bridge installed on the NPU client side. It wraps an inner RdmaTransport and translates between NPU device memory and CPU-registered bounce buffers using Ascend ACL APIs.

Architecture

NPU Client Node
┌──────────────────────────────────────┐
│  AscendRdmaTransport                  │
│  ┌────────────────────────────┐      │
│  │  RdmaTransport (inner)      │──────┼── RDMA network ──► Store (rdma)
│  └───────────┬────────────────┘      │
│              │ bounce buffer          │
│              ▼                        │
│   ┌────────────────────┐             │
│   │  kHostBufferSize    │ (CPU mem,  │
│   │   = 3 GB            │  RDMA reg) │
│   └────────────────────┘             │
│              │                        │
│    aclrtMemcpyAsync                   │
│    (D2H / H2D)                        │
│              │                        │
│              ▼                        │
│   ┌────────────────────┐             │
│   │  NPU Device Memory  │             │
│   └────────────────────┘             │
└──────────────────────────────────────┘

Flow: `selectTransport()` remap

In MultiTransport::selectTransport() (multi_transport.cpp:410), when USE_ASCEND_RDMA is enabled:

if (target_segment_desc->protocol == "rdma") {
    proto = "ascend_rdma";
}

The store always registers with "rdma" (native RDMA). On the NPU initiator side, the protocol is transparently remapped to "ascend_rdma", so AscendRdmaTransport handles the transfer. If the initiator is a DCU node without USE_ASCEND_RDMA, the protocol stays as "rdma" and a standard RdmaTransport is used.

Transfer Paths

WRITE (NPU memory → remote Store via RDMA):

NPU mem → D2H copy → bounce buffer → RDMA WRITE → remote Store

READ (remote Store → NPU memory via RDMA):

remote Store → RDMA READ → bounce buffer → H2D copy → NPU mem

Two code paths handle these:

Path	Condition	Behavior
Large (`handleLargeTransfer`, line 382)	Any READ, or WRITE ≥ 2 MB	Per-request D2H copy + direct RDMA submit
Small (`handleSmallTransfer`, line 456)	WRITE < 2 MB	Aggregate multiple requests into NPU block → background thread does D2H + RDMA submit

Lifecycle

install() — Allocates a 3 GB CPU bounce buffer (host_addr_), registers it with RDMA, pre-allocates NPU device blocks (dev_addr_), and starts a background transferLoop thread.
submitTransferTask() — Delegates to handleLargeTransfer or handleSmallTransfer based on opcode/size.
getTransferStatus() — On completion, calls completeReadBacklog() to perform the final H2D copy for READ operations.
registerLocalMemory() — CPU memory is forwarded to RdmaTransport. NPU memory is recorded in metadata (RDMA cannot access it directly).
Destructor — Joins the background thread and frees ACL resources.

Component Registration

AscendRdmaTransport registers its local segment with protocol "ascend_rdma":

desc->protocol = "ascend_rdma";

This is a local-only protocol descriptor — the segment protocol exposed to remote nodes remains "rdma" (set by the inner RdmaTransport's own segment registration). Remote peers see "rdma" and the remap only happens on the initiator side that has USE_ASCEND_RDMA enabled.

Summary

AscendRdmaTransport enables heterogeneous NPU+DCU clusters by:

Stores register with "rdma" — DCU nodes connect directly; NPU nodes transparently bridge via AscendRdmaTransport.
Bounce buffer pattern — NPU device memory is copied to/from a CPU buffer that RDMA can access.
No store changes — The bridging is entirely on the initiator side. Stores remain pure RDMA targets.
Coexistence — When both USE_ASCEND_DIRECT (Ascend-native, for NPU↔NPU) and USE_ASCEND_RDMA (Ascend↔RDMA bridge) are enabled, the system selects the optimal transport per transfer.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @tianlang-wq on GitHub (Jun 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2518 Originally assigned to: @tianlang-wq on GitHub. ### Changes proposed ## Problem A Mooncake cluster may contain both **NPU (Ascend)** and **DCU (Hygon)** compute nodes that share a pool of Mooncake-Store nodes for KVCache data. ``` Mooncake-Store (registered as "rdma" or "ascend") │ ┌──────┴──────────┐ ▼ ▼ NPU Client DCU Client (speaks ascend) (speaks rdma) ``` The store registers a **single protocol** to Mooncake-Metadata. This creates a split: | Store protocol | NPU client | DCU client | |---------------|-----------|-----------| | `"rdma"` | ❌ protocol mismatch | ✅ works | | `"ascend"` | ✅ works | ❌ protocol mismatch | At the hardware level, NPU device memory cannot be directly addressed by the RDMA NIC, so a standard `RdmaTransport` cannot transfer NPU data. Conversely, the `AscendDirectTransport` uses Ascend HCCL which DCU does not support. ## Design: AscendRdmaTransport `AscendRdmaTransport` is a **protocol bridge** installed on the NPU client side. It wraps an inner `RdmaTransport` and translates between NPU device memory and CPU-registered bounce buffers using Ascend ACL APIs. ### Architecture ``` NPU Client Node ┌──────────────────────────────────────┐ │ AscendRdmaTransport │ │ ┌────────────────────────────┐ │ │ │ RdmaTransport (inner) │──────┼── RDMA network ──► Store (rdma) │ └───────────┬────────────────┘ │ │ │ bounce buffer │ │ ▼ │ │ ┌────────────────────┐ │ │ │ kHostBufferSize │ (CPU mem, │ │ │ = 3 GB │ RDMA reg) │ │ └────────────────────┘ │ │ │ │ │ aclrtMemcpyAsync │ │ (D2H / H2D) │ │ │ │ │ ▼ │ │ ┌────────────────────┐ │ │ │ NPU Device Memory │ │ │ └────────────────────┘ │ └──────────────────────────────────────┘ ``` ### Flow: `selectTransport()` remap In `MultiTransport::selectTransport()` (`multi_transport.cpp:410`), when `USE_ASCEND_RDMA` is enabled: ```cpp if (target_segment_desc->protocol == "rdma") { proto = "ascend_rdma"; } ``` The **store** always registers with `"rdma"` (native RDMA). On the NPU initiator side, the protocol is transparently remapped to `"ascend_rdma"`, so `AscendRdmaTransport` handles the transfer. If the initiator is a DCU node without `USE_ASCEND_RDMA`, the protocol stays as `"rdma"` and a standard `RdmaTransport` is used. ### Transfer Paths **WRITE (NPU memory → remote Store via RDMA):** ``` NPU mem → D2H copy → bounce buffer → RDMA WRITE → remote Store ``` **READ (remote Store → NPU memory via RDMA):** ``` remote Store → RDMA READ → bounce buffer → H2D copy → NPU mem ``` Two code paths handle these: | Path | Condition | Behavior | |------|-----------|----------| | **Large** (`handleLargeTransfer`, line 382) | Any READ, or WRITE ≥ 2 MB | Per-request D2H copy + direct RDMA submit | | **Small** (`handleSmallTransfer`, line 456) | WRITE < 2 MB | Aggregate multiple requests into NPU block → background thread does D2H + RDMA submit | ### Lifecycle 1. **`install()`** — Allocates a 3 GB CPU bounce buffer (`host_addr_`), registers it with RDMA, pre-allocates NPU device blocks (`dev_addr_`), and starts a background `transferLoop` thread. 2. **`submitTransferTask()`** — Delegates to `handleLargeTransfer` or `handleSmallTransfer` based on opcode/size. 3. **`getTransferStatus()`** — On completion, calls `completeReadBacklog()` to perform the final H2D copy for READ operations. 4. **`registerLocalMemory()`** — CPU memory is forwarded to `RdmaTransport`. NPU memory is recorded in metadata (RDMA cannot access it directly). 5. **Destructor** — Joins the background thread and frees ACL resources. ### Component Registration `AscendRdmaTransport` registers its local segment with protocol `"ascend_rdma"`: ```cpp desc->protocol = "ascend_rdma"; ``` This is a **local-only** protocol descriptor — the segment protocol exposed to remote nodes remains `"rdma"` (set by the inner `RdmaTransport`'s own segment registration). Remote peers see `"rdma"` and the remap only happens on the initiator side that has `USE_ASCEND_RDMA` enabled. ## Summary `AscendRdmaTransport` enables heterogeneous NPU+DCU clusters by: - **Stores register with `"rdma"`** — DCU nodes connect directly; NPU nodes transparently bridge via `AscendRdmaTransport`. - **Bounce buffer pattern** — NPU device memory is copied to/from a CPU buffer that RDMA can access. - **No store changes** — The bridging is entirely on the initiator side. Stores remain pure RDMA targets. - **Coexistence** — When both `USE_ASCEND_DIRECT` (Ascend-native, for NPU↔NPU) and `USE_ASCEND_RDMA` (Ascend↔RDMA bridge) are enabled, the system selects the optimal transport per transfer. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:17:53 +00:00

bsakaguchi commented

2026-10-29 21:17:54 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 17, 2026):

Thanks for opening this issue, @tianlang-wq!

Field	Value
Issue	#2518
GitHub user ID	`225992489`
Reporter	@tianlang-wq

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 17, 2026): Thanks for opening this issue, @tianlang-wq! | Field | Value | |-------|-------| | **Issue** | #2518 | | **GitHub user ID** | `225992489` | | **Reporter** | @tianlang-wq | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:29:53 +00:00

[PR #667] [MERGED] chore: bump version to 0.3.5 in pyproject.toml #1730