kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3093] [RFC]: TENT-Native UB/URMA Transport #793

New issue

Open

opened 2026-10-29 21:19:37 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:19:37 +00:00

Owner

Originally created by @zchuango on GitHub (Jul 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3093

Changes proposed

Summary

This RFC proposes adding a TENT-native UB transport backed by the openEuler UMDK URMA APIs.

The goal is to make UB a first-class TENT transport whose runtime-facing interfaces and architectural responsibilities are aligned with the existing TENT RDMA transport.

The intended data path is:

TENT Request
    → UbTask / UbSlice
    → TENT UB Workers
    → UbContext / UbEndpoint / UbJfc
    → URMA Adapter
    → liburma

TENT owns task scheduling, slicing, endpoint lifecycle, telemetry, cancellation, and failure recovery. The URMA adapter is limited to native resource management and URMA API translation.

The initial hardware scope is host DRAM-to-DRAM transfer. Additional memory types should be enabled only after their registration, topology, and access semantics have been validated.

UB/URMA support was previously introduced for the classic Mooncake Transfer Engine in #1773, #1805, and #1855. That work established the underlying URMA device, memory-registration, endpoint, build, mock, and testing capabilities.

This RFC builds on that foundation but focuses on a different architectural scope: implementing UB as a TENT-native transport with TENT-owned tasks, slices, workers, endpoint lifecycle, scheduling signals, cancellation, and failure recovery.

Motivation

TENT provides transport-independent runtime capabilities such as:

transport selection;
task and slice lifecycle management;
admission control and priority scheduling;
cancellation;
receiver-side credits;
runtime bandwidth estimation;
path failover and health monitoring.

To participate fully in this model, UB should expose the same logical capabilities as TENT RDMA.

A native implementation allows the TENT scheduler to observe UB runtime information such as:

UB device availability;
Jetty and remote-path health;
JFC completion pressure;
outstanding requests and inflight bytes;
estimated bandwidth;
endpoint generations;
timeout and error statistics.

Proposed Architecture

flowchart TD
    APP[Application / Store] --> RT[TENT Runtime]

    RT --> AQ[Admission Queue]
    AQ --> TS[Transport Selector]
    TS --> RC[Receiver Credit Gate]

    RC --> UB[UbTransport]
    UB --> WK[UbTask / UbSlice / Workers]

    WK --> PS[Device and Path Selection]
    PS --> CTX[UbContext]
    PS --> ES[EndpointStore]

    ES --> EP[UbEndpoint]
    CTX --> JFC[UbJfc]

    CTX --> UA[URMA Adapter]
    EP --> UA
    JFC --> UA
    UA --> LIB[liburma / UMDK]

    JFC --> TM[Telemetry and Health]
    TM --> WK
    TM --> AQ

The main architectural boundary is:

TENT owns scheduling and policy. The URMA adapter owns only native URMA operations and resource lifetime.

Alignment with TENT RDMA

The UB implementation should follow the same logical decomposition and public semantics as TENT RDMA.

TENT RDMA	Proposed TENT UB
`RdmaTransport`	`UbTransport`
`RdmaSubBatch`	`UbSubBatch`
`RdmaTask`	`UbTask`
`RdmaSlice`	`UbSlice`
`RdmaContext`	`UbContext`
`RdmaCQ`	`UbJfc`
`RdmaEndPoint`	`UbEndpoint`
QP	Jetty
CQ	JFC
MR	URMA target segment
GID	EID
`ibv_post_send()`	`urma_post_jetty_send_wr()`
`ibv_poll_cq()`	`urma_poll_jfc()`

The TENT-facing interfaces and state semantics should remain consistent, while native URMA terminology should be preserved inside the transport.

TENT Transport Interface

UbTransport should directly implement the common TENT transport operations:

install()
uninstall()

allocateSubBatch()
freeSubBatch()

submitTransferTasks()
getTransferStatus()

addMemoryBuffer()
removeMemoryBuffer()

getEstimatedBandwidth()

supportsCancellation()
cancelTransferTask()

Notification support may remain disabled initially until a native UB notification path is implemented.

TENT runtime code should not directly depend on URMA handles such as:

urma_jetty_t
urma_jfc_t
urma_target_seg_t

Proposed File Layout

mooncake-transfer-engine/tent/
├── include/tent/transport/ub/
│   ├── ub_transport.h
│   ├── context.h
│   ├── jfc.h
│   ├── endpoint.h
│   ├── endpoint_store.h
│   ├── slice.h
│   ├── workers.h
│   ├── buffers.h
│   ├── rail_monitor.h
│   ├── quota.h
│   ├── params.h
│   └── urma_adapter.h
│
└── src/transport/ub/
    ├── CMakeLists.txt
    ├── ub_transport.cpp
    ├── context.cpp
    ├── jfc.cpp
    ├── endpoint.cpp
    ├── endpoint_store.cpp
    ├── workers.cpp
    ├── buffers.cpp
    ├── rail_monitor.cpp
    ├── quota.cpp
    └── urma_adapter.cpp

The first implementation does not need to complete every optional scheduling feature, but the module boundaries should support later extensions without redesigning the data path.

URMA Adapter Boundary

The URMA adapter should only expose low-level operations:

URMA initialization and shutdown;
device and EID discovery;
context creation and destruction;
asynchronous event FD exposure;
local segment registration;
remote segment import;
JFC/JFCE creation and polling;
Jetty creation, import, bind, reset, quiesce, and destruction;
READ/WRITE work-request submission.

It should not contain:

TENT task scheduling;
slice spraying;
receiver-credit policy;
transport selection;
retry policy;
admission control;
rail-health decisions.

Full URMA asynchronous-event consumption and acknowledgement may be added later if required by the provider and recovery model.

Endpoint Lifecycle

At the conceptual level, UB endpoints should follow the same forward-only lifecycle semantics as TENT RDMA:

stateDiagram-v2
    [*] --> UNINIT
    UNINIT --> HANDSHAKING
    HANDSHAKING --> READY
    HANDSHAKING --> DESTROYING
    READY --> DESTROYING
    DESTROYING --> DESTROYED
    DESTROYED --> [*]

The implementation may use additional internal states such as:

PREPARED
BINDING
FAILED

Important requirements:

lifecycle states only move forward;
failed endpoint incarnations are retired rather than reconnected;
each reconstruction creates a new generation;
slices retain the endpoint generation used for submission;
stale completions cannot update a newer endpoint;
already-posted operations must be fenced or drained before native resources are destroyed;
destruction and cleanup should be retryable when the provider reports a temporary busy or failure condition.

Topology and Path Selection

UB should be represented as an independent TENT transport and topology type:

TransportType::UB
Topology::NIC_UB

UB devices should not be represented as RDMA NICs.

Where available, topology information should include:

native device name;
stable TENT-facing device identity;
device and EID indices;
EID;
NUMA node;
PCI location;
active state.

The initial path-selection model may consider:

topology locality
local-device availability
remote-path availability
device and endpoint inflight pressure
transport quota
endpoint readiness
runtime bandwidth samples
retry history

This allows TENT to evaluate UB paths using the same logical model used for RDMA without exposing Jetty or JFC internals to the runtime.

Native Control-Plane Bootstrap

UB should use a dedicated TENT bootstrap description rather than reusing RDMA QP fields.

The bootstrap message should include:

protocol version
segment identity
local and peer NIC paths
local UB device identity
EID
Jetty IDs
endpoint generation
segment generation
capabilities

sequenceDiagram
    participant W as UB Workers
    participant S as EndpointStore
    participant E as Local Endpoint
    participant C as TENT Control Plane
    participant P as Peer UB Transport
    participant U as URMA Adapter

    W->>S: getOrCreate(path)
    S->>E: create endpoint generation N
    E->>U: create local Jetty set
    E->>C: bootstrap(EID, Jetty IDs, generation)
    C->>P: invoke UB bootstrap handler
    P->>U: create and bind peer Jetty set
    P-->>C: peer EID and Jetty IDs
    C-->>E: bootstrap response
    E->>U: import and bind remote Jettys
    E-->>S: endpoint READY

Scheduling and Runtime Signals

The native implementation should expose sufficient runtime information for TENT scheduling:

per-device inflight bytes
per-endpoint outstanding operations
completion latency
EWMA bandwidth
timeout count
completion error count
endpoint reconstruction count
local device state
remote path state

The initial implementation may use a deterministic rule-based score. More advanced QoS or deadline-aware arbitration can be added incrementally.

Failure and Cancellation Semantics

The implementation should distinguish at least:

local device error
remote path error
endpoint error
memory registration or import error
timeout
cancellation before post
unknown completion error

Suggested behavior:

local-device failures retry through another local UB context;
remote-path failures pause the path and try another remote device;
endpoint failures retire the current generation;
remote-segment generation changes trigger a new import;
timeout retires the affected endpoint and initiates safe reconstruction;
invalid address, permission, or range errors fail without retry.

Cancellation should follow the existing TENT best-effort model:

unposted slices should not reach the device;
queued and retrying slices may be canceled;
already-posted operations must still be completed or safely fenced;
buffers must not be reused while native DMA may still access them.

Proposed Implementation Phases

Phase 1: Native skeleton

add TransportType::UB;
add Topology::NIC_UB;
add loader and CMake integration;
add UbTransport and module skeletons;
add an injectable URMA adapter.

Phase 2: Context and memory

discover URMA devices and EIDs;
initialize contexts and JFC resources;
register local segments;
import remote segments;
publish TENT-native UB metadata.

Phase 3: Endpoint bootstrap

add the UB control-plane message;
implement Jetty bootstrap;
implement EndpointStore;
implement endpoint lifecycle and generations.

Phase 4: READ/WRITE data path

implement UB tasks and slices;
implement posting and polling workers;
post URMA READ/WRITE operations;
dispatch JFC completions;
expose task status.

Phase 5: Reliability and scheduling

cancellation;
error classification;
path retry and failover;
timeout and endpoint reconstruction;
runtime bandwidth estimation;
admission-queue and receiver-credit integration.

Each phase should remain independently buildable, testable, and reviewable.

Validation

CI should use an injectable/mock URMA adapter to validate deterministic behavior without UB hardware.

Real URMA hardware validation remains a separate acceptance requirement before the transport can be considered production-ready.

Hardware validation should cover:

two-node READ and WRITE correctness;
mixed-operation consistency checking;
multiple UB devices and paths;
endpoint reuse and generation isolation;
peer restart;
remote segment re-registration;
UB port down/up;
timeout and cancellation;
safe repeated shutdown;
performance comparison against the existing UB implementation.

The architecture is designed to support multi-path scheduling and recovery, but those capabilities should not be considered validated until the real-hardware test matrix passes.

Non-Goals

The initial work does not need to:

replace existing transport implementations;
support GPU or other non-DRAM memory immediately;
implement every TENT QoS capability;
implement native UB notifications in the first version;
rename URMA concepts using RDMA terminology;
place scheduling logic inside the URMA adapter;
redesign unrelated TENT runtime APIs.

Feedback on the module boundaries, bootstrap protocol, metadata format, PR decomposition, and hardware validation plan would be appreciated

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @zchuango on GitHub (Jul 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3093 ### Changes proposed ## Summary This RFC proposes adding a **TENT-native UB transport** backed by the openEuler UMDK URMA APIs. The goal is to make UB a first-class TENT transport whose runtime-facing interfaces and architectural responsibilities are aligned with the existing TENT RDMA transport. The intended data path is: ```text TENT Request → UbTask / UbSlice → TENT UB Workers → UbContext / UbEndpoint / UbJfc → URMA Adapter → liburma ``` TENT owns task scheduling, slicing, endpoint lifecycle, telemetry, cancellation, and failure recovery. The URMA adapter is limited to native resource management and URMA API translation. The initial hardware scope is **host DRAM-to-DRAM transfer**. Additional memory types should be enabled only after their registration, topology, and access semantics have been validated. --- ## Related Work UB/URMA support was previously introduced for the classic Mooncake Transfer Engine in #1773, #1805, and #1855. That work established the underlying URMA device, memory-registration, endpoint, build, mock, and testing capabilities. This RFC builds on that foundation but focuses on a different architectural scope: implementing UB as a TENT-native transport with TENT-owned tasks, slices, workers, endpoint lifecycle, scheduling signals, cancellation, and failure recovery. --- ## Motivation TENT provides transport-independent runtime capabilities such as: - transport selection; - task and slice lifecycle management; - admission control and priority scheduling; - cancellation; - receiver-side credits; - runtime bandwidth estimation; - path failover and health monitoring. To participate fully in this model, UB should expose the same logical capabilities as TENT RDMA. A native implementation allows the TENT scheduler to observe UB runtime information such as: - UB device availability; - Jetty and remote-path health; - JFC completion pressure; - outstanding requests and inflight bytes; - estimated bandwidth; - endpoint generations; - timeout and error statistics. --- ## Proposed Architecture ```mermaid flowchart TD APP[Application / Store] --> RT[TENT Runtime] RT --> AQ[Admission Queue] AQ --> TS[Transport Selector] TS --> RC[Receiver Credit Gate] RC --> UB[UbTransport] UB --> WK[UbTask / UbSlice / Workers] WK --> PS[Device and Path Selection] PS --> CTX[UbContext] PS --> ES[EndpointStore] ES --> EP[UbEndpoint] CTX --> JFC[UbJfc] CTX --> UA[URMA Adapter] EP --> UA JFC --> UA UA --> LIB[liburma / UMDK] JFC --> TM[Telemetry and Health] TM --> WK TM --> AQ ``` The main architectural boundary is: > TENT owns scheduling and policy. The URMA adapter owns only native URMA operations and resource lifetime. --- ## Alignment with TENT RDMA The UB implementation should follow the same logical decomposition and public semantics as TENT RDMA. | TENT RDMA | Proposed TENT UB | | --- | --- | | `RdmaTransport` | `UbTransport` | | `RdmaSubBatch` | `UbSubBatch` | | `RdmaTask` | `UbTask` | | `RdmaSlice` | `UbSlice` | | `RdmaContext` | `UbContext` | | `RdmaCQ` | `UbJfc` | | `RdmaEndPoint` | `UbEndpoint` | | QP | Jetty | | CQ | JFC | | MR | URMA target segment | | GID | EID | | `ibv_post_send()` | `urma_post_jetty_send_wr()` | | `ibv_poll_cq()` | `urma_poll_jfc()` | The TENT-facing interfaces and state semantics should remain consistent, while native URMA terminology should be preserved inside the transport. --- ## TENT Transport Interface `UbTransport` should directly implement the common TENT transport operations: ```text install() uninstall() allocateSubBatch() freeSubBatch() submitTransferTasks() getTransferStatus() addMemoryBuffer() removeMemoryBuffer() getEstimatedBandwidth() supportsCancellation() cancelTransferTask() ``` Notification support may remain disabled initially until a native UB notification path is implemented. TENT runtime code should not directly depend on URMA handles such as: ```text urma_jetty_t urma_jfc_t urma_target_seg_t ``` --- ## Proposed File Layout ```text mooncake-transfer-engine/tent/ ├── include/tent/transport/ub/ │ ├── ub_transport.h │ ├── context.h │ ├── jfc.h │ ├── endpoint.h │ ├── endpoint_store.h │ ├── slice.h │ ├── workers.h │ ├── buffers.h │ ├── rail_monitor.h │ ├── quota.h │ ├── params.h │ └── urma_adapter.h │ └── src/transport/ub/ ├── CMakeLists.txt ├── ub_transport.cpp ├── context.cpp ├── jfc.cpp ├── endpoint.cpp ├── endpoint_store.cpp ├── workers.cpp ├── buffers.cpp ├── rail_monitor.cpp ├── quota.cpp └── urma_adapter.cpp ``` The first implementation does not need to complete every optional scheduling feature, but the module boundaries should support later extensions without redesigning the data path. --- ## URMA Adapter Boundary The URMA adapter should only expose low-level operations: - URMA initialization and shutdown; - device and EID discovery; - context creation and destruction; - asynchronous event FD exposure; - local segment registration; - remote segment import; - JFC/JFCE creation and polling; - Jetty creation, import, bind, reset, quiesce, and destruction; - READ/WRITE work-request submission. It should not contain: - TENT task scheduling; - slice spraying; - receiver-credit policy; - transport selection; - retry policy; - admission control; - rail-health decisions. Full URMA asynchronous-event consumption and acknowledgement may be added later if required by the provider and recovery model. --- ## Endpoint Lifecycle At the conceptual level, UB endpoints should follow the same forward-only lifecycle semantics as TENT RDMA: ```mermaid stateDiagram-v2 [*] --> UNINIT UNINIT --> HANDSHAKING HANDSHAKING --> READY HANDSHAKING --> DESTROYING READY --> DESTROYING DESTROYING --> DESTROYED DESTROYED --> [*] ``` The implementation may use additional internal states such as: ```text PREPARED BINDING FAILED ``` Important requirements: - lifecycle states only move forward; - failed endpoint incarnations are retired rather than reconnected; - each reconstruction creates a new generation; - slices retain the endpoint generation used for submission; - stale completions cannot update a newer endpoint; - already-posted operations must be fenced or drained before native resources are destroyed; - destruction and cleanup should be retryable when the provider reports a temporary busy or failure condition. --- ## Topology and Path Selection UB should be represented as an independent TENT transport and topology type: ```text TransportType::UB Topology::NIC_UB ``` UB devices should not be represented as RDMA NICs. Where available, topology information should include: - native device name; - stable TENT-facing device identity; - device and EID indices; - EID; - NUMA node; - PCI location; - active state. The initial path-selection model may consider: ```text topology locality local-device availability remote-path availability device and endpoint inflight pressure transport quota endpoint readiness runtime bandwidth samples retry history ``` This allows TENT to evaluate UB paths using the same logical model used for RDMA without exposing Jetty or JFC internals to the runtime. --- ## Native Control-Plane Bootstrap UB should use a dedicated TENT bootstrap description rather than reusing RDMA QP fields. The bootstrap message should include: ```text protocol version segment identity local and peer NIC paths local UB device identity EID Jetty IDs endpoint generation segment generation capabilities ``` ```mermaid sequenceDiagram participant W as UB Workers participant S as EndpointStore participant E as Local Endpoint participant C as TENT Control Plane participant P as Peer UB Transport participant U as URMA Adapter W->>S: getOrCreate(path) S->>E: create endpoint generation N E->>U: create local Jetty set E->>C: bootstrap(EID, Jetty IDs, generation) C->>P: invoke UB bootstrap handler P->>U: create and bind peer Jetty set P-->>C: peer EID and Jetty IDs C-->>E: bootstrap response E->>U: import and bind remote Jettys E-->>S: endpoint READY ``` --- ## Scheduling and Runtime Signals The native implementation should expose sufficient runtime information for TENT scheduling: ```text per-device inflight bytes per-endpoint outstanding operations completion latency EWMA bandwidth timeout count completion error count endpoint reconstruction count local device state remote path state ``` The initial implementation may use a deterministic rule-based score. More advanced QoS or deadline-aware arbitration can be added incrementally. --- ## Failure and Cancellation Semantics The implementation should distinguish at least: ```text local device error remote path error endpoint error memory registration or import error timeout cancellation before post unknown completion error ``` Suggested behavior: - local-device failures retry through another local UB context; - remote-path failures pause the path and try another remote device; - endpoint failures retire the current generation; - remote-segment generation changes trigger a new import; - timeout retires the affected endpoint and initiates safe reconstruction; - invalid address, permission, or range errors fail without retry. Cancellation should follow the existing TENT best-effort model: - unposted slices should not reach the device; - queued and retrying slices may be canceled; - already-posted operations must still be completed or safely fenced; - buffers must not be reused while native DMA may still access them. --- ## Proposed Implementation Phases ### Phase 1: Native skeleton - add `TransportType::UB`; - add `Topology::NIC_UB`; - add loader and CMake integration; - add `UbTransport` and module skeletons; - add an injectable URMA adapter. ### Phase 2: Context and memory - discover URMA devices and EIDs; - initialize contexts and JFC resources; - register local segments; - import remote segments; - publish TENT-native UB metadata. ### Phase 3: Endpoint bootstrap - add the UB control-plane message; - implement Jetty bootstrap; - implement `EndpointStore`; - implement endpoint lifecycle and generations. ### Phase 4: READ/WRITE data path - implement UB tasks and slices; - implement posting and polling workers; - post URMA READ/WRITE operations; - dispatch JFC completions; - expose task status. ### Phase 5: Reliability and scheduling - cancellation; - error classification; - path retry and failover; - timeout and endpoint reconstruction; - runtime bandwidth estimation; - admission-queue and receiver-credit integration. Each phase should remain independently buildable, testable, and reviewable. --- ## Validation CI should use an injectable/mock URMA adapter to validate deterministic behavior without UB hardware. Real URMA hardware validation remains a separate acceptance requirement before the transport can be considered production-ready. Hardware validation should cover: - two-node READ and WRITE correctness; - mixed-operation consistency checking; - multiple UB devices and paths; - endpoint reuse and generation isolation; - peer restart; - remote segment re-registration; - UB port down/up; - timeout and cancellation; - safe repeated shutdown; - performance comparison against the existing UB implementation. The architecture is designed to support multi-path scheduling and recovery, but those capabilities should not be considered validated until the real-hardware test matrix passes. --- ## Non-Goals The initial work does not need to: - replace existing transport implementations; - support GPU or other non-DRAM memory immediately; - implement every TENT QoS capability; - implement native UB notifications in the first version; - rename URMA concepts using RDMA terminology; - place scheduling logic inside the URMA adapter; - redesign unrelated TENT runtime APIs. Feedback on the module boundaries, bootstrap protocol, metadata format, PR decomposition, and hardware validation plan would be appreciated ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

stale

labels

2026-10-29 21:19:37 +00:00

bsakaguchi commented

2026-10-29 21:19:37 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 24, 2026):

Thanks for opening this issue, @zchuango!

Field	Value
Issue	#3093
GitHub user ID	`237737874`
Reporter	@zchuango

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 24, 2026): Thanks for opening this issue, @zchuango! | Field | Value | |-------|-------| | **Issue** | #3093 | | **GitHub user ID** | `237737874` | | **Reporter** | @zchuango | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:37 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi referenced this issue

2026-10-29 21:30:39 +00:00

[PR #793] [CLOSED] [TransferEngine] Introduce generic NVMeoF transport implementation (#780) #1827