kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3322] [RFC]: MPComm Transport for TENT — an RDMA-native multi-rail transport backend #843

New issue

Open

opened 2026-10-29 21:20:06 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:20:06 +00:00

Owner

Originally created by @c-guo16 on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3322

Changes proposed

Summary

This RFC proposes adding MPComm as a TENT transport backend, registered as TransportType::MPCOMM.

MPComm (Memory Pool Communication) is an RDMA-native data-plane library for large-scale
heterogeneous memory pools, open-sourced by the Tencent Astral Network Team under Apache-2.0:
https://github.com/Tencent/UCL-MPComm. It drives multiple RDMA NICs concurrently with two-level
load balancing (across NICs, and across QPs within a NIC) and NUMA-aware worker placement, exposing
one-sided put/get primitives.

Data path:

TENT Request (WRITE/READ)
  -> MpcommTransport::submitTransferTasks()
  -> MPComm putAsync() / getAsync()        (slicing, NIC/QP selection, workers are internal to MPComm)
  -> libmpcomm -> libibverbs

Validated scope is host DRAM and GPU device memory over RoCE/IB, in both directions and in
mixed combinations. GPU memory is registered through nvidia-peermem, the same mechanism
RdmaTransport depends on for GPU-Direct. The implementation is complete and validated; this RFC
exists because the change exceeds the 500-LOC threshold in CONTRIBUTING.md
("For major architectural changes (>500 LOC excluding tests), we would expect a GitHub issue (RFC)").

Related roadmap: #1058 (Transfer Engine NEXT).

Motivation

The obvious question is why a second RDMA transport when RdmaTransport already exists. Three
reasons:

1. Measured throughput advantage over the existing TENT RDMA path.
Numbers published in the MPComm repository (2 nodes, single NUMA, 4x CX-7, dual-plane 2x200G bond
per NIC, batch=4, 1GB block):

Path	Platform	MPComm PUT/GET	Baseline PUT/GET	Ratio
Zero-copy	AMD Turin / RTX PRO 5000	195.7 / 195.7 GB/s	Mooncake TENT 142.2 / 142.3	~1.38x
Zero-copy	AMD Genoa / NVIDIA H20	165.7 / 152.3 GB/s	Mooncake TENT 122.0 / 111.7	~1.36x
Non-zero-copy	AMD Turin / RTX PRO 5000	91.9 / 84.5 GB/s	UCX 20.9 / 16.7	up to ~5x

The gain comes from aggressive multi-rail striping plus per-QP outstanding-request tracking, i.e.
from scheduling policy inside a purpose-built library, not from a different verbs API.

2. It gives TENT a second, independently-developed RDMA implementation.
Where RdmaTransport is Mooncake's own stack, MPComm is maintained and tuned outside the project.
Having both lets deployments pick per workload, and makes it possible to cross-check TENT-level
behaviour against an independent data plane.

3. Vendor-library integration is an established pattern here.
sunrise_link, kunpeng_ub / UB (#3093, #3282), TPU and EFA are all transports that wrap an
external SDK behind the TENT Transport interface. MPComm follows the same shape.

Transport	Relationship
`RdmaTransport`	Same underlying verbs, different scheduling. MPComm is not a replacement — both stay selectable.
UB / URMA (#3093, #3282)	Same integration shape (new tent-native transport, vendor SDK, `USE_` gate, enum appended). Different division of labour* — see the trade-off section below.
`sunrise_link`	Closest precedent structurally: TENT-only, vendor SDK, absolute-path library discovery in `common.cmake`. Layout and CMake style were copied from it.

Proposed design

Architecture

flowchart TB
    App["App / Mooncake Store"] --> RT["TENT Runtime<br/>(TransferEngineImpl)"]
    RT --> SEL["transport selector<br/>(policy / transport_hint)"]
    SEL --> MT["MpcommTransport<br/>(this RFC, ~550 LOC)"]

    MT --> API["MPComm API<br/>init / registerMemory / publishBuffer<br/>connect / putAsync / getAsync"]
    API --> LIB["libmpcomm"]
    LIB --> WK["MPComm internals:<br/>worker threads, slicing,<br/>NIC + QP two-level balancing"]
    WK --> V["libibverbs"]

    MT -. "publish endpoint" .-> SM["SegmentManager<br/>transport_attrs[MPCOMM]"]
    SM -. "peer lookup" .-> MT

Division of labour, and the trade-off worth discussing

This is the part I most want feedback on, because it differs from the UB RFC on purpose.

#3093 states the invariant "TENT owns scheduling and policy; the URMA adapter only performs native
operations and resource lifecycle." MPComm cannot follow that invariant: slicing, NIC selection, QP
selection, outstanding-request accounting and worker threads are inside the library, and they are
precisely where its throughput advantage comes from. Decomposing them to hand control back to TENT
would mean forking MPComm and giving up the thing we integrate it for.

So the boundary is deliberately coarse:

What the transport does

translate WRITE/READ requests into putAsync() / getAsync(), one MPComm transfer per request
derive MPComm's RDMA device list from the TENT Topology
register/deregister buffers (registerMemory() + publishBuffer())
publish and resolve endpoints through transport_attrs (see below)
poll completions and map MPComm error codes onto TENT task status

What it deliberately does not do

no slicing or spraying (MPComm's job)
no NIC / QP selection (MPComm's job)
no separate control-plane message type (reuses transport_attrs)
no re-implementation of MPComm's load balancing inside TENT

Consequence, stated plainly: TENT's transport-agnostic facilities that require fine-grained
cooperation are not available for this transport in its first version —
supportsCancellation() returns false, and there is no bandwidth estimation, no NIC load
statistics, no use of SubBatch's qp_pool or progress-notification hooks. Failover and QoS
features depending on those hooks fall back to defaults.

I think this is an acceptable trade-off for a first version, and it is documented rather than
hidden. Open question O1 below asks whether the maintainers agree, since it is a precedent
question for wrapping any self-scheduling vendor library.

Endpoint bootstrap: reusing `transport_attrs` instead of a new control-plane message

MPComm performs its own TCP metadata handshake and needs the peer's host:port. Rather than adding a
UB-style dedicated bootstrap message, the transport publishes <host>:<mpcomm_tcp_port> into the
local segment's transport_attrs under the MPCOMM key via
SegmentManager::updateLocal() + synchronizeLocal(), and peers read it back on first use.

updateLocal() (not writing through the copy-on-write getLocal() snapshot) is what makes this
correct.
Fallback when the attribute is absent: segment name + MPCOMM_REMOTE_TCP_PORT.
Connections are established lazily on first request to a segment and cached per SegmentID.

This keeps the metadata plane untouched. Worth confirming this is the intended use of
transport_attrs (open question O2).

Alignment with the TENT `Transport` interface

Implemented: install / uninstall, allocateSubBatch / freeSubBatch,
submitTransferTasks / getTransferStatus, addMemoryBuffer / removeMemoryBuffer,
getSupportedTransports registration.
Not implemented (fall back to base behaviour): cancellation, bandwidth estimation, NIC load stats,
custom allocator (allocateLocalMemory is not overridden — the transport is pass-through).

File layout

mooncake-transfer-engine/tent/
  include/tent/transport/mpcomm/mpcomm_transport.h
  src/transport/mpcomm/mpcomm_transport.cpp
  src/transport/mpcomm/CMakeLists.txt
  tests/mpcomm_transport_test.cpp
docs/source/design/transfer-engine/mpcomm_transport.md

Registration points touched: types.h (enum + name/parse maps), tent/transfer_engine.h
(C API macro), transport_loader.cpp, transfer_engine_impl.cpp (+1 line),
tent/src/CMakeLists.txt, tent/src/transport/CMakeLists.txt, pybind.cpp,
common.cmake, benchmark utils.cpp / tent_backend.cpp, plus docs.

Enum and C API stability

MPCOMM is appended at the end of TransportType, immediately after UB, keeping existing
serialized values stable — same rule #3282 followed. The C macro TRANSPORT_MPCOMM is added with
the matching value. transportTypeName() / parseTransportType() gain one line each; without them a
"transports": ["mpcomm"] policy would silently parse to UNSPEC.

Build integration

Gated by USE_MPCOMM=ON, which requires MPCOMM_ROOT. common.cmake resolves
include/mpcomm.h and lib/libmpcomm.so with find_path / find_library and fails at configure
time with the resolved paths printed — matching the EFA/CXI blocks. The library is linked by
absolute path, which also survives $<LINK_ONLY:...> when downstream targets (e.g.
mooncake_master) link transfer_engine privately. With USE_MPCOMM=OFF nothing is built,
linked or registered.

Validation

Completed on 2 nodes with RoCE (multi-NIC, dual NUMA):

Check	Result
Build with `USE_TENT=ON USE_MPCOMM=ON`, no `-Wswitch` warnings	pass
Build with `USE_MPCOMM=OFF`	pass
`tebench --backend=tent --xport_type=mpcomm`, 2 nodes, DRAM, READ	> 390 GB/s aggregate
VRAM <-> VRAM (`--seg_type=VRAM` on both sides), i.e. `gpu_to_gpu`	pass, throughput in line with single-GPU expectations
VRAM <-> DRAM (VRAM on one side only), i.e. `gpu_to_dram` / `dram_to_gpu`	pass
Payload correctness (`--check_consistency=true`)	pass
`tent_mpcomm_transport_test` (fork'd two-process WRITE+READ, byte-exact verify)	2 tests pass
`transport_attrs` publish/resolve path exercised	pass — with `MPCOMM_REMOTE_TCP_PORT` deliberately unset, the initiator still reached the peer's advertised port
pre-commit (clang-format / cmake-format / codespell) on changed files	pass

CI limitation, stated up front. The functional test needs real RDMA hardware and
libmpcomm.so, so it self-skips otherwise and cannot validate the data path in CI. UB solved the
equivalent problem with an injectable mock adapter plus a ub-mock CI job (#3282). MPComm is
consumed as a shared library through a concrete API rather than an injected interface, so the same
trick needs an interposer or a seam that does not exist today. What CI can run today is the
enum/name-mapping test (no hardware). Open question O3 asks how much more is expected.

Non-goals

Not replacing RdmaTransport; both remain selectable.
TENT only — no legacy Transfer Engine transport path, so MOONCAKE_PROTOCOL=mpcomm and
transfer_engine_bench --protocol=mpcomm are intentionally unsupported. TENT is where the
project is investing, so a legacy-path backend is not planned.
No CXL / file / non-RDMA segment types; MPComm is an RDMA data plane only.
No cancellation, QoS or bandwidth-estimation integration in this version.
No changes to metadata-plane or control-plane message formats.
No re-implementation of MPComm scheduling inside TENT.

Open questions

O1 — Is the coarse-grained delegation acceptable, i.e. a transport that owns its own
scheduling and therefore opts out of cancellation / bandwidth estimation / credit integration? This
sets a precedent for self-scheduling vendor libraries and is the main thing I would like settled
before the PR is reviewed in detail.
O2 — Is transport_attrs + updateLocal() the intended mechanism for transport-specific
endpoint advertisement, or is a dedicated bootstrap message preferred?
O3 — What level of CI coverage is expected for a transport whose data path cannot run without
vendor hardware? Is a hardware-gated self-skipping test plus a hardware-free mapping test
sufficient for the first PR?
O4 — Naming: keep the functional name mpcomm (USE_MPCOMM, tent_xport_mpcomm), consistent
with sunrise_link / kunpeng_ub?

Implementation status

The implementation is finished and validated as tabulated above:

~630 LOC of code (excluding tests and docs) across 14 files; the new transport itself is
~550 LOC, the rest is registration wiring of 1-4 lines per file
+259 LOC of tests, +342 LOC of docs (a new mpcomm_transport.md plus entries in
supported-protocols.md, build.md and the design index)
deliverable as a single PR; no phased rollout is needed given the size

I will open the PR once this RFC has been seen, rebased onto current main at that point. Feedback
on O1-O3 in particular would shape whether anything needs restructuring first.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @c-guo16 on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3322 ### Changes proposed ### Summary This RFC proposes adding **MPComm** as a TENT transport backend, registered as `TransportType::MPCOMM`. MPComm (Memory Pool Communication) is an RDMA-native data-plane library for large-scale heterogeneous memory pools, open-sourced by the Tencent Astral Network Team under Apache-2.0: <https://github.com/Tencent/UCL-MPComm>. It drives multiple RDMA NICs concurrently with two-level load balancing (across NICs, and across QPs within a NIC) and NUMA-aware worker placement, exposing one-sided `put`/`get` primitives. Data path: ``` TENT Request (WRITE/READ) -> MpcommTransport::submitTransferTasks() -> MPComm putAsync() / getAsync() (slicing, NIC/QP selection, workers are internal to MPComm) -> libmpcomm -> libibverbs ``` Validated scope is **host DRAM and GPU device memory over RoCE/IB**, in both directions and in mixed combinations. GPU memory is registered through `nvidia-peermem`, the same mechanism `RdmaTransport` depends on for GPU-Direct. The implementation is complete and validated; this RFC exists because the change exceeds the 500-LOC threshold in `CONTRIBUTING.md` ("For major architectural changes (>500 LOC excluding tests), we would expect a GitHub issue (RFC)"). Related roadmap: #1058 (Transfer Engine NEXT). ### Motivation The obvious question is *why a second RDMA transport when `RdmaTransport` already exists*. Three reasons: **1. Measured throughput advantage over the existing TENT RDMA path.** Numbers published in the MPComm repository (2 nodes, single NUMA, 4x CX-7, dual-plane 2x200G bond per NIC, batch=4, 1GB block): | Path | Platform | MPComm PUT/GET | Baseline PUT/GET | Ratio | |---|---|---|---|---| | Zero-copy | AMD Turin / RTX PRO 5000 | 195.7 / 195.7 GB/s | Mooncake TENT 142.2 / 142.3 | ~1.38x | | Zero-copy | AMD Genoa / NVIDIA H20 | 165.7 / 152.3 GB/s | Mooncake TENT 122.0 / 111.7 | ~1.36x | | Non-zero-copy | AMD Turin / RTX PRO 5000 | 91.9 / 84.5 GB/s | UCX 20.9 / 16.7 | up to ~5x | The gain comes from aggressive multi-rail striping plus per-QP outstanding-request tracking, i.e. from *scheduling policy inside a purpose-built library*, not from a different verbs API. **2. It gives TENT a second, independently-developed RDMA implementation.** Where `RdmaTransport` is Mooncake's own stack, MPComm is maintained and tuned outside the project. Having both lets deployments pick per workload, and makes it possible to cross-check TENT-level behaviour against an independent data plane. **3. Vendor-library integration is an established pattern here.** `sunrise_link`, `kunpeng_ub` / UB (#3093, #3282), TPU and EFA are all transports that wrap an external SDK behind the TENT `Transport` interface. MPComm follows the same shape. ### Related work and how this differs | Transport | Relationship | |---|---| | `RdmaTransport` | Same underlying verbs, different scheduling. MPComm is not a replacement — both stay selectable. | | UB / URMA (#3093, #3282) | Same integration *shape* (new tent-native transport, vendor SDK, `USE_*` gate, enum appended). **Different division of labour** — see the trade-off section below. | | `sunrise_link` | Closest precedent structurally: TENT-only, vendor SDK, absolute-path library discovery in `common.cmake`. Layout and CMake style were copied from it. | ### Proposed design #### Architecture ```mermaid flowchart TB App["App / Mooncake Store"] --> RT["TENT Runtime (TransferEngineImpl)"] RT --> SEL["transport selector (policy / transport_hint)"] SEL --> MT["MpcommTransport (this RFC, ~550 LOC)"] MT --> API["MPComm API init / registerMemory / publishBuffer connect / putAsync / getAsync"] API --> LIB["libmpcomm"] LIB --> WK["MPComm internals: worker threads, slicing, NIC + QP two-level balancing"] WK --> V["libibverbs"] MT -. "publish endpoint" .-> SM["SegmentManager transport_attrs[MPCOMM]"] SM -. "peer lookup" .-> MT ``` #### Division of labour, and the trade-off worth discussing This is the part I most want feedback on, because it **differs from the UB RFC on purpose**. #3093 states the invariant *"TENT owns scheduling and policy; the URMA adapter only performs native operations and resource lifecycle."* MPComm cannot follow that invariant: slicing, NIC selection, QP selection, outstanding-request accounting and worker threads are **inside the library**, and they are precisely where its throughput advantage comes from. Decomposing them to hand control back to TENT would mean forking MPComm and giving up the thing we integrate it for. So the boundary is deliberately coarse: *What the transport does* - translate `WRITE`/`READ` requests into `putAsync()` / `getAsync()`, one MPComm transfer per request - derive MPComm's RDMA device list from the TENT `Topology` - register/deregister buffers (`registerMemory()` + `publishBuffer()`) - publish and resolve endpoints through `transport_attrs` (see below) - poll completions and map MPComm error codes onto TENT task status *What it deliberately does not do* - no slicing or spraying (MPComm's job) - no NIC / QP selection (MPComm's job) - no separate control-plane message type (reuses `transport_attrs`) - no re-implementation of MPComm's load balancing inside TENT **Consequence, stated plainly:** TENT's transport-agnostic facilities that require fine-grained cooperation are not available for this transport in its first version — `supportsCancellation()` returns `false`, and there is no bandwidth estimation, no NIC load statistics, no use of `SubBatch`'s `qp_pool` or progress-notification hooks. Failover and QoS features depending on those hooks fall back to defaults. I think this is an acceptable trade-off for a first version, and it is documented rather than hidden. **Open question O1 below asks whether the maintainers agree**, since it is a precedent question for wrapping any self-scheduling vendor library. #### Endpoint bootstrap: reusing `transport_attrs` instead of a new control-plane message MPComm performs its own TCP metadata handshake and needs the peer's `host:port`. Rather than adding a UB-style dedicated bootstrap message, the transport publishes `<host>:<mpcomm_tcp_port>` into the local segment's `transport_attrs` under the `MPCOMM` key via `SegmentManager::updateLocal()` + `synchronizeLocal()`, and peers read it back on first use. - `updateLocal()` (not writing through the copy-on-write `getLocal()` snapshot) is what makes this correct. - Fallback when the attribute is absent: segment name + `MPCOMM_REMOTE_TCP_PORT`. - Connections are established lazily on first request to a segment and cached per `SegmentID`. This keeps the metadata plane untouched. Worth confirming this is the intended use of `transport_attrs` (**open question O2**). #### Alignment with the TENT `Transport` interface Implemented: `install` / `uninstall`, `allocateSubBatch` / `freeSubBatch`, `submitTransferTasks` / `getTransferStatus`, `addMemoryBuffer` / `removeMemoryBuffer`, `getSupportedTransports` registration. Not implemented (fall back to base behaviour): cancellation, bandwidth estimation, NIC load stats, custom allocator (`allocateLocalMemory` is not overridden — the transport is pass-through). #### File layout ``` mooncake-transfer-engine/tent/ include/tent/transport/mpcomm/mpcomm_transport.h src/transport/mpcomm/mpcomm_transport.cpp src/transport/mpcomm/CMakeLists.txt tests/mpcomm_transport_test.cpp docs/source/design/transfer-engine/mpcomm_transport.md ``` Registration points touched: `types.h` (enum + name/parse maps), `tent/transfer_engine.h` (C API macro), `transport_loader.cpp`, `transfer_engine_impl.cpp` (+1 line), `tent/src/CMakeLists.txt`, `tent/src/transport/CMakeLists.txt`, `pybind.cpp`, `common.cmake`, benchmark `utils.cpp` / `tent_backend.cpp`, plus docs. #### Enum and C API stability `MPCOMM` is **appended at the end** of `TransportType`, immediately after `UB`, keeping existing serialized values stable — same rule #3282 followed. The C macro `TRANSPORT_MPCOMM` is added with the matching value. `transportTypeName()` / `parseTransportType()` gain one line each; without them a `"transports": ["mpcomm"]` policy would silently parse to `UNSPEC`. #### Build integration Gated by `USE_MPCOMM=ON`, which requires `MPCOMM_ROOT`. `common.cmake` resolves `include/mpcomm.h` and `lib/libmpcomm.so` with `find_path` / `find_library` and fails at configure time with the resolved paths printed — matching the EFA/CXI blocks. The library is linked **by absolute path**, which also survives `$<LINK_ONLY:...>` when downstream targets (e.g. `mooncake_master`) link `transfer_engine` privately. With `USE_MPCOMM=OFF` nothing is built, linked or registered. ### Validation Completed on 2 nodes with RoCE (multi-NIC, dual NUMA): | Check | Result | |---|---| | Build with `USE_TENT=ON USE_MPCOMM=ON`, no `-Wswitch` warnings | pass | | Build with `USE_MPCOMM=OFF` | pass | | `tebench --backend=tent --xport_type=mpcomm`, 2 nodes, DRAM, READ | **> 390 GB/s** aggregate | | VRAM <-> VRAM (`--seg_type=VRAM` on both sides), i.e. `gpu_to_gpu` | pass, throughput in line with single-GPU expectations | | VRAM <-> DRAM (VRAM on one side only), i.e. `gpu_to_dram` / `dram_to_gpu` | pass | | Payload correctness (`--check_consistency=true`) | pass | | `tent_mpcomm_transport_test` (fork'd two-process WRITE+READ, byte-exact verify) | 2 tests pass | | `transport_attrs` publish/resolve path exercised | pass — with `MPCOMM_REMOTE_TCP_PORT` deliberately unset, the initiator still reached the peer's advertised port | | pre-commit (clang-format / cmake-format / codespell) on changed files | pass | **CI limitation, stated up front.** The functional test needs real RDMA hardware *and* `libmpcomm.so`, so it self-skips otherwise and cannot validate the data path in CI. UB solved the equivalent problem with an injectable mock adapter plus a `ub-mock` CI job (#3282). MPComm is consumed as a shared library through a concrete API rather than an injected interface, so the same trick needs an interposer or a seam that does not exist today. What CI *can* run today is the enum/name-mapping test (no hardware). **Open question O3** asks how much more is expected. ### Non-goals 1. Not replacing `RdmaTransport`; both remain selectable. 2. **TENT only** — no legacy Transfer Engine transport path, so `MOONCAKE_PROTOCOL=mpcomm` and `transfer_engine_bench --protocol=mpcomm` are intentionally unsupported. TENT is where the project is investing, so a legacy-path backend is not planned. 3. No CXL / file / non-RDMA segment types; MPComm is an RDMA data plane only. 4. No cancellation, QoS or bandwidth-estimation integration in this version. 5. No changes to metadata-plane or control-plane message formats. 6. No re-implementation of MPComm scheduling inside TENT. ### Open questions - **O1** — Is the coarse-grained delegation acceptable, i.e. a transport that owns its own scheduling and therefore opts out of cancellation / bandwidth estimation / credit integration? This sets a precedent for self-scheduling vendor libraries and is the main thing I would like settled before the PR is reviewed in detail. - **O2** — Is `transport_attrs` + `updateLocal()` the intended mechanism for transport-specific endpoint advertisement, or is a dedicated bootstrap message preferred? - **O3** — What level of CI coverage is expected for a transport whose data path cannot run without vendor hardware? Is a hardware-gated self-skipping test plus a hardware-free mapping test sufficient for the first PR? - **O4** — Naming: keep the functional name `mpcomm` (`USE_MPCOMM`, `tent_xport_mpcomm`), consistent with `sunrise_link` / `kunpeng_ub`? ### Implementation status The implementation is finished and validated as tabulated above: - **~630 LOC of code** (excluding tests and docs) across 14 files; the new transport itself is ~550 LOC, the rest is registration wiring of 1-4 lines per file - **+259 LOC of tests**, **+342 LOC of docs** (a new `mpcomm_transport.md` plus entries in `supported-protocols.md`, `build.md` and the design index) - deliverable as a **single PR**; no phased rollout is needed given the size I will open the PR once this RFC has been seen, rebased onto current `main` at that point. Feedback on O1-O3 in particular would shape whether anything needs restructuring first. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:20:06 +00:00

bsakaguchi commented

2026-10-29 21:20:07 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 7, 2026):

Thanks for opening this issue, @c-guo16!

Field	Value
Issue	#3322
GitHub user ID	`35829535`
Reporter	@c-guo16

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 7, 2026): Thanks for opening this issue, @c-guo16! | Field | Value | |-------|-------| | **Issue** | #3322 | | **GitHub user ID** | `35829535` | | **Reporter** | @c-guo16 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:20:07 +00:00

Author

Owner

@alogfans commented on GitHub (Aug 7, 2026):

Looks good! Maybe you can add documentation to describe how to use it.

@alogfans commented on GitHub (Aug 7, 2026): Looks good! Maybe you can add documentation to describe how to use it.

bsakaguchi commented

2026-10-29 21:20:07 +00:00

Author

Owner

@c-guo16 commented on GitHub (Aug 10, 2026):

Looks good! Maybe you can add documentation to describe how to use it.

Thanks for the quick look!

The documentation is written and ships with the PR. Rather than paste it here, it is browsable on
my fork:

Usage doc: https://github.com/c-guo16/Mooncake/blob/mpcomm-integration-v0.3.12/docs/source/design/transfer-engine/mpcomm_transport.md
Full diff (two commits, nothing else): https://github.com/kvcache-ai/Mooncake/compare/main...c-guo16:mpcomm-integration-v0.3.12

It is deliberately usage-oriented rather than a design write-up, modelled on
sunrise_link_transport.md:

Build — USE_TENT=ON USE_MPCOMM=ON, MPCOMM_ROOT, and the configure output to expect
Enabling — separates loading the transport (config gate) from selecting it (policy or
transport_hint), and notes that one of the two is effectively required, since MPCOMM sits near
the end of the default ordering
Run and Test — complete two-sided tebench invocation, a single-host variant, and the log
lines that confirm traffic actually went over MPComm instead of silently falling back
GPU (device) memory — VRAM segments and the three preconditions, only one of which can be
detected at run time
Environment variables — 12 MPCOMM_* knobs with defaults
Important Notes / Troubleshooting — the TENT-only constraint, the rpc_server_hostname caveat
on multi-homed or containerized hosts, and seven failure modes with symptoms and fixes

It is linked from getting_started/supported-protocols.md, getting_started/build.md and the
design/transfer-engine/index.md toctree, so it is reachable from the usual entry points.

The branch above has been through an extra review and test pass since the RFC was filed, so the
Troubleshooting section covers the failure modes we actually hit rather than just the happy path.

For the record, per the issue bot's checklist: the work is based on v0.3.12.post1
(7da06d93) and validated on two nodes with RoCE (multi-NIC, dual NUMA) over DRAM and VRAM
segments, plus a single-node RDMA loopback run. It will be rebased onto current main before the
PR is opened.

Happy to expand any section. Unless you see a problem with O2 (transport_attrs +
updateLocal() for endpoint advertisement) or O3 (CI coverage for a transport whose data path
needs vendor hardware), I will open the PR with the current design and we can take the details
there.

@c-guo16 commented on GitHub (Aug 10, 2026): > Looks good! Maybe you can add documentation to describe how to use it. Thanks for the quick look! The documentation is written and ships with the PR. Rather than paste it here, it is browsable on my fork: - **Usage doc**: https://github.com/c-guo16/Mooncake/blob/mpcomm-integration-v0.3.12/docs/source/design/transfer-engine/mpcomm_transport.md - **Full diff** (two commits, nothing else): https://github.com/kvcache-ai/Mooncake/compare/main...c-guo16:mpcomm-integration-v0.3.12 It is deliberately usage-oriented rather than a design write-up, modelled on `sunrise_link_transport.md`: - **Build** — `USE_TENT=ON USE_MPCOMM=ON`, `MPCOMM_ROOT`, and the configure output to expect - **Enabling** — separates *loading* the transport (config gate) from *selecting* it (policy or `transport_hint`), and notes that one of the two is effectively required, since `MPCOMM` sits near the end of the default ordering - **Run and Test** — complete two-sided `tebench` invocation, a single-host variant, and the log lines that confirm traffic actually went over MPComm instead of silently falling back - **GPU (device) memory** — VRAM segments and the three preconditions, only one of which can be detected at run time - **Environment variables** — 12 `MPCOMM_*` knobs with defaults - **Important Notes / Troubleshooting** — the TENT-only constraint, the `rpc_server_hostname` caveat on multi-homed or containerized hosts, and seven failure modes with symptoms and fixes It is linked from `getting_started/supported-protocols.md`, `getting_started/build.md` and the `design/transfer-engine/index.md` toctree, so it is reachable from the usual entry points. The branch above has been through an extra review and test pass since the RFC was filed, so the Troubleshooting section covers the failure modes we actually hit rather than just the happy path. For the record, per the issue bot's checklist: the work is based on **v0.3.12.post1 (`7da06d93`)** and validated on two nodes with RoCE (multi-NIC, dual NUMA) over DRAM and VRAM segments, plus a single-node RDMA loopback run. It will be rebased onto current `main` before the PR is opened. Happy to expand any section. Unless you see a problem with **O2** (`transport_attrs` + `updateLocal()` for endpoint advertisement) or **O3** (CI coverage for a transport whose data path needs vendor hardware), I will open the PR with the current design and we can take the details there.

bsakaguchi commented

2026-10-29 21:20:08 +00:00

Author

Owner

@c-guo16 commented on GitHub (Aug 10, 2026):

cc @lqqqihang

@c-guo16 commented on GitHub (Aug 10, 2026): cc @lqqqihang

bsakaguchi commented

2026-10-29 21:20:08 +00:00

Author

Owner

@alogfans commented on GitHub (Aug 11, 2026):

The overall direction looks reasonable to me. My answers to the open questions would be: O1 yes, provided the missing capabilities remain explicit and policy code cannot assume them; O2 yes, transport_attrs is an appropriate place for this endpoint; O3 mapping-only CI is not sufficient for the adapter logic; O4 keep the mpcomm name.

A few points are worth addressing before opening the PR (assisted by CodeX, please recheck):

The branch is currently 84 commits behind main, where UB already occupies wire value 11. After rebasing, please keep UB=11, append MPCOMM=12, and add explicit wire-value assertions covering the C API and Python binding as well as the C++ enum.
peer_map_ is keyed by SegmentID, while MPComm connections are keyed by host_id. Closing and reopening the same segment produces a new SegmentID, so the transport may reconnect to the same MPComm host.
Remote buffer metadata is queried only on the first connection and then cached indefinitely. TENT supports registering and unregistering memory after a segment is opened, but later metadata changes never refresh MPComm's rkey cache. (***)
If connect() succeeds but queryRemoteBuffer() fails, the placeholder is removed and the next request performs another connect(). That conflicts with the implementation's own observation that reconnecting replaces the connection and leaks resources. A state such as CONNECTED_NO_KEYS would allow retrying only the buffer query.
For hardware-free CI, I suggest a thin injectable MPComm adapter. It does not need to mock MPComm scheduling; it only needs to test the TENT boundary: endpoint publication/parsing, single-flight connection, query retry, dynamic buffer refresh, READ/WRITE mapping, error and short-transfer handling, exact-once handle release, and teardown.

It would also help to version the endpoint attribute, validate the port without atoi(), provide separate listen/advertise addresses for multi-homed deployments, document the IPv4/security trust boundary, and pin or check the supported MPComm ABI/version.

@alogfans commented on GitHub (Aug 11, 2026): The overall direction looks reasonable to me. My answers to the open questions would be: O1 yes, provided the missing capabilities remain explicit and policy code cannot assume them; O2 yes, `transport_attrs` is an appropriate place for this endpoint; O3 mapping-only CI is not sufficient for the adapter logic; O4 keep the `mpcomm` name. A few points are worth addressing before opening the PR (assisted by CodeX, please recheck): 1. The branch is currently 84 commits behind `main`, where `UB` already occupies wire value 11. After rebasing, please keep `UB=11`, append `MPCOMM=12`, and add explicit wire-value assertions covering the C API and Python binding as well as the C++ enum. 2. `peer_map_` is keyed by `SegmentID`, while MPComm connections are keyed by `host_id`. Closing and reopening the same segment produces a new `SegmentID`, so the transport may reconnect to the same MPComm host. 3. Remote buffer metadata is queried only on the first connection and then cached indefinitely. TENT supports registering and unregistering memory after a segment is opened, but later metadata changes never refresh MPComm's rkey cache. (***) 4. If `connect()` succeeds but `queryRemoteBuffer()` fails, the placeholder is removed and the next request performs another `connect()`. That conflicts with the implementation's own observation that reconnecting replaces the connection and leaks resources. A state such as `CONNECTED_NO_KEYS` would allow retrying only the buffer query. 5. For hardware-free CI, I suggest a thin injectable MPComm adapter. It does not need to mock MPComm scheduling; it only needs to test the TENT boundary: endpoint publication/parsing, single-flight connection, query retry, dynamic buffer refresh, READ/WRITE mapping, error and short-transfer handling, exact-once handle release, and teardown. It would also help to version the endpoint attribute, validate the port without `atoi()`, provide separate listen/advertise addresses for multi-homed deployments, document the IPv4/security trust boundary, and pin or check the supported MPComm ABI/version.

bsakaguchi commented

2026-10-29 21:20:08 +00:00

Author

Owner

@c-guo16 commented on GitHub (Aug 11, 2026):

@alogfans Thank you for your attention!

We'll revise the code according to your comments~

@c-guo16 commented on GitHub (Aug 11, 2026): @alogfans Thank you for your attention! We'll revise the code according to your comments~

bsakaguchi commented

2026-10-29 21:20:08 +00:00

Author

Owner

@c-guo16 commented on GitHub (Aug 12, 2026):

Hello @alogfans — very useful review. All six are addressed and the branch is rebased onto main.

1. Wire values. UB keeps 11, MPCOMM appended as 12. Assertions on all three surfaces: the C++
enum and sentinel, the C API macro checked through the conversion function rather than by repeating
the constant, and a static_assert in pybind.cpp. Note your existing
WireValuesRemainStableWithUbAppended asserts kNumTransportTypes == 12, so it needed updating in
the same commit.

2. SegmentID vs host_id. Fixed — the cache is keyed by MPComm host id, so reopening a segment
reuses the existing connection.

3. Metadata refresh. Keys are refetched when the peer's buffer set grows. Comparing a fingerprint
for equality turned out to be the wrong test: descriptors are cached per thread and expire
independently, so threads observing different sets would alternately invalidate each other's work,
each time issuing a blocking query. The comparison is coverage instead — refetch only when a range
appears that the keys do not cover. That is monotonic, and it also skips a refetch after the peer
merely unregisters memory.

4. CONNECTED_NO_KEYS. Implemented under that name. A connection is recorded before the query
runs, so a failed query retries the query alone. Ownership is tracked separately from the state, so a
refresh in progress does not block requests whose ranges the current keys already cover.

5. Hardware-free CI. Added an MpcommAdapter boundary following UrmaAdapter / TpuPjrtShim,
with the peer cache and the request/completion mapping moved into their own units. Only the adapter
implementation includes mpcomm.h and it compiles to a stub without the provider, so
tent_mpcomm_boundary_test is built unconditionally and needs neither an RDMA device nor
libmpcomm — 26 cases, 0.1 s, mapping one-for-one onto the eight items you listed (case names in
tent/tests/mpcomm_boundary_test.cpp). The real data path stays in tent_mpcomm_transport_test,
which skips itself without hardware.

6. Smaller items.

endpoint attribute versioned as v1:<ipv4>:<port>; an unprefixed value reads as v1, and the check
requires v + digits + : so a hostname starting with v is not misread as a version
atoi() replaced with range-checked parsing — an invalid MPCOMM_TCP_PORT now fails install()
instead of silently becoming 0
trust boundary is its own doc section: the handshake is unauthenticated plain TCP and exchanges
addr + rkeys, i.e. everything an RDMA op needs, so that port belongs at the same trust level as
the fabric; IPv4-only is stated there
ABI: MPComm already ships a CMake package config with SOVERSION / SameMajorVersion that the
build was bypassing with a bare find_library(). Configuration now reads it and fails on a too-old
version or a different major

Not done — separate listen/advertise addresses. MPComm binds INADDR_ANY with no way to pick an
interface, so the listen side is not ours to fix; on the advertise side rpc_server_hostname steers
it today. Listed as a known limitation rather than half-implemented — happy to add a dedicated
setting if you want it in the first version.

On O1's condition. Cancellation, bandwidth estimation, NIC load statistics and notification stay
at the base-class defaults, so they report as unsupported rather than being assumed, and they are
listed in the doc. The GPU capabilities are no longer unconditional either — same nvidia_peermem
probe RdmaTransport uses, since MPComm registers device memory through plain ibv_reg_mr with no
dma-buf fallback, so a host without the module is excluded during selection instead of failing at
registration.

6 commits behind main, no conflicts. Will rebase once more and open the PR unless you see anything
else here first.

@c-guo16 commented on GitHub (Aug 12, 2026): Hello @alogfans — very useful review. All six are addressed and the branch is rebased onto `main`. - code: https://github.com/kvcache-ai/Mooncake/compare/main...c-guo16:mpcomm-integration-v0.3.12 - doc: https://github.com/c-guo16/Mooncake/blob/mpcomm-integration-v0.3.12/docs/source/design/transfer-engine/mpcomm_transport.md **1. Wire values.** `UB` keeps 11, `MPCOMM` appended as 12. Assertions on all three surfaces: the C++ enum and sentinel, the C API macro checked *through* the conversion function rather than by repeating the constant, and a `static_assert` in `pybind.cpp`. Note your existing `WireValuesRemainStableWithUbAppended` asserts `kNumTransportTypes == 12`, so it needed updating in the same commit. **2. SegmentID vs host_id.** Fixed — the cache is keyed by MPComm host id, so reopening a segment reuses the existing connection. **3. Metadata refresh.** Keys are refetched when the peer's buffer set grows. Comparing a fingerprint for equality turned out to be the wrong test: descriptors are cached per thread and expire independently, so threads observing different sets would alternately invalidate each other's work, each time issuing a blocking query. The comparison is *coverage* instead — refetch only when a range appears that the keys do not cover. That is monotonic, and it also skips a refetch after the peer merely unregisters memory. **4. `CONNECTED_NO_KEYS`.** Implemented under that name. A connection is recorded before the query runs, so a failed query retries the query alone. Ownership is tracked separately from the state, so a refresh in progress does not block requests whose ranges the current keys already cover. **5. Hardware-free CI.** Added an `MpcommAdapter` boundary following `UrmaAdapter` / `TpuPjrtShim`, with the peer cache and the request/completion mapping moved into their own units. Only the adapter implementation includes `mpcomm.h` and it compiles to a stub without the provider, so `tent_mpcomm_boundary_test` is **built unconditionally and needs neither an RDMA device nor libmpcomm** — 26 cases, 0.1 s, mapping one-for-one onto the eight items you listed (case names in `tent/tests/mpcomm_boundary_test.cpp`). The real data path stays in `tent_mpcomm_transport_test`, which skips itself without hardware. **6. Smaller items.** - endpoint attribute versioned as `v1:<ipv4>:<port>`; an unprefixed value reads as v1, and the check requires `v` + digits + `:` so a hostname starting with `v` is not misread as a version - `atoi()` replaced with range-checked parsing — an invalid `MPCOMM_TCP_PORT` now fails `install()` instead of silently becoming 0 - trust boundary is its own doc section: the handshake is unauthenticated plain TCP and exchanges `addr` + `rkeys`, i.e. everything an RDMA op needs, so that port belongs at the same trust level as the fabric; IPv4-only is stated there - ABI: MPComm already ships a CMake package config with `SOVERSION` / `SameMajorVersion` that the build was bypassing with a bare `find_library()`. Configuration now reads it and fails on a too-old version or a different major **Not done — separate listen/advertise addresses.** MPComm binds `INADDR_ANY` with no way to pick an interface, so the listen side is not ours to fix; on the advertise side `rpc_server_hostname` steers it today. Listed as a known limitation rather than half-implemented — happy to add a dedicated setting if you want it in the first version. **On O1's condition.** Cancellation, bandwidth estimation, NIC load statistics and notification stay at the base-class defaults, so they report as unsupported rather than being assumed, and they are listed in the doc. The GPU capabilities are no longer unconditional either — same `nvidia_peermem` probe `RdmaTransport` uses, since MPComm registers device memory through plain `ibv_reg_mr` with no dma-buf fallback, so a host without the module is excluded during selection instead of failing at registration. 6 commits behind `main`, no conflicts. Will rebase once more and open the PR unless you see anything else here first.

bsakaguchi referenced this issue

2026-10-29 21:30:56 +00:00

[PR #843] [MERGED] store: Add json file and improve doc #1864