kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4412] [RFC]: libfabric (EFA / CXI) transport for TENT #1113

New issue

Open

opened 2026-10-29 21:23:00 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:00 +00:00

Owner

Originally created by @whn09 on GitHub (Sep 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4412

Changes proposed

Motivation

TENT has no libfabric backend, so AWS EFA and HPE CXI users can't use it.
- TENT's rdma backend uses RC QPs (tent/src/transport/rdma/endpoint.cpp:136). EFA has no RC and is only usable through libfabric FI_EP_RDM (SRD).
- Today, MC_USE_TENT=1 on an EFA build fails at initialize() (#3995 P0-efa-ascend / P1-install-transport).
TENT discovers NICs only through ibverbs. EFA devices (rdmap*) show up as NIC_RDMA and the rdma backend tries to install on them. CXI has no verbs device, so it isn't discovered at all.
Classic TE has two libfabric transports that are mostly copies of each other.
- cxi_transport was forked from the #1944-era efa_transport (#2535). After renaming, about 80–90% of it is identical to EFA.
- The 13 EFA fixes since then were never ported. CXI still lacks the stale-AV locking (#2023), chunk-straddling slices (#3505), provider-derived WR/CQ depth (#3296) and bounded parallel registration. This is the maintenance debt tracked in #2657.

Proposal

Add one TENT transport, FABRIC, built on libfabric. Provider differences go into a small per-provider profile. The classic EFA transport as it is today is the baseline.

Provider profile

	efa	cxi	tcp;ofi_rxm (CI only)
`fi_getinfo` version	`FI_VERSION(1,18)` (#2041)	`fi_version()`	`fi_version()`
domain	`<dev>-rdm`	`<dev>`	auto
`mr_mode` hints	LOCAL, VIRT_ADDR, ALLOCATED, PROV_KEY [, HMEM]	ENDPOINT, LOCAL, ALLOCATED, PROV_KEY [, HMEM]	provider default
MR bind to EP	no	yes	from `mr_mode`
remote address	VA	offset	from `mr_mode`
MR size cap	min(`max_mr_size`, PTE budget), as in #1912	`max_mr_size`	`max_mr_size`
quirks	`FI_HMEM=system` in non-GPU builds; no `fi_av_remove` after EP close	—	—

Wherever possible, "bind to EP" and "VA vs offset" are derived from the mr_mode that fi_getinfo actually returns, not from the provider name.

Discovery

Enumerate domains with fi_getinfo and take PCI/NUMA from fi_info->nic->bus_attr. This works the same for EFA and CXI.
Add NIC_FABRIC to the TENT topology.
The rdma backend skips EFA devices, the same as #4355 does for classic TE.

Metadata (no bootstrap RPC)

RDM is connectionless.
Each NIC's fi_getname blob is stored in DeviceDesc.transport_attrs[FABRIC]. Peers call fi_av_insert lazily on first use.
Buffer keys are 64-bit, so they don't go in the uint32_t rkey vector. They are stored in BufferDesc.transport_attrs[FABRIC] as {base, chunks: [{off, len, keys[nic]}]}, which also carries the chunk layout of large registrations.
A missing peer address or key returns NeedsRefreshCache.

Data path

Same shape as the TENT rdma backend: SubBatch, slab tasks/slices, and worker threads that post and poll the CQ.
Slices are cut at chunk boundaries on both sides, then striped across NICs.
WR/CQ credits come from the provider's tx_attr->size. -FI_EAGAIN hands the unposted tail back. The credit wait is bounded by wall-clock time.
fi_cq_readerr feeds slice retry. After max_retry_count, the slice is marked FAILED and the engine's failover takes over. The RailMonitor gets markFailed / markRecovered.
The peer table read lock is held across posting (#2023).
I'd like to share the generic pieces with the rdma backend rather than copy them: RailMonitor, DeviceSelector, EndpointStore, and slice planning (keys widened to 64-bit). Feedback on that is welcome.

Memory

Chunked registration with bounded parallelism (runBoundedRegMrBatch), and a per-profile NIC assignment policy.
GPU memory is registered with fi_mr_regattr (CUDA/ROCR iface) under ScopedCudaContext (#3177).
Neuron (#3888) comes later.

Loading and build

libfabric is loaded with dlopen, using the loader from #4354, so non-fabric processes don't depend on it.
New tent_xport_fabric target behind USE_FABRIC (requires USE_TENT). It always compiles, the way MPCOMM does.

Classic → TENT shim

init(..., protocol="efa"|"cxi") pins the fabric backend with that provider and disables rdma.
Under TENT, EFA/CXI Python initialize() no longer relies on installTransport() returning non-null.
This closes the two EFA items in #3995.

Testing

CI without hardware: unit tests on the libfabric tcp;ofi_rxm provider. Covers install, loopback, write/read, chunk-straddling transfers, large chunked buffers, peer loss reported as FAILED, and metadata refresh.
EFA: p5en, two nodes, TENT bench against classic EFA bandwidth, plus Store e2e with MC_USE_TENT=1.
CXI: I don't have Slingshot hardware. The CXI profile will be compile-tested and share the tcp-provider coverage for the generic logic. Validation help from CXI users (cc #2535 authors) would be very welcome.

PR plan

TENT fabric core with the efa and tcp profiles: DRAM, chunking, failover, rdma skipping EFA devices, tcp-provider tests.
HMEM (CUDA) memory.
cxi profile.
Shim: protocol=efa/cxi pinning (#3995).

Migrating classic efa_transport / cxi_transport onto the shared core is out of scope for now. It can follow once the TENT backend is stable (#2657).

Questions for TENT owners (@alogfans @staryxchen)

One FABRIC transport type with providers selected by config (transports/fabric/provider = auto|efa|cxi), or separate EFA / CXI transport types sharing an implementation?
OK to factor RailMonitor, DeviceSelector, EndpointStore and slice planning out of transport/rdma into a shared location?
Is there a timeline for making TENT the default? That affects how much classic EFA work to keep doing.

Related: #2657, #3995, #3500, #4354, #4355.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Drafted with the help of Claude Code; design and code reviewed by me.

Originally created by @whn09 on GitHub (Sep 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4412 ### Changes proposed ## Motivation - **TENT has no libfabric backend, so AWS EFA and HPE CXI users can't use it.** - TENT's `rdma` backend uses RC QPs (`tent/src/transport/rdma/endpoint.cpp:136`). EFA has no RC and is only usable through libfabric `FI_EP_RDM` (SRD). - Today, `MC_USE_TENT=1` on an EFA build fails at `initialize()` (#3995 P0-efa-ascend / P1-install-transport). - **TENT discovers NICs only through ibverbs.** EFA devices (`rdmap*`) show up as `NIC_RDMA` and the rdma backend tries to install on them. CXI has no verbs device, so it isn't discovered at all. - **Classic TE has two libfabric transports that are mostly copies of each other.** - `cxi_transport` was forked from the #1944-era `efa_transport` (#2535). After renaming, about 80–90% of it is identical to EFA. - The 13 EFA fixes since then were never ported. CXI still lacks the stale-AV locking (#2023), chunk-straddling slices (#3505), provider-derived WR/CQ depth (#3296) and bounded parallel registration. This is the maintenance debt tracked in #2657. ## Proposal Add **one** TENT transport, `FABRIC`, built on libfabric. Provider differences go into a small per-provider profile. The classic EFA transport as it is today is the baseline. **Provider profile** | | efa | cxi | tcp;ofi_rxm (CI only) | |---|---|---|---| | `fi_getinfo` version | `FI_VERSION(1,18)` (#2041) | `fi_version()` | `fi_version()` | | domain | `<dev>-rdm` | `<dev>` | auto | | `mr_mode` hints | LOCAL, VIRT_ADDR, ALLOCATED, PROV_KEY [, HMEM] | ENDPOINT, LOCAL, ALLOCATED, PROV_KEY [, HMEM] | provider default | | MR bind to EP | no | yes | from `mr_mode` | | remote address | VA | offset | from `mr_mode` | | MR size cap | min(`max_mr_size`, PTE budget), as in #1912 | `max_mr_size` | `max_mr_size` | | quirks | `FI_HMEM=system` in non-GPU builds; no `fi_av_remove` after EP close | — | — | Wherever possible, "bind to EP" and "VA vs offset" are derived from the `mr_mode` that `fi_getinfo` actually returns, not from the provider name. **Discovery** - Enumerate domains with `fi_getinfo` and take PCI/NUMA from `fi_info->nic->bus_attr`. This works the same for EFA and CXI. - Add `NIC_FABRIC` to the TENT topology. - The `rdma` backend skips EFA devices, the same as #4355 does for classic TE. **Metadata (no bootstrap RPC)** - RDM is connectionless. - Each NIC's `fi_getname` blob is stored in `DeviceDesc.transport_attrs[FABRIC]`. Peers call `fi_av_insert` lazily on first use. - Buffer keys are 64-bit, so they don't go in the `uint32_t rkey` vector. They are stored in `BufferDesc.transport_attrs[FABRIC]` as `{base, chunks: [{off, len, keys[nic]}]}`, which also carries the chunk layout of large registrations. - A missing peer address or key returns `NeedsRefreshCache`. **Data path** - Same shape as the TENT rdma backend: SubBatch, slab tasks/slices, and worker threads that post and poll the CQ. - Slices are cut at chunk boundaries on both sides, then striped across NICs. - WR/CQ credits come from the provider's `tx_attr->size`. `-FI_EAGAIN` hands the unposted tail back. The credit wait is bounded by wall-clock time. - `fi_cq_readerr` feeds slice retry. After `max_retry_count`, the slice is marked FAILED and the engine's failover takes over. The RailMonitor gets `markFailed` / `markRecovered`. - The peer table read lock is held across posting (#2023). - I'd like to share the generic pieces with the rdma backend rather than copy them: RailMonitor, DeviceSelector, EndpointStore, and slice planning (keys widened to 64-bit). Feedback on that is welcome. **Memory** - Chunked registration with bounded parallelism (`runBoundedRegMrBatch`), and a per-profile NIC assignment policy. - GPU memory is registered with `fi_mr_regattr` (CUDA/ROCR iface) under `ScopedCudaContext` (#3177). - Neuron (#3888) comes later. **Loading and build** - libfabric is loaded with `dlopen`, using the loader from #4354, so non-fabric processes don't depend on it. - New `tent_xport_fabric` target behind `USE_FABRIC` (requires `USE_TENT`). It always compiles, the way MPCOMM does. **Classic → TENT shim** - `init(..., protocol="efa"|"cxi")` pins the `fabric` backend with that provider and disables rdma. - Under TENT, EFA/CXI Python `initialize()` no longer relies on `installTransport()` returning non-null. - This closes the two EFA items in #3995. ## Testing - **CI without hardware:** unit tests on the libfabric `tcp;ofi_rxm` provider. Covers install, loopback, write/read, chunk-straddling transfers, large chunked buffers, peer loss reported as FAILED, and metadata refresh. - **EFA:** p5en, two nodes, TENT bench against classic EFA bandwidth, plus Store e2e with `MC_USE_TENT=1`. - **CXI:** I don't have Slingshot hardware. The CXI profile will be compile-tested and share the tcp-provider coverage for the generic logic. Validation help from CXI users (cc #2535 authors) would be very welcome. ## PR plan 1. TENT fabric core with the efa and tcp profiles: DRAM, chunking, failover, rdma skipping EFA devices, tcp-provider tests. 2. HMEM (CUDA) memory. 3. cxi profile. 4. Shim: `protocol=efa/cxi` pinning (#3995). Migrating classic `efa_transport` / `cxi_transport` onto the shared core is out of scope for now. It can follow once the TENT backend is stable (#2657). ## Questions for TENT owners (@alogfans @staryxchen) 1. One `FABRIC` transport type with providers selected by config (`transports/fabric/provider = auto|efa|cxi`), or separate `EFA` / `CXI` transport types sharing an implementation? 2. OK to factor RailMonitor, DeviceSelector, EndpointStore and slice planning out of `transport/rdma` into a shared location? 3. Is there a timeline for making TENT the default? That affects how much classic EFA work to keep doing. Related: #2657, #3995, #3500, #4354, #4355. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/) --- Drafted with the help of Claude Code; design and code reviewed by me.

bsakaguchi added the

RFC

label

2026-10-29 21:23:00 +00:00

bsakaguchi commented

2026-10-29 21:23:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 30, 2026):

Thanks for opening this issue, @whn09!

Field	Value
Issue	#4412
GitHub user ID	`964527`
Reporter	@whn09

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 30, 2026): Thanks for opening this issue, @whn09! | Field | Value | |-------|-------| | **Issue** | #4412 | | **GitHub user ID** | `964527` | | **Reporter** | @whn09 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:11 +00:00

[PR #1113] [CLOSED] Ci test #2022