[GH-ISSUE #2022] [Bug]: Segfault in submitSlicesOnPeer → fi_write when peer reconnects during an in-flight transfer (EFA) #561

New issue

Closed

opened 2026-10-29 21:16:27 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Owner

Originally created by @whn09 on GitHub (Apr 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2022

[EFA] Segfault in `submitSlicesOnPeer` → `fi_write` when peer reconnects during an in-flight transfer

Summary

Under high-concurrency KV transfers on AWS EFA (SGLang 2P2D prefill, 16 NICs,
MC_MAX_WR=16384, MOONCAKE_PROTOCOL=efa, P2P handshake mode), Mooncake
segfaults inside libfabric's fi_write a short while after a peer changes
its P2P handshake port (the normal "random port per run" behaviour):

Peer reconnected with new address, re-establishing:
  172.31.22.180:15431@rdmap111s0 -> 172.31.22.180:16364@rdmap111s0
...
!!!!!!! Segfault encountered !!!!!!!
  /opt/amazon/efa/include/rdma/fi_rma.h:122  fi_write
  EfaContext::submitSlicesOnPeer        efa_context.cpp:813
  EfaContext::submitPostSend            efa_context.cpp:652
  EfaTransport::submitTransferTask      efa_transport.cpp:911
  MultiTransport::submitTransfer        multi_transport.cpp:134
  TransferEngineImpl::submitTransfer    transfer_engine_impl.h:118
  TransferEngine::submitTransfer        transfer_engine.cpp:92
  TransferEnginePy::batchTransferSync   transfer_engine_py.cpp:513

Called from SGLang's transfer_worker → send_kvcache → batch_transfer_sync.

Reproduction

Commit: kvcache-ai/Mooncake@b0bda8c (main, 2026-04-30)
Hardware: 4 × AWS p5en.48xlarge (8 × H200, 16 × EFA 100 Gbps each)
Transport: protocol=efa, metadata = P2PHANDSHAKE, hugepages=65536×2MB
Tuning: MC_NUM_CQ_PER_CTX=4, MC_MAX_CQE_PER_CTX=65536, MC_MAX_WR=16384,
FI_EFA_USE_HUGE_PAGE=1, FI_EFA_USE_DEVICE_RDMA=1
Workload: SGLang GLM-5.1-FP8 2P2D disaggregated prefill/decode,
topology pp2 tp8 cp8 deepep8 (prefill) + tp16 dp16 deepep16 (decode),
then vllm bench serve equivalent of 180 prompts, 3K input / 1K output,
rate=18 req/s.

The segfault occurs consistently at the end of the benchmark burst when
one of the 16 decode DP ranks rebinds its Mooncake RPC port (random port
selection in P2P handshake mode is documented in
common.h::normalizeNicPath), triggering
EfaEndPoint::setPeerNicPath to take the CONNECTED-branch disconnect.

Root cause hypothesis

EfaEndPoint::setPeerNicPath (efa_endpoint.cpp:31-40):

void EfaEndPoint::setPeerNicPath(const std::string& peer_nic_path) {
    RWSpinlock::WriteGuard guard(lock_);
    if (peer_nic_path_ == peer_nic_path) return;
    if (status_.load(std::memory_order_relaxed) == CONNECTED) {
        LOG(INFO) << "Peer reconnected with new address, re-establishing: ...";
        disconnectUnlocked();
    }
    peer_nic_path_ = peer_nic_path;
}

disconnectUnlocked() tears down AV entries / endpoint state associated
with the old peer_fi_addr. But a concurrent EfaContext::submitSlicesOnPeer
(efa_context.cpp:813) takes its own post_lock_ spinlock and then calls
fi_write(shared_ep_, ..., peer_fi_addr, ...) with a peer_fi_addr it
latched earlier from EfaEndPoint::peer_fi_addr_:

ret = fi_write(shared_ep_, (void*)entry.slice->source_addr,
               entry.slice->length, entry.local_desc,
               peer_fi_addr,   // ← stale after setPeerNicPath teardown
               entry.slice->rdma.dest_addr,
               entry.slice->rdma.dest_rkey,
               &entry.op_ctx->fi_ctx);

post_lock_ (on EfaContext) and EfaEndPoint::lock_ are different
locks, so disconnect and post_send are not mutually exclusive. Libfabric
dereferences the torn-down AV entry → segfault.

Why it shows up now

P2P handshake mode assigns a random RPC port per run (the
documented behaviour in common.h:476-488). So periodically, especially
when a peer worker process re-initialises a Mooncake RPC socket, the
peer_nic_path_ string changes (same IP + same NIC, different port
suffix).
Under high concurrency (16 sender threads × 16 NICs × large
MC_MAX_WR), the probability of a pending fi_write being in flight
when setPeerNicPath fires is high.
Lower concurrency configurations (e.g. tp16 dp4 moe_a2a=none on the
same hardware, same benchmark) did not reproduce.

Proposed fix directions

Option A: Have setPeerNicPath acquire/share EfaContext::post_lock_
around the disconnectUnlocked() call, so it cannot race with
in-flight submitSlicesOnPeer.
Option B: Before disconnect, fence all CQEs associated with the
old peer (drain pending WRs) and only then remove AV entries.
Option C: Ref-count peer_fi_addr in EfaEndPoint and have the
sender path hold a reference across fi_write, letting disconnect block
until all refs drop.

Option A is the smallest change but couples two lock hierarchies. Option
B is cleanest semantically but adds latency to legitimate reconnects.

Workarounds for users hitting this today

Reduce reconnect probability by reducing DP/worker cardinality so
fewer unique (peer_ip, peer_port) tuples exist.
Use a stable metadata server (etcd) instead of P2PHANDSHAKE so ports
don't change per run — but note that getDefaultHandshakePort()
is currently bypassed by the P2P path (each process calls
findAvailableTcpPort() regardless); using etcd won't stabilise ports
unless combined with MC_LEGACY_RPC_PORT_BINDING.
Valid benchmark numbers can still be extracted: the crash happens
after all prompts in a burst complete, so steady-state latencies are
captured correctly before the segfault takes down the server.

Originally created by @whn09 on GitHub (Apr 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2022 # [EFA] Segfault in `submitSlicesOnPeer` → `fi_write` when peer reconnects during an in-flight transfer ## Summary Under high-concurrency KV transfers on AWS EFA (SGLang 2P2D prefill, 16 NICs, `MC_MAX_WR=16384`, `MOONCAKE_PROTOCOL=efa`, P2P handshake mode), Mooncake segfaults inside libfabric's `fi_write` a short while after a peer changes its P2P handshake port (the normal "random port per run" behaviour): ``` Peer reconnected with new address, re-establishing: 172.31.22.180:15431@rdmap111s0 -> 172.31.22.180:16364@rdmap111s0 ... !!!!!!! Segfault encountered !!!!!!! /opt/amazon/efa/include/rdma/fi_rma.h:122 fi_write EfaContext::submitSlicesOnPeer efa_context.cpp:813 EfaContext::submitPostSend efa_context.cpp:652 EfaTransport::submitTransferTask efa_transport.cpp:911 MultiTransport::submitTransfer multi_transport.cpp:134 TransferEngineImpl::submitTransfer transfer_engine_impl.h:118 TransferEngine::submitTransfer transfer_engine.cpp:92 TransferEnginePy::batchTransferSync transfer_engine_py.cpp:513 ``` Called from SGLang's `transfer_worker` → `send_kvcache` → `batch_transfer_sync`. ## Reproduction - **Commit**: `kvcache-ai/Mooncake@b0bda8c` (main, 2026-04-30) - **Hardware**: 4 × AWS p5en.48xlarge (8 × H200, 16 × EFA 100 Gbps each) - **Transport**: `protocol=efa`, metadata = `P2PHANDSHAKE`, hugepages=65536×2MB - **Tuning**: `MC_NUM_CQ_PER_CTX=4`, `MC_MAX_CQE_PER_CTX=65536`, `MC_MAX_WR=16384`, `FI_EFA_USE_HUGE_PAGE=1`, `FI_EFA_USE_DEVICE_RDMA=1` - **Workload**: SGLang GLM-5.1-FP8 2P2D disaggregated prefill/decode, topology `pp2 tp8 cp8 deepep8` (prefill) + `tp16 dp16 deepep16` (decode), then `vllm bench serve` equivalent of 180 prompts, 3K input / 1K output, rate=18 req/s. The segfault occurs consistently at the end of the benchmark burst when one of the 16 decode DP ranks rebinds its Mooncake RPC port (random port selection in P2P handshake mode is documented in `common.h::normalizeNicPath`), triggering `EfaEndPoint::setPeerNicPath` to take the CONNECTED-branch disconnect. ## Root cause hypothesis `EfaEndPoint::setPeerNicPath` (efa_endpoint.cpp:31-40): ```cpp void EfaEndPoint::setPeerNicPath(const std::string& peer_nic_path) { RWSpinlock::WriteGuard guard(lock_); if (peer_nic_path_ == peer_nic_path) return; if (status_.load(std::memory_order_relaxed) == CONNECTED) { LOG(INFO) << "Peer reconnected with new address, re-establishing: ..."; disconnectUnlocked(); } peer_nic_path_ = peer_nic_path; } ``` `disconnectUnlocked()` tears down AV entries / endpoint state associated with the old `peer_fi_addr`. But a concurrent `EfaContext::submitSlicesOnPeer` (efa_context.cpp:813) takes its own `post_lock_` spinlock and then calls `fi_write(shared_ep_, ..., peer_fi_addr, ...)` with a `peer_fi_addr` it latched earlier from `EfaEndPoint::peer_fi_addr_`: ```cpp ret = fi_write(shared_ep_, (void*)entry.slice->source_addr, entry.slice->length, entry.local_desc, peer_fi_addr, // ← stale after setPeerNicPath teardown entry.slice->rdma.dest_addr, entry.slice->rdma.dest_rkey, &entry.op_ctx->fi_ctx); ``` `post_lock_` (on `EfaContext`) and `EfaEndPoint::lock_` are different locks, so disconnect and post_send are not mutually exclusive. Libfabric dereferences the torn-down AV entry → segfault. ## Why it shows up now 1. P2P handshake mode **assigns a random RPC port per run** (the documented behaviour in `common.h:476-488`). So periodically, especially when a peer worker process re-initialises a Mooncake RPC socket, the `peer_nic_path_` string changes (same IP + same NIC, different port suffix). 2. Under high concurrency (16 sender threads × 16 NICs × large `MC_MAX_WR`), the probability of a pending `fi_write` being in flight when `setPeerNicPath` fires is high. 3. Lower concurrency configurations (e.g. `tp16 dp4 moe_a2a=none` on the same hardware, same benchmark) did not reproduce. ## Proposed fix directions - **Option A**: Have `setPeerNicPath` acquire/share `EfaContext::post_lock_` around the `disconnectUnlocked()` call, so it cannot race with in-flight `submitSlicesOnPeer`. - **Option B**: Before disconnect, fence all CQEs associated with the old peer (drain pending WRs) and only then remove AV entries. - **Option C**: Ref-count `peer_fi_addr` in `EfaEndPoint` and have the sender path hold a reference across `fi_write`, letting disconnect block until all refs drop. Option A is the smallest change but couples two lock hierarchies. Option B is cleanest semantically but adds latency to legitimate reconnects. ## Workarounds for users hitting this today 1. Reduce reconnect probability by reducing DP/worker cardinality so fewer unique `(peer_ip, peer_port)` tuples exist. 2. Use a stable metadata server (etcd) instead of `P2PHANDSHAKE` so ports don't change per run — **but note** that `getDefaultHandshakePort()` is currently bypassed by the P2P path (each process calls `findAvailableTcpPort()` regardless); using etcd won't stabilise ports unless combined with `MC_LEGACY_RPC_PORT_BINDING`. 3. Valid benchmark numbers can still be extracted: the crash happens *after* all prompts in a burst complete, so steady-state latencies are captured correctly before the segfault takes down the server.

bsakaguchi closed this issue

2026-10-29 21:16:27 +00:00

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Author

Owner

@whn09 commented on GitHub (Apr 30, 2026):

Follow-up: the reconnect path itself is transport-agnostic

Confirmed by reading source that all three transports implement
setPeerNicPath with the same disconnect-on-CONNECTED branch:

EFA (efa_endpoint.cpp:31-40): LOG(INFO) "Peer reconnected with new address"
RDMA (rdma_endpoint.cpp:140-147): LOG(WARNING) "Previous connection will be discarded"
Urma / Kunpeng (urma_endpoint.cpp:681-688): same as RDMA

So users on RoCE/IB and Kunpeng see the same reconnect/teardown behavior
whenever a peer rebinds its P2P handshake port.

What makes EFA particularly segfault-prone is a structural difference:

RdmaEndPoint::disconnectUnlocked() tears down the endpoint's own
QP, isolated from other endpoints on the same context.
EfaEndPoint::disconnectUnlocked() removes entries from
EfaContext::av_, which is shared across all endpoints hosted by
that context. A concurrent fi_write(shared_ep_, ..., peer_fi_addr, ...) on another endpoint hits the stale AV entry and segfaults.

The race window exists on RDMA too, but crashing requires the stale QP to
outlive a post_send from the same endpoint (harder to hit because each
endpoint has its own QP), whereas on EFA any concurrent transfer to the
same peer over any of the ~16 EFA NICs can trip it.

This also explains the log frequency asymmetry: an RDMA deployment has
1-2 endpoints per peer, so one port change logs 1-2 lines; an EFA
deployment has one endpoint per (local NIC, peer NIC), so 16 NICs × 16
peer NICs can log hundreds of lines for the same physical port-change
event.

Suggests any fix should consider both transports — RDMA is latently
racy even if not currently observed to crash.

@whn09 commented on GitHub (Apr 30, 2026): ## Follow-up: the reconnect path itself is transport-agnostic Confirmed by reading source that all three transports implement `setPeerNicPath` with the same disconnect-on-CONNECTED branch: - EFA (`efa_endpoint.cpp:31-40`): `LOG(INFO) "Peer reconnected with new address"` - RDMA (`rdma_endpoint.cpp:140-147`): `LOG(WARNING) "Previous connection will be discarded"` - Urma / Kunpeng (`urma_endpoint.cpp:681-688`): same as RDMA So users on RoCE/IB and Kunpeng see the same reconnect/teardown behavior whenever a peer rebinds its P2P handshake port. What makes EFA particularly segfault-prone is a structural difference: - `RdmaEndPoint::disconnectUnlocked()` tears down the endpoint's **own** QP, isolated from other endpoints on the same context. - `EfaEndPoint::disconnectUnlocked()` removes entries from `EfaContext::av_`, which is **shared across all endpoints** hosted by that context. A concurrent `fi_write(shared_ep_, ..., peer_fi_addr, ...)` on another endpoint hits the stale AV entry and segfaults. The race window exists on RDMA too, but crashing requires the stale QP to outlive a post_send from the same endpoint (harder to hit because each endpoint has its own QP), whereas on EFA any concurrent transfer to the same peer over any of the ~16 EFA NICs can trip it. This also explains the log frequency asymmetry: an RDMA deployment has 1-2 endpoints per peer, so one port change logs 1-2 lines; an EFA deployment has one endpoint per (local NIC, peer NIC), so 16 NICs × 16 peer NICs can log hundreds of lines for the same physical port-change event. Suggests any fix should consider both transports — RDMA is latently racy even if not currently observed to crash.