kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1369] [Bug] Segfault/LogicError in SocketHandShakePlugin during high-concurrency RDMA transfers #441

New issue

Closed

opened 2026-10-29 21:14:32 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:14:32 +00:00

Owner

Originally created by @herrluk on GitHub (Jan 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1369

Originally assigned to: @alogfans on GitHub.

Description

Component: Mooncake Transfer Engine (Python binding via mooncake.engine.TransferEngine)

When running high-concurrency RDMA transfers (e.g., 32+ concurrent recv operations across multiple machines), the Transfer Engine crashes with either a segmentation fault or a Json::LogicError in the SocketHandShake daemon thread.

Error Messages

Error 1: Segmentation Fault

dmesg output:

python[4136606]: segfault at 7fbacbb73f77 ip 00007fd8a7f982cd sp 00007fcefb5fc6e8 error 4 in libc.so.6

GDB backtrace:

#0  0x00007fd8a7f982cd in ?? () from libc.so.6
#1  0x00007fd424bdbb72 in ?? () from libjsoncpp-7d699962.so.1.9.5
#2  0x00007fd424be06a3 in Json::Value::Value(std::string const&) () from libjsoncpp-7d699962.so.1.9.5
#3  0x00007fd1a43c9065 in mooncake::TransferMetadata::encodeSegmentDesc(...) () from engine.so
#4  0x00007fd1a43cb023 in mooncake::TransferMetadata::receivePeerMetadata(...) () from engine.so
#5  0x00007fd1a43dc4d7 in mooncake::SocketHandShakePlugin::startDaemon(...)::{lambda()#1}::operator()() from engine.so

Error 2: Json::LogicError

terminate called after throwing an instance of 'Json::LogicError'
  what():  in Json::Value::duplicateAndPrefixStringValue(): length too big for prefixing

Environment

Mooncake version: 0.3.7 (Transfer Engine Python binding)
OS: Ubuntu 24.04.2 LTS (Linux kernel 5.15.0-124-generic)
Python version: 3.10
libjsoncpp version: 1.9.5-6build1 (libjsoncpp25)
RDMA NIC: Mellanox ConnectX-5/6 (mlx5), All machines and all instances exclusively use a single network card (mlx5_3); no other network cards are enabled or utilized
Cluster size: ~32 machines, 8 GPUs per machine
Cluster size: ~32 machines, 8 GPUs per machine

Use Case: Distributed Data Exchange with High-Concurrency RDMA

We are running a distributed workload where multiple processes across machines exchange data using Mooncake Transfer Engine.

Cluster Configuration

Component	Configuration
Machines	32
Sender processes per machine	2
TransferEngine instances per sender	32
Receiver processes per machine	1
TransferEngine instances per receiver	32

Data Flow

Phase 1: Send (High-concurrency)
├── 32 machines × 2 sender processes = 64 processes
├── Each process: 32 TransferEngine instances doing concurrent send
├── Total: ~2048 concurrent RDMA send operations
└── Data: ~10,000 buffers registered per machine

Phase 2: Receive (High-concurrency)
├── After ALL sends complete
├── 32 machines × 1 receiver process = 32 processes
├── Each process: 32 TransferEngine instances doing concurrent recv
├── Total: ~1024 concurrent RDMA recv operations
└── Each machine receives data from **all** other machines **as well as its own local sending processes**.(N-to-N pattern)

Reproduction Steps

Deploy a 32-node cluster with RDMA connectivity

Send phase: Each machine runs 2 processes, each creates 32 TransferEngine instances:

from mooncake.engine import TransferEngine

# Create 32 TransferEngine instances per process
engines = []
for i in range(32):
    engine = TransferEngine()
    engine.init(local_hostname, device_name)
    engines.append(engine)

# Concurrent send operations
for engine in engines:
    engine.register_buffer(key, buffer_ptr, size)

Receive phase: After sends complete, each machine runs 1 process with 32 TransferEngine instances doing concurrent recv:

# Create 32 TransferEngine instances
engines = []
for i in range(32):
    engine = TransferEngine()
    engine.init(local_hostname, device_name)
    engines.append(engine)

# Concurrent recv from multiple remote hosts
# This triggers many simultaneous RDMA handshakes
with ThreadPoolExecutor(max_workers=32) as executor:
    futures = [executor.submit(engine.transfer_sync, ...) for engine in engines]

The crash occurs sporadically during Phase 2, when multiple RDMA connections are being established simultaneously across the cluster (N-to-N handshake pattern)

Analysis

Both errors occur in SocketHandShakePlugin::startDaemon when processing peer metadata:

encodeSegmentDesc() encodes a SegmentDesc structure to JSON
When constructing Json::Value from a string field in SegmentDesc:
- If the string pointer is corrupted → Segfault
- If the string length is corrupted → "length too big for prefixing"

This suggests a memory corruption issue in SegmentDesc, possibly due to:

Race condition in the handshake daemon thread
Use-after-free of SegmentDesc structure
Missing synchronization when multiple connections are established concurrently

Workaround

Reducing concurrency mitigates the issue:

Use fewer TransferEngine instances per process (1-4 instead of 32)
Reduce concurrent recv operations (4-8 instead of 32)

Expected Behavior

The Transfer Engine should handle high-concurrency RDMA connections without crashing.

Originally created by @herrluk on GitHub (Jan 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1369 Originally assigned to: @alogfans on GitHub. ## Description **Component:** Mooncake Transfer Engine (Python binding via `mooncake.engine.TransferEngine`) When running high-concurrency RDMA transfers (e.g., 32+ concurrent recv operations across multiple machines), the Transfer Engine crashes with either a segmentation fault or a `Json::LogicError` in the SocketHandShake daemon thread. ## Error Messages ### Error 1: Segmentation Fault dmesg output: ``` python[4136606]: segfault at 7fbacbb73f77 ip 00007fd8a7f982cd sp 00007fcefb5fc6e8 error 4 in libc.so.6 ``` GDB backtrace: ``` #0 0x00007fd8a7f982cd in ?? () from libc.so.6 #1 0x00007fd424bdbb72 in ?? () from libjsoncpp-7d699962.so.1.9.5 #2 0x00007fd424be06a3 in Json::Value::Value(std::string const&) () from libjsoncpp-7d699962.so.1.9.5 #3 0x00007fd1a43c9065 in mooncake::TransferMetadata::encodeSegmentDesc(...) () from engine.so #4 0x00007fd1a43cb023 in mooncake::TransferMetadata::receivePeerMetadata(...) () from engine.so #5 0x00007fd1a43dc4d7 in mooncake::SocketHandShakePlugin::startDaemon(...)::{lambda()#1}::operator()() from engine.so ``` ### Error 2: Json::LogicError ``` terminate called after throwing an instance of 'Json::LogicError' what(): in Json::Value::duplicateAndPrefixStringValue(): length too big for prefixing ``` ## Environment - **Mooncake version:** 0.3.7 (Transfer Engine Python binding) - **OS:** Ubuntu 24.04.2 LTS (Linux kernel 5.15.0-124-generic) - **Python version:** 3.10 - **libjsoncpp version:** 1.9.5-6build1 (libjsoncpp25) - **RDMA NIC:** Mellanox ConnectX-5/6 (mlx5), **All machines and all instances exclusively use a single network card (mlx5_3); no other network cards are enabled or utilized** - **Cluster size:** ~32 machines, 8 GPUs per machine - **Cluster size:** ~32 machines, 8 GPUs per machine ## Use Case: Distributed Data Exchange with High-Concurrency RDMA We are running a distributed workload where multiple processes across machines exchange data using Mooncake Transfer Engine. ### Cluster Configuration | Component | Configuration | |-----------|---------------| | Machines | 32 | | Sender processes per machine | 2 | | TransferEngine instances per sender | 32 | | Receiver processes per machine | 1 | | TransferEngine instances per receiver | 32 | ### Data Flow ``` Phase 1: Send (High-concurrency) ├── 32 machines × 2 sender processes = 64 processes ├── Each process: 32 TransferEngine instances doing concurrent send ├── Total: ~2048 concurrent RDMA send operations └── Data: ~10,000 buffers registered per machine Phase 2: Receive (High-concurrency) ├── After ALL sends complete ├── 32 machines × 1 receiver process = 32 processes ├── Each process: 32 TransferEngine instances doing concurrent recv ├── Total: ~1024 concurrent RDMA recv operations └── Each machine receives data from **all** other machines **as well as its own local sending processes**.(N-to-N pattern) ``` ### Reproduction Steps 1. Deploy a 32-node cluster with RDMA connectivity 2. **Send phase**: Each machine runs 2 processes, each creates 32 TransferEngine instances: ```python from mooncake.engine import TransferEngine # Create 32 TransferEngine instances per process engines = [] for i in range(32): engine = TransferEngine() engine.init(local_hostname, device_name) engines.append(engine) # Concurrent send operations for engine in engines: engine.register_buffer(key, buffer_ptr, size) ``` 3. **Receive phase**: After sends complete, each machine runs 1 process with 32 TransferEngine instances doing concurrent recv: ```python # Create 32 TransferEngine instances engines = [] for i in range(32): engine = TransferEngine() engine.init(local_hostname, device_name) engines.append(engine) # Concurrent recv from multiple remote hosts # This triggers many simultaneous RDMA handshakes with ThreadPoolExecutor(max_workers=32) as executor: futures = [executor.submit(engine.transfer_sync, ...) for engine in engines] ``` 4. The crash occurs sporadically during Phase 2, when multiple RDMA connections are being established simultaneously across the cluster (N-to-N handshake pattern) ## Analysis Both errors occur in `SocketHandShakePlugin::startDaemon` when processing peer metadata: 1. `encodeSegmentDesc()` encodes a `SegmentDesc` structure to JSON 2. When constructing `Json::Value` from a string field in `SegmentDesc`: - If the string pointer is corrupted → **Segfault** - If the string length is corrupted → **"length too big for prefixing"** This suggests a **memory corruption issue** in `SegmentDesc`, possibly due to: - Race condition in the handshake daemon thread - Use-after-free of `SegmentDesc` structure - Missing synchronization when multiple connections are established concurrently ## Workaround Reducing concurrency mitigates the issue: - Use fewer TransferEngine instances per process (1-4 instead of 32) - Reduce concurrent recv operations (4-8 instead of 32) ## Expected Behavior The Transfer Engine should handle high-concurrency RDMA connections without crashing.

bsakaguchi closed this issue

2026-10-29 21:14:32 +00:00

bsakaguchi commented

2026-10-29 21:14:33 +00:00

Author

Owner

@staryxchen commented on GitHub (Jan 14, 2026):

@herrluk Please verify whether #1373 resolves the issue.

@staryxchen commented on GitHub (Jan 14, 2026): @herrluk Please verify whether #1373 resolves the issue.

bsakaguchi commented

2026-10-29 21:14:33 +00:00

Author

Owner

@herrluk commented on GitHub (Jan 19, 2026):

@staryxchen Thanks for your PR #1373! I tested it and it successfully fixed the original race condition crash.

Original crash scenario:

Error: corrupted size vs. prev_size (glibc heap corruption detected)
Location: SocketHandShakePlugin::startDaemon → receivePeerMetadata → free() when releasing shared_ptr<SegmentDesc>
Root cause: Concurrent unsynchronized access to segment_id_to_desc_map_ corrupted the heap metadata

However, as you mentioned in the PR description, I also encountered the deadlock issue in P2P metadata exchange scenarios - the openSegment operation would hang indefinitely when trying to connect to a peer.

After analyzing the code, I found a solution that avoids the deadlock while maintaining thread safety:

Key Insight

The deadlock occurs because getSegmentDesc() tries to acquire segment_lock_ while its callers (getSegmentDescByName, syncSegmentCache, getSegmentID) already hold the write lock.

My Fix

receivePeerMetadata(): Use getLocalSegmentDesc() (with read lock) - this function is called from startDaemon callback thread which does NOT hold any lock.
getSegmentDesc(): Do NOT acquire lock again - its callers already hold the write lock. Direct map access is safe in this context.

// In getSegmentDesc():
// Note: Callers (getSegmentDescByName, syncSegmentCache, getSegmentID)
// already hold segment_lock_, so direct map access is safe here
auto desc = segment_id_to_desc_map_[LOCAL_SEGMENT_ID];

This approach follows the same "read-release-operate" pattern you described, but with the awareness that different call paths have different locking contexts.

The fix is working well in my high-concurrency tests (~2048 concurrent sends, ~1024 concurrent receives).

@herrluk commented on GitHub (Jan 19, 2026): @staryxchen Thanks for your PR #1373! I tested it and it successfully fixed the original race condition crash. **Original crash scenario:** - Error: `corrupted size vs. prev_size` (glibc heap corruption detected) - Location: `SocketHandShakePlugin::startDaemon` → `receivePeerMetadata` → `free()` when releasing `shared_ptr<SegmentDesc>` - Root cause: Concurrent unsynchronized access to `segment_id_to_desc_map_` corrupted the heap metadata However, as you mentioned in the PR description, I also encountered the **deadlock issue in P2P metadata exchange scenarios** - the `openSegment` operation would hang indefinitely when trying to connect to a peer. After analyzing the code, I found a solution that avoids the deadlock while maintaining thread safety: ### Key Insight The deadlock occurs because `getSegmentDesc()` tries to acquire `segment_lock_` while its callers (`getSegmentDescByName`, `syncSegmentCache`, `getSegmentID`) **already hold the write lock**. ### My Fix 1. **`receivePeerMetadata()`**: Use `getLocalSegmentDesc()` (with read lock) - this function is called from `startDaemon` callback thread which does **NOT** hold any lock. 2. **`getSegmentDesc()`**: Do **NOT** acquire lock again - its callers already hold the write lock. Direct map access is safe in this context. ```cpp // In getSegmentDesc(): // Note: Callers (getSegmentDescByName, syncSegmentCache, getSegmentID) // already hold segment_lock_, so direct map access is safe here auto desc = segment_id_to_desc_map_[LOCAL_SEGMENT_ID]; ``` This approach follows the same "read-release-operate" pattern you described, but with the awareness that different call paths have different locking contexts. The fix is working well in my high-concurrency tests (~2048 concurrent sends, ~1024 concurrent receives).

bsakaguchi commented

2026-10-29 21:14:34 +00:00

Author

Owner

@staryxchen commented on GitHub (Jan 19, 2026):

@staryxchen Thanks for your PR #1373! I tested it and it successfully fixed the original race condition crash.

Original crash scenario:

Error: corrupted size vs. prev_size (glibc heap corruption detected)

Location: SocketHandShakePlugin::startDaemon → receivePeerMetadata → free() when releasing shared_ptr<SegmentDesc>

Root cause: Concurrent unsynchronized access to segment_id_to_desc_map_ corrupted the heap metadata

However, as you mentioned in the PR description, I also encountered the deadlock issue in P2P metadata exchange scenarios - the openSegment operation would hang indefinitely when trying to connect to a peer.

After analyzing the code, I found a solution that avoids the deadlock while maintaining thread safety:

Key Insight

The deadlock occurs because getSegmentDesc() tries to acquire segment_lock_ while its callers (getSegmentDescByName, syncSegmentCache, getSegmentID) already hold the write lock.

My Fix

receivePeerMetadata(): Use getLocalSegmentDesc() (with read lock) - this function is called from startDaemon callback thread which does NOT hold any lock.

getSegmentDesc(): Do NOT acquire lock again - its callers already hold the write lock. Direct map access is safe in this context.

// In getSegmentDesc():
// Note: Callers (getSegmentDescByName, syncSegmentCache, getSegmentID)
// already hold segment_lock_, so direct map access is safe here
auto desc = segment_id_to_desc_map_[LOCAL_SEGMENT_ID];
This approach follows the same "read-release-operate" pattern you described, but with the awareness that different call paths have different locking contexts.

The fix is working well in my high-concurrency tests (~2048 concurrent sends, ~1024 concurrent receives).

Does the latest commit also have deadlock issues? If so, you can push your fix to PR #1373 , thank you!

@staryxchen commented on GitHub (Jan 19, 2026): > [@staryxchen](https://github.com/staryxchen) Thanks for your PR [#1373](https://github.com/kvcache-ai/Mooncake/pull/1373)! I tested it and it successfully fixed the original race condition crash. > > **Original crash scenario:** > > * Error: `corrupted size vs. prev_size` (glibc heap corruption detected) > * Location: `SocketHandShakePlugin::startDaemon` → `receivePeerMetadata` → `free()` when releasing `shared_ptr<SegmentDesc>` > * Root cause: Concurrent unsynchronized access to `segment_id_to_desc_map_` corrupted the heap metadata > > However, as you mentioned in the PR description, I also encountered the **deadlock issue in P2P metadata exchange scenarios** - the `openSegment` operation would hang indefinitely when trying to connect to a peer. > > After analyzing the code, I found a solution that avoids the deadlock while maintaining thread safety: > > ### Key Insight > The deadlock occurs because `getSegmentDesc()` tries to acquire `segment_lock_` while its callers (`getSegmentDescByName`, `syncSegmentCache`, `getSegmentID`) **already hold the write lock**. > > ### My Fix > 1. **`receivePeerMetadata()`**: Use `getLocalSegmentDesc()` (with read lock) - this function is called from `startDaemon` callback thread which does **NOT** hold any lock. > 2. **`getSegmentDesc()`**: Do **NOT** acquire lock again - its callers already hold the write lock. Direct map access is safe in this context. > > // In getSegmentDesc(): > // Note: Callers (getSegmentDescByName, syncSegmentCache, getSegmentID) > // already hold segment_lock_, so direct map access is safe here > auto desc = segment_id_to_desc_map_[LOCAL_SEGMENT_ID]; > This approach follows the same "read-release-operate" pattern you described, but with the awareness that different call paths have different locking contexts. > > The fix is working well in my high-concurrency tests (~2048 concurrent sends, ~1024 concurrent receives). Does [the latest commit](https://github.com/kvcache-ai/Mooncake/pull/1373/changes/901085f96cac1f932e7b1ee15612efbf18aa2177) also have deadlock issues? If so, you can push your fix to PR #1373 , thank you!