[GH-ISSUE #4674] [Bug]: [TransferEngine] HTTP metadata server rejects valid rpc_meta republish as duplicate #1189

New issue

Open

opened 2026-10-29 21:23:46 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:46 +00:00

Owner

Originally created by @thrower01 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4674

Bug Report

Context

The Python HTTP metadata backend (python/mooncake/http_metadata_server.py ) is a lightweight alternative to etcd/Redis for Mooncake Transfer Engine deployments. On the C++ side,TransferMetadata::rePublishRpcMetaEntry (inmooncake-transfer-engine/src/transfer_metadata.cpp:1928 ) implements a versioned idempotency rule: if the same(ip_or_host_name, rpc_port) publishes again, the client bumpsmetadata_version and PUTs the new JSON. This is the normal path for process restart, multi-worker duplicate registration, and any republish scenario.

The current Python server gatesrpc_meta keys with two bugs, and together they make a legitimate republish always fail.

Bug 1 — substring match is too broad

if key.find("rpc_meta") != -1 and key in self.store:

A key such asmooncake/segment_rpc_meta_cache/x is NOT an rpc_meta key, but the substring check pulls it into the gate. Non-rpc_meta business keys whose names happen to containrpc_meta can no longer be overwritten.

Bug 2 — byte-level equality rejects version bumps

if self.store[key] == data: return 200 return 400 "Duplicate rpc_meta key not allowed"

The C++ client always incrementsmetadata_version on republish, so the new payload bytes are never byte-identical to the existing value. Every valid republish is treated as a duplicate conflict.

Impact

Any cluster using the HTTP metadata server will get 400 responses when clients republish their own rpc_meta (process restart, duplicate registration), forcing the client into retry or misleading the operator into thinking the metadata backend is broken. Non-rpc_meta keys whose name merely contains the substring can also stop being writable.

Fix direction

Classify rpc_meta keys by path-segment membership: split on/ , require"rpc_meta" as a literal segment.
Replace byte-level equality with a JSON-aware rule:
- identical bytes → idempotent accept
- metadata_version strictly increasing and(ip_or_host_name, rpc_port) unchanged → accept overwrite
- metadata_version rollback or identity-field conflict → 400 with a machine-readable reason
  Scope of fix

The same broken gate is also copied intomooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py (the example had the older version that rejected ALL duplicate rpc_meta keys, not even allowing idempotent same-byte writes). Both need updating.

Acceptance

New unit tests for substring false-positive, version-bump acceptance, rollback rejection, identity-conflict rejection. Existing same-byte idempotency / conflict tests keep passing.
Semantics aligned withTransferMetadata::rePublishRpcMetaEntry .

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @thrower01 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4674 ### Bug Report Context The Python HTTP metadata backend (`python/mooncake/http_metadata_server.py` ) is a lightweight alternative to etcd/Redis for Mooncake Transfer Engine deployments. On the C++ side,`TransferMetadata::rePublishRpcMetaEntry` (in`mooncake-transfer-engine/src/transfer_metadata.cpp:1928` ) implements a versioned idempotency rule: if the same`(ip_or_host_name, rpc_port)` publishes again, the client bumps`metadata_version` and PUTs the new JSON. This is the normal path for process restart, multi-worker duplicate registration, and any republish scenario. The current Python server gates`rpc_meta` keys with two bugs, and together they make a legitimate republish always fail. Bug 1 — substring match is too broad `if key.find("rpc_meta") != -1 and key in self.store:` A key such as`mooncake/segment_rpc_meta_cache/x` is NOT an rpc_meta key, but the substring check pulls it into the gate. Non-rpc_meta business keys whose names happen to contain`rpc_meta` can no longer be overwritten. Bug 2 — byte-level equality rejects version bumps `if self.store[key] == data: return 200 return 400 "Duplicate rpc_meta key not allowed"` The C++ client always increments`metadata_version` on republish, so the new payload bytes are never byte-identical to the existing value. Every valid republish is treated as a duplicate conflict. Impact Any cluster using the HTTP metadata server will get 400 responses when clients republish their own rpc_meta (process restart, duplicate registration), forcing the client into retry or misleading the operator into thinking the metadata backend is broken. Non-rpc_meta keys whose name merely contains the substring can also stop being writable. Fix direction - Classify rpc_meta keys by path-segment membership: split on`/` , require`"rpc_meta"` as a literal segment. - Replace byte-level equality with a JSON-aware rule: - identical bytes → idempotent accept - `metadata_version` strictly increasing and`(ip_or_host_name, rpc_port)` unchanged → accept overwrite - `metadata_version` rollback or identity-field conflict → 400 with a machine-readable reason Scope of fix The same broken gate is also copied into`mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py` (the example had the older version that rejected ALL duplicate rpc_meta keys, not even allowing idempotent same-byte writes). Both need updating. Acceptance - New unit tests for substring false-positive, version-bump acceptance, rollback rejection, identity-conflict rejection. Existing same-byte idempotency / conflict tests keep passing. - Semantics aligned with`TransferMetadata::rePublishRpcMetaEntry` . ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:46 +00:00

bsakaguchi commented

2026-10-29 21:23:47 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 15, 2026):

Thanks for opening this issue, @thrower01!

Field	Value
Issue	#4674
GitHub user ID	`340463471`
Reporter	@thrower01

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 15, 2026): Thanks for opening this issue, @thrower01! | Field | Value | |-------|-------| | **Issue** | #4674 | | **GitHub user ID** | `340463471` | | **Reporter** | @thrower01 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:29 +00:00

[PR #1189] [CLOSED] [NvlinkTransport] Fix HIP Shareable POSIX File Descriptor Handles #2062