kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4479] [Bug]: Fake completed replica injection and key spoofing via NotifyOffloadSuccess in Mooncake Store Master #1135

New issue

Open

opened 2026-10-29 21:23:14 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:14 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4479

Bug Report

Summary

Mooncake Store Master's NotifyOffloadSuccess RPC (unauthenticated, default 0.0.0.0:50051) accepts caller-supplied offload results with no validation. An unauthenticated peer first forges MountLocalDiskSegment with its own UUID (satisfying the mounted-segment precondition) and then injects a COMPLETE disk-tier replica pointing at an attacker-chosen endpoint into any existing object, or fabricates metadata for keys that never existed — the master then reports those keys as existing with a completed replica at the attacker's endpoint. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release); no fix available. Suggested severity: High, CVSS 8.2.

Details

MasterService::NotifyOffloadSuccess (mooncake-store/src/master_service.cpp) processes each caller-supplied (key, StorageObjectMetadata) pair: it clears any in-flight offloading task for the key, then calls AddReplica(client_id, key, LOCAL_DISK replica{metadata.data_size, metadata.transport_endpoint, ReplicaStatus::COMPLETE}). Nothing verifies that the caller owns the key, that an offload was ever scheduled, or that the endpoint belongs to the caller.

AddReplica widens the effect: when the object has no LOCAL_DISK replica yet (the normal case for a memory-resident object) the forged replica is appended; when the object does not exist at all, the call creates fresh metadata owned by the attacker — so the master's existence oracle and replica lists now answer for keys that were never written.

In 0.3.13 the handler requires HasMountedLocalDiskSegment(client_id) when offload is enabled; this is satisfied by first sending one forged MountLocalDiskSegment(attacker_uuid, enable_offloading=true), which is itself unauthenticated. The client_id in these RPCs is a self-declared UUID with no binding to the connection.

This is distinct from CVE-2026-96763: that issue forges a segment at mount time to redirect transfers, while this one forges completed object replicas through the offload-completion path — a different method with a different missing check, operating on the object layer rather than the segment layer.

PoC

Environment: Linux, the 0.3.13.post1 wheel's mooncake_master (offload enabled) and a victim client holding a memory-resident object victim_inject_ky. The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack bodies; observed function_ids: MountLocalDiskSegment=673169545, NotifyOffloadSuccess=771250242, GetReplicaList=528413044, ExistKey=2302172937). The 0.3.13 NotifyOffloadSuccess body is (UUID, vector<OffloadTaskItem{tenant,key,size}>, vector<StorageObjectMetadata{bucket_id,offset,key_size,data_size,transport_endpoint}>):

call(MountLocalDiskSegment, uuid=ATTACKER_UUID, enable=True)          # ok

call(NotifyOffloadSuccess, uuid=ATTACKER_UUID,
     items=[OffloadTaskItem(tenant="default", key="victim_inject_ky", size=16)],
     metas=[StorageObjectMetadata(bucket_id=1, data_size=16,
                                  endpoint="192.168.105.123:39999")]) # ok
# GetReplicaList("victim_inject_ky") now shows a COMPLETE LOCAL_DISK
# replica owned by the attacker UUID with transport_endpoint
# "192.168.105.123:39999" alongside the victim's memory replica

# key spoof on a key that never existed:
call(GetReplicaList, key="spoofed_key_xyz0")   # OBJECT_NOT_FOUND (-704)
call(NotifyOffloadSuccess, ... key="spoofed_key_xyz0" ...)            # ok
call(ExistKey,     key="spoofed_key_xyz0")     # expected<bool> = true
call(GetReplicaList, key="spoofed_key_xyz0")   # attacker's UUID +
                                               # attacker endpoint, COMPLETE

All steps observed verbatim in the captured request/response hex on 0.3.13.post1; the before/after GetReplicaList pair for spoofed_key_xyz0 (-704 → completed replica at the attacker endpoint) is the complete spoof demonstration. Expected secure behavior: offload completion must be bound to an authenticated client and to a task the master scheduled, with the endpoint taken from the registered segment rather than the request.

Impact

Any network peer that can reach the master RPC port, without credentials, corrupts the persistence-tier metadata plane in two ways. Replica forgery: the master believes an attacker-controlled endpoint holds completed disk replicas of arbitrary victims' keys; any later promotion or disk-tier read that trusts this record can be served from the attacker's endpoint (attacker-supplied cache contents — an integrity attack on the read path; the end-to-end promotion fetch was not demonstrated in this report). Key spoofing: keys that never existed become "existing" with attacker-owned metadata, poisoning the existence oracle that clients and eviction rely on. The call also clears in-flight offloading tasks for arbitrary keys (the disruption variant was not claimed). Integrity of the metadata and read path; availability is not directly affected. Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:L (8.2), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4479 ### Bug Report ### Summary Mooncake Store Master's `NotifyOffloadSuccess` RPC (unauthenticated, default 0.0.0.0:50051) accepts caller-supplied offload results with no validation. An unauthenticated peer first forges `MountLocalDiskSegment` with its own UUID (satisfying the mounted-segment precondition) and then injects a COMPLETE disk-tier replica pointing at an attacker-chosen endpoint into any existing object, or fabricates metadata for keys that never existed — the master then reports those keys as existing with a completed replica at the attacker's endpoint. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release); no fix available. Suggested severity: High, CVSS 8.2. ### Details `MasterService::NotifyOffloadSuccess` (mooncake-store/src/master_service.cpp) processes each caller-supplied (key, StorageObjectMetadata) pair: it clears any in-flight offloading task for the key, then calls `AddReplica(client_id, key, LOCAL_DISK replica{metadata.data_size, metadata.transport_endpoint, ReplicaStatus::COMPLETE})`. Nothing verifies that the caller owns the key, that an offload was ever scheduled, or that the endpoint belongs to the caller. `AddReplica` widens the effect: when the object has no LOCAL_DISK replica yet (the normal case for a memory-resident object) the forged replica is appended; when the object does not exist at all, the call creates fresh metadata owned by the attacker — so the master's existence oracle and replica lists now answer for keys that were never written. In 0.3.13 the handler requires `HasMountedLocalDiskSegment(client_id)` when offload is enabled; this is satisfied by first sending one forged `MountLocalDiskSegment(attacker_uuid, enable_offloading=true)`, which is itself unauthenticated. The `client_id` in these RPCs is a self-declared UUID with no binding to the connection. This is distinct from CVE-2026-96763: that issue forges a segment at mount time to redirect transfers, while this one forges completed object replicas through the offload-completion path — a different method with a different missing check, operating on the object layer rather than the segment layer. ### PoC Environment: Linux, the 0.3.13.post1 wheel's `mooncake_master` (offload enabled) and a victim client holding a memory-resident object `victim_inject_ky`. The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack bodies; observed function_ids: MountLocalDiskSegment=673169545, NotifyOffloadSuccess=771250242, GetReplicaList=528413044, ExistKey=2302172937). The 0.3.13 `NotifyOffloadSuccess` body is (UUID, vector<OffloadTaskItem{tenant,key,size}>, vector<StorageObjectMetadata{bucket_id,offset,key_size,data_size,transport_endpoint}>): ```python call(MountLocalDiskSegment, uuid=ATTACKER_UUID, enable=True) # ok call(NotifyOffloadSuccess, uuid=ATTACKER_UUID, items=[OffloadTaskItem(tenant="default", key="victim_inject_ky", size=16)], metas=[StorageObjectMetadata(bucket_id=1, data_size=16, endpoint="192.168.105.123:39999")]) # ok # GetReplicaList("victim_inject_ky") now shows a COMPLETE LOCAL_DISK # replica owned by the attacker UUID with transport_endpoint # "192.168.105.123:39999" alongside the victim's memory replica # key spoof on a key that never existed: call(GetReplicaList, key="spoofed_key_xyz0") # OBJECT_NOT_FOUND (-704) call(NotifyOffloadSuccess, ... key="spoofed_key_xyz0" ...) # ok call(ExistKey, key="spoofed_key_xyz0") # expected<bool> = true call(GetReplicaList, key="spoofed_key_xyz0") # attacker's UUID + # attacker endpoint, COMPLETE ``` All steps observed verbatim in the captured request/response hex on 0.3.13.post1; the before/after `GetReplicaList` pair for `spoofed_key_xyz0` (-704 → completed replica at the attacker endpoint) is the complete spoof demonstration. Expected secure behavior: offload completion must be bound to an authenticated client and to a task the master scheduled, with the endpoint taken from the registered segment rather than the request. ### Impact Any network peer that can reach the master RPC port, without credentials, corrupts the persistence-tier metadata plane in two ways. Replica forgery: the master believes an attacker-controlled endpoint holds completed disk replicas of arbitrary victims' keys; any later promotion or disk-tier read that trusts this record can be served from the attacker's endpoint (attacker-supplied cache contents — an integrity attack on the read path; the end-to-end promotion fetch was not demonstrated in this report). Key spoofing: keys that never existed become "existing" with attacker-owned metadata, poisoning the existence oracle that clients and eviction rely on. The call also clears in-flight offloading tasks for arbitrary keys (the disruption variant was not claimed). Integrity of the metadata and read path; availability is not directly affected. Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:L (8.2), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:14 +00:00

bsakaguchi commented

2026-10-29 21:23:15 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 5, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4479
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 5, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4479 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:15 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 6, 2026):

Code-confirmed on current main (245e7106). Both halves of the report check out.

Replica forgery: MasterService::NotifyOffloadSuccess (mooncake-store/src/master_service.cpp:8648) clears the in-flight offload task for a key when one exists, but when none exists it still falls through to AddReplicaForRetainedClient, which appends the caller-supplied COMPLETE LOCAL_DISK replica (attacker-chosen transport_endpoint included) with no check that an offload was ever scheduled or that the caller owns the key. The only admission gate is HasMountedLocalDiskSegment(client_id) under enable_offload_ (5479), and that registry is populated by the equally unauthenticated MountLocalDiskSegment, so the precondition is self-declared end to end.

Key spoofing: for a key that never existed, AddReplicaForRetainedClient runs accessor.Create(client_id, ...) at 5484, so the master mints fresh attacker-owned metadata and starts answering ExistKey/GetReplicaList for it, exactly as the PoC's -704 to completed-replica flip shows.

One adjacent observation, not security-relevant: the OBJECT_NOT_FOUND continue in NotifyOffloadSuccess's fall-through (8806) looks unreachable from this caller, since AddReplicaForRetainedClient creates the object rather than returning that error. Dead-ish error handling worth tidying whenever this area gets its real fix.

Same root cause as #4474/#4475/#4478: no caller identity on the master RPC plane, so field-level checks inside the request bind nothing. The report's suggested shape (bind completion to a scheduled task, take the endpoint from the registered segment instead of the request) is the right one once identity exists. Severity and advisory handling I leave to the core team.

@he-yufeng commented on GitHub (Oct 6, 2026): Code-confirmed on current main (245e7106). Both halves of the report check out. Replica forgery: `MasterService::NotifyOffloadSuccess` (mooncake-store/src/master_service.cpp:8648) clears the in-flight offload task for a key when one exists, but when none exists it still falls through to `AddReplicaForRetainedClient`, which appends the caller-supplied COMPLETE LOCAL_DISK replica (attacker-chosen `transport_endpoint` included) with no check that an offload was ever scheduled or that the caller owns the key. The only admission gate is `HasMountedLocalDiskSegment(client_id)` under `enable_offload_` (5479), and that registry is populated by the equally unauthenticated `MountLocalDiskSegment`, so the precondition is self-declared end to end. Key spoofing: for a key that never existed, `AddReplicaForRetainedClient` runs `accessor.Create(client_id, ...)` at 5484, so the master mints fresh attacker-owned metadata and starts answering `ExistKey`/`GetReplicaList` for it, exactly as the PoC's -704 to completed-replica flip shows. One adjacent observation, not security-relevant: the `OBJECT_NOT_FOUND` continue in `NotifyOffloadSuccess`'s fall-through (8806) looks unreachable from this caller, since `AddReplicaForRetainedClient` creates the object rather than returning that error. Dead-ish error handling worth tidying whenever this area gets its real fix. Same root cause as #4474/#4475/#4478: no caller identity on the master RPC plane, so field-level checks inside the request bind nothing. The report's suggested shape (bind completion to a scheduled task, take the endpoint from the registered segment instead of the request) is the right one once identity exists. Severity and advisory handling I leave to the core team.

bsakaguchi referenced this issue

2026-10-29 21:32:15 +00:00

[PR #1135] [MERGED] [Misc] Add Codeowners #2034

bsakaguchi referenced this issue

2026-10-29 21:33:38 +00:00

[PR #1370] [CLOSED] [TE] feat: zmq communicator based on coro_rpc #2204