[GH-ISSUE #4478] [Bug]: Unauthenticated disk replica erasure and object deletion via EvictDiskReplica in Mooncake Store Master #1136

New issue

Open

opened 2026-10-29 21:23:14 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:14 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4478

Bug Report

Summary

Mooncake Store Master serves EvictDiskReplica and BatchEvictDiskReplica on its unauthenticated RPC port (default 0.0.0.0:50051). When called with replica_type=DISK, the master erases the disk-tier replica metadata of any object without checking caller ownership, tenant, or lease — while the same function's LOCAL_DISK branch does filter by client_id. An unauthenticated peer erases the persistence-tier replica of any key, and when the disk replica is the object's only remaining replica, the whole object is deleted. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release, with the documented --root_fs_dir persistence configuration); no fix available. Suggested severity: High, CVSS 9.1.

Details

In MasterService::EvictDiskReplica (mooncake-store/src/master_service.cpp) the replica_type selects the branch:

if (replica_type == ReplicaType::DISK) {
    metadata.EraseReplicas([](const Replica& r) { return r.is_disk_replica(); });
} else if (replica_type == ReplicaType::LOCAL_DISK) {
    metadata.EraseReplicas([&client_id](const Replica& r) {
        return r.is_local_disk_replica() && r.client_id == client_id;
    });
}

The DISK branch has no ownership, lease, or tenant check of any kind — the request carries a client_id that is never consulted for this branch. The LOCAL_DISK branch of the same function does compare the replica's client_id, showing the asymmetry is not an intended trust decision. After erasure, if (!metadata.IsValid()) accessor.Erase() deletes the whole object once no valid replica remains.

Disk replicas are created by the master's own storage backend in the documented persistence configuration (--root_fs_dir): after a memory Put, the client's PutToLocalFile path stores the object and commits it with PutEnd(ReplicaType::DISK). Deployments running with --root_fs_dir (the persistence/HA form) are therefore affected; BatchEvictDiskReplica loops the same path over a key list.

This is distinct from the pending Remove-family report and from CVE-2026-96762: different method, different asset (persistence-tier replica metadata vs object metadata or segment registration), and no force flag involved.

PoC

Environment: Linux, the 0.3.13.post1 wheel's mooncake_master started with --rpc_address=0.0.0.0 --rpc_port=50051 --root_fs_dir=<dir> --enable_offload, and a victim store client that has put objects (each object then shows replicas ['MEMORY','DISK'] via get_replica_desc). The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack body: UUID client_id, length-prefixed key, tenant, u32 replica_type where MEMORY=0, DISK=1, LOCAL_DISK=2; observed function_id 3490366507 for EvictDiskReplica, 1959913856 for the batch variant):

# control: LOCAL_DISK branch with the attacker's own UUID -> returns OK,
# but the client_id filter means nothing of the victim's is touched
call(EvictDiskReplica, uuid=ATTACKER_UUID, key="victim_dsk_0011", type=LOCAL_DISK)
# snapshot: ['MEMORY', 'DISK']  (unchanged)

# attack: the same frame with only the replica_type byte changed to DISK
call(EvictDiskReplica, uuid=ATTACKER_UUID, key="victim_dsk_0011", type=DISK)
# snapshot: ['MEMORY']  (disk replica erased)

call(BatchEvictDiskReplica, uuid=ATTACKER_UUID, keys=["victim_dsk_0012", "victim_dsk_0013"], type=DISK)
# response: [ok, ok]; both keys lose their disk replicas

Object-loss escalation (verified end to end): after the victim unmounts its memory segment, victim_dsk_only1 holds only its DISK replica (exist=1, tags=['DISK']); one forged EvictDiskReplica(..., DISK) frame later, exist=0 and the replica list is empty — the persistent object is gone from the cluster.

Expected secure behavior: erasing another client's replicas requires proving ownership; at minimum the client_id comparison the LOCAL_DISK branch already performs.

Impact

Any network peer that can reach the master RPC port — no credentials, no identity — silently erases the persistence-tier replica metadata of arbitrary keys across all tenants, batch-wipes them, and deletes objects outright once the disk replica is their last copy. In persistence deployments this defeats the durability the tier exists to provide: offloaded KV entries disappear from the cluster's index, cache hit rates collapse, and deleted entries force recomputation. Integrity and availability of stored object metadata; no confidentiality effect. The affected deployment form is the documented persistence configuration (--root_fs_dir). Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:H (9.1), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4478 ### Bug Report ### Summary Mooncake Store Master serves `EvictDiskReplica` and `BatchEvictDiskReplica` on its unauthenticated RPC port (default 0.0.0.0:50051). When called with `replica_type=DISK`, the master erases the disk-tier replica metadata of any object without checking caller ownership, tenant, or lease — while the same function's `LOCAL_DISK` branch does filter by client_id. An unauthenticated peer erases the persistence-tier replica of any key, and when the disk replica is the object's only remaining replica, the whole object is deleted. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release, with the documented `--root_fs_dir` persistence configuration); no fix available. Suggested severity: High, CVSS 9.1. ### Details In `MasterService::EvictDiskReplica` (mooncake-store/src/master_service.cpp) the `replica_type` selects the branch: ```cpp if (replica_type == ReplicaType::DISK) { metadata.EraseReplicas([](const Replica& r) { return r.is_disk_replica(); }); } else if (replica_type == ReplicaType::LOCAL_DISK) { metadata.EraseReplicas([&client_id](const Replica& r) { return r.is_local_disk_replica() && r.client_id == client_id; }); } ``` The DISK branch has no ownership, lease, or tenant check of any kind — the request carries a client_id that is never consulted for this branch. The LOCAL_DISK branch of the same function does compare the replica's client_id, showing the asymmetry is not an intended trust decision. After erasure, `if (!metadata.IsValid()) accessor.Erase()` deletes the whole object once no valid replica remains. Disk replicas are created by the master's own storage backend in the documented persistence configuration (`--root_fs_dir`): after a memory Put, the client's `PutToLocalFile` path stores the object and commits it with `PutEnd(ReplicaType::DISK)`. Deployments running with `--root_fs_dir` (the persistence/HA form) are therefore affected; `BatchEvictDiskReplica` loops the same path over a key list. This is distinct from the pending Remove-family report and from CVE-2026-96762: different method, different asset (persistence-tier replica metadata vs object metadata or segment registration), and no force flag involved. ### PoC Environment: Linux, the 0.3.13.post1 wheel's `mooncake_master` started with `--rpc_address=0.0.0.0 --rpc_port=50051 --root_fs_dir=<dir> --enable_offload`, and a victim store client that has put objects (each object then shows replicas `['MEMORY','DISK']` via `get_replica_desc`). The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack body: UUID client_id, length-prefixed key, tenant, u32 replica_type where MEMORY=0, DISK=1, LOCAL_DISK=2; observed function_id 3490366507 for EvictDiskReplica, 1959913856 for the batch variant): ```python # control: LOCAL_DISK branch with the attacker's own UUID -> returns OK, # but the client_id filter means nothing of the victim's is touched call(EvictDiskReplica, uuid=ATTACKER_UUID, key="victim_dsk_0011", type=LOCAL_DISK) # snapshot: ['MEMORY', 'DISK'] (unchanged) # attack: the same frame with only the replica_type byte changed to DISK call(EvictDiskReplica, uuid=ATTACKER_UUID, key="victim_dsk_0011", type=DISK) # snapshot: ['MEMORY'] (disk replica erased) call(BatchEvictDiskReplica, uuid=ATTACKER_UUID, keys=["victim_dsk_0012", "victim_dsk_0013"], type=DISK) # response: [ok, ok]; both keys lose their disk replicas ``` Object-loss escalation (verified end to end): after the victim unmounts its memory segment, `victim_dsk_only1` holds only its DISK replica (`exist=1, tags=['DISK']`); one forged `EvictDiskReplica(..., DISK)` frame later, `exist=0` and the replica list is empty — the persistent object is gone from the cluster. Expected secure behavior: erasing another client's replicas requires proving ownership; at minimum the client_id comparison the LOCAL_DISK branch already performs. ### Impact Any network peer that can reach the master RPC port — no credentials, no identity — silently erases the persistence-tier replica metadata of arbitrary keys across all tenants, batch-wipes them, and deletes objects outright once the disk replica is their last copy. In persistence deployments this defeats the durability the tier exists to provide: offloaded KV entries disappear from the cluster's index, cache hit rates collapse, and deleted entries force recomputation. Integrity and availability of stored object metadata; no confidentiality effect. The affected deployment form is the documented persistence configuration (`--root_fs_dir`). Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:H (9.1), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:14 +00:00

bsakaguchi commented

2026-10-29 21:23:15 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 5, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4478
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 5, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4478 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:15 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 6, 2026):

Code-confirmed on current main (245e7106).

MasterService::EvictDiskReplica (mooncake-store/src/master_service.cpp:6368) builds its target_pred exactly as quoted: the DISK branch matches r.is_disk_replica() and nothing else, while the LOCAL_DISK branch of the same lambda additionally requires the descriptor's client_id to equal the caller-supplied one (6390-6398). When the erasure leaves the object invalid, accessor.Erase() at 6467 deletes the whole object, and BatchEvictDiskReplica (6482) loops the same path. So the asymmetry the report leans on is real: ownership filtering was considered for this function and applied to exactly one of its two branches.

Same root cause as #4474/#4475: client_id is a self-declared request field with nothing binding it to the connection, so copying the LOCAL_DISK filter into the DISK branch would stop accidents, not an attacker who can read victim UUIDs through the same unauthenticated surface. One wrinkle the eventual fix has to answer: persistence-tier replicas can legitimately outlive the client that wrote them (owner gone, object still served from disk), so an ownership rule here needs to decide who may retire an orphaned DISK replica, not just compare UUIDs. Grouping this with the master-RPC identity discussion rather than patching the branch in isolation. Severity and advisory handling are the core team's call, as with the rest of this series.

@he-yufeng commented on GitHub (Oct 6, 2026): Code-confirmed on current main (245e7106). `MasterService::EvictDiskReplica` (mooncake-store/src/master_service.cpp:6368) builds its `target_pred` exactly as quoted: the DISK branch matches `r.is_disk_replica()` and nothing else, while the LOCAL_DISK branch of the same lambda additionally requires the descriptor's client_id to equal the caller-supplied one (6390-6398). When the erasure leaves the object invalid, `accessor.Erase()` at 6467 deletes the whole object, and `BatchEvictDiskReplica` (6482) loops the same path. So the asymmetry the report leans on is real: ownership filtering was considered for this function and applied to exactly one of its two branches. Same root cause as #4474/#4475: client_id is a self-declared request field with nothing binding it to the connection, so copying the LOCAL_DISK filter into the DISK branch would stop accidents, not an attacker who can read victim UUIDs through the same unauthenticated surface. One wrinkle the eventual fix has to answer: persistence-tier replicas can legitimately outlive the client that wrote them (owner gone, object still served from disk), so an ownership rule here needs to decide who may retire an orphaned DISK replica, not just compare UUIDs. Grouping this with the master-RPC identity discussion rather than patching the branch in isolation. Severity and advisory handling are the core team's call, as with the rest of this series.