kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4475] [Bug]: Unauthenticated task assignment, queue theft, and fake completion in Mooncake Store Master #1134

New issue

Open

opened 2026-10-29 21:23:12 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:12 +00:00

Owner

Originally created by @yyymk on GitHub (Oct 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4475

Bug Report

Summary

The replication/migration task system of Mooncake Store Master is exposed on the unauthenticated RPC port (default 0.0.0.0:50051) without caller validation. CreateCopyTask/CreateMoveTask accept any object key and attacker-chosen target segments and assign the resulting work to the owner of the source segment; FetchTasks hands out task queues based on a self-declared client_id; MarkTaskToComplete closes a task when the caller claims that same client_id. Together this gives an unauthenticated peer three primitives: force real transfer work onto a victim, steal a victim's task queue, and mark tasks successful without any work. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release); no fix available. Suggested severity: High, CVSS 8.2.

Details

MasterService::CreateCopyTask(key, targets) (mooncake-store/src/master_service.cpp) validates only that each target segment exists and is allocatable. It never checks who owns key: it looks up the victim object's metadata, picks a source replica, resolves the source segment's owner via GetClientIdBySegmentName, and submits the task assigned to that owner's client_id. CreateMoveTask behaves the same. So any caller decides which object is replicated or moved and into which segments, and the work lands in the victim's queue.

The task store (mooncake-store/src/task_manager.cpp) keys everything on a caller-supplied UUID: pop_tasks(client_id, batch) returns the pending queue of whatever client_id the request names, and complete_task(client_id, task_id, ...) succeeds whenever task.assigned_client == client_id from the request. No authentication binds the claimed identity to the connection. QueryTask(task_id) returns the full task record including assigned_client, so the needed victim client_id comes from the same unauthenticated API.

Runtime-verified on 0.3.13.post1 (frames forged over coro_rpc, same transport as the Remove issue): a task created with CreateCopyTask(victim_key, [attacker_segment]) was assigned to the victim's client_id — the returned task payload names the victim's segment as source and the attacker's segment as target; FetchTasks claiming the victim's client_id handed the task to the attacker; MarkTaskToComplete then flipped it to SUCCESS two milliseconds after creation, before any copy RPC could occur (the task had also left the victim's pending queue via the theft, so the victim's worker could no longer pick it up).

PoC

Environment as for the Remove report: the 0.3.13.post1 wheel's mooncake_master on 0.0.0.0:50051, a victim store client with segment seg_victim holding object victim_task_key, and an attacker store client with its own segment seg_attacker. The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack bodies captured once by proxying a legit client, then replayed with patched fields; observed function_ids: CreateCopyTask=2013387680, QueryTask=125921908, FetchTasks=3742636100, MarkTaskToComplete=2802188309):

task_id = call(CreateCopyTask, key="victim_task_key", tenant="default",
               targets=["seg_attacker"])          # succeeds, returns task UUID
task = call(QueryTask, task_id=task_id)            # assigned_client == victim's UUID
tasks = call(FetchTasks, client_id=<victim_uuid>, batch=16)   # queue stolen
call(MarkTaskToComplete, client_id=<victim_uuid>, task_id=task_id,
     status=SUCCESS, message="")                   # no data ever moved
task2 = call(QueryTask, task_id=task_id)           # status == SUCCESS

Observed: each step returns success; the final QueryTask reports SUCCESS although the task was created two milliseconds earlier and no copy RPC ever ran. The task payload returned by FetchTasks is {"tenant_id":"default","key":"victim_task_0001","source":"<victim segment>","targets":["<attacker segment>"],...}, confirming both the forced assignment and the attacker-directed target. Expected secure behavior: task creation requires ownership of the key, and fetching/completing requires an authenticated identity matching the assignee.

Impact

Any network peer that can reach the master RPC port, without credentials: (1) submits copy/move tasks for arbitrary victims' objects — the victim's client picks them up and performs real data transfers, so an attacker drives other tenants' CPU and network at zero cost, with attacker-chosen targets including the attacker's own segments; (2) claims a victim's queue with the self-declared client_id, stalling the victim's migration and offload; (3) marks tasks successful without work, so replication the system believes happened never did — silent failure of the data-protection goal. Availability and integrity of the replication subsystem are affected; no confidentiality effect was demonstrated in this report. Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:H (8.2), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @yyymk on GitHub (Oct 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4475 ### Bug Report ### Summary The replication/migration task system of Mooncake Store Master is exposed on the unauthenticated RPC port (default 0.0.0.0:50051) without caller validation. `CreateCopyTask`/`CreateMoveTask` accept any object key and attacker-chosen target segments and assign the resulting work to the owner of the source segment; `FetchTasks` hands out task queues based on a self-declared client_id; `MarkTaskToComplete` closes a task when the caller claims that same client_id. Together this gives an unauthenticated peer three primitives: force real transfer work onto a victim, steal a victim's task queue, and mark tasks successful without any work. Runtime-confirmed on the master binary shipped in the 0.3.13.post1 wheel (latest release); no fix available. Suggested severity: High, CVSS 8.2. ### Details `MasterService::CreateCopyTask(key, targets)` (mooncake-store/src/master_service.cpp) validates only that each target segment exists and is allocatable. It never checks who owns `key`: it looks up the victim object's metadata, picks a source replica, resolves the source segment's owner via `GetClientIdBySegmentName`, and submits the task assigned to that owner's client_id. `CreateMoveTask` behaves the same. So any caller decides which object is replicated or moved and into which segments, and the work lands in the victim's queue. The task store (mooncake-store/src/task_manager.cpp) keys everything on a caller-supplied UUID: `pop_tasks(client_id, batch)` returns the pending queue of whatever client_id the request names, and `complete_task(client_id, task_id, ...)` succeeds whenever `task.assigned_client == client_id` from the request. No authentication binds the claimed identity to the connection. `QueryTask(task_id)` returns the full task record including `assigned_client`, so the needed victim client_id comes from the same unauthenticated API. Runtime-verified on 0.3.13.post1 (frames forged over coro_rpc, same transport as the Remove issue): a task created with `CreateCopyTask(victim_key, [attacker_segment])` was assigned to the victim's client_id — the returned task payload names the victim's segment as source and the attacker's segment as target; `FetchTasks` claiming the victim's client_id handed the task to the attacker; `MarkTaskToComplete` then flipped it to SUCCESS two milliseconds after creation, before any copy RPC could occur (the task had also left the victim's pending queue via the theft, so the victim's worker could no longer pick it up). ### PoC Environment as for the Remove report: the 0.3.13.post1 wheel's `mooncake_master` on 0.0.0.0:50051, a victim store client with segment `seg_victim` holding object `victim_task_key`, and an attacker store client with its own segment `seg_attacker`. The attacker forges coro_rpc frames (20-byte header, magic=21; struct_pack bodies captured once by proxying a legit client, then replayed with patched fields; observed function_ids: CreateCopyTask=2013387680, QueryTask=125921908, FetchTasks=3742636100, MarkTaskToComplete=2802188309): ```python task_id = call(CreateCopyTask, key="victim_task_key", tenant="default", targets=["seg_attacker"]) # succeeds, returns task UUID task = call(QueryTask, task_id=task_id) # assigned_client == victim's UUID tasks = call(FetchTasks, client_id=<victim_uuid>, batch=16) # queue stolen call(MarkTaskToComplete, client_id=<victim_uuid>, task_id=task_id, status=SUCCESS, message="") # no data ever moved task2 = call(QueryTask, task_id=task_id) # status == SUCCESS ``` Observed: each step returns success; the final QueryTask reports SUCCESS although the task was created two milliseconds earlier and no copy RPC ever ran. The task payload returned by FetchTasks is `{"tenant_id":"default","key":"victim_task_0001","source":"<victim segment>","targets":["<attacker segment>"],...}`, confirming both the forced assignment and the attacker-directed target. Expected secure behavior: task creation requires ownership of the key, and fetching/completing requires an authenticated identity matching the assignee. ### Impact Any network peer that can reach the master RPC port, without credentials: (1) submits copy/move tasks for arbitrary victims' objects — the victim's client picks them up and performs real data transfers, so an attacker drives other tenants' CPU and network at zero cost, with attacker-chosen targets including the attacker's own segments; (2) claims a victim's queue with the self-declared client_id, stalling the victim's migration and offload; (3) marks tasks successful without work, so replication the system believes happened never did — silent failure of the data-protection goal. Availability and integrity of the replication subsystem are affected; no confidentiality effect was demonstrated in this report. Suggested CVSS:3.1 vector AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:H (8.2), CWE-862. Affected: mooncake-store master <= 0.3.13.post1 (latest release), no patched version yet. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:12 +00:00

bsakaguchi commented

2026-10-29 21:23:13 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 5, 2026):

Thanks for opening this issue, @yyymk!

Field	Value
Issue	#4475
GitHub user ID	`133664083`
Reporter	@yyymk

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 5, 2026): Thanks for opening this issue, @yyymk! | Field | Value | |-------|-------| | **Issue** | #4475 | | **GitHub user ID** | `133664083` | | **Reporter** | @yyymk | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:13 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 5, 2026):

Code-confirmed on current main (0d1a8040).

(Edit: an earlier revision of this comment cited line numbers from my stale local branch; the references below are against main.)

CreateCopyTask (mooncake-store/src/master_service.cpp:13728) validates that each target segment exists and is allocatable, then resolves the source segment's owner through GetClientIdBySegmentName and submits the task under that owner's UUID. Who is calling never enters the picture. CreateMoveTask is the same shape. On the store side, pop_tasks and complete_task (mooncake-store/src/task_manager.cpp:58 and :113) key everything on the caller-supplied UUID, with task.assigned_client != client_id as the only guard, and QueryTask handing out assigned_client closes the loop you describe: the victim's UUID is available from the same unauthenticated API.

This shares the root cause with #4474: nothing on the master RPC plane binds a connection to an identity, so field comparisons inside the request cannot fix queue theft or fake completion. Once identity exists, an ownership check on task creation (caller must own key) and binding fetch/complete to the authenticated assignee both become meaningful. Grouping this with the master-RPC auth discussion rather than patching the checks in isolation.

@he-yufeng commented on GitHub (Oct 5, 2026): Code-confirmed on current main (0d1a8040). (Edit: an earlier revision of this comment cited line numbers from my stale local branch; the references below are against main.) `CreateCopyTask` (mooncake-store/src/master_service.cpp:13728) validates that each target segment exists and is allocatable, then resolves the source segment's owner through `GetClientIdBySegmentName` and submits the task under that owner's UUID. Who is calling never enters the picture. `CreateMoveTask` is the same shape. On the store side, `pop_tasks` and `complete_task` (mooncake-store/src/task_manager.cpp:58 and :113) key everything on the caller-supplied UUID, with `task.assigned_client != client_id` as the only guard, and `QueryTask` handing out `assigned_client` closes the loop you describe: the victim's UUID is available from the same unauthenticated API. This shares the root cause with #4474: nothing on the master RPC plane binds a connection to an identity, so field comparisons inside the request cannot fix queue theft or fake completion. Once identity exists, an ownership check on task creation (caller must own `key`) and binding fetch/complete to the authenticated assignee both become meaningful. Grouping this with the master-RPC auth discussion rather than patching the checks in isolation.

bsakaguchi referenced this issue

2026-10-29 21:32:15 +00:00

[PR #1134] [MERGED] [Store] fix compilation warning in storage backend #2033

bsakaguchi referenced this issue

2026-10-29 21:33:38 +00:00

[PR #1370] [CLOSED] [TE] feat: zmq communicator based on coro_rpc #2204