[GH-ISSUE #5037] [RFC]: [Store] Shared-DFS to MEMORY promotion for queued KV prefetch #1275

New issue

Open

opened 2026-10-29 21:24:42 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 21:24:42 +00:00

Owner

Copy link

Originally created by @lihenga on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5037

Changes proposed

Extend Mooncake's existing promotion machinery to support shared-DFS objects promoted into store-managed MEMORY for SGLang waiting-queue KV prefetch. This RFC fixes the backend contract and failure/lifetime rules before the implementation is upstreamed.

Cross-project integration and PR sequence: SGLang companion RFC #47031.

The implementation is deliberately the last feature phase, after independent correctness, I/O, health/recovery, and ReadPlan work. Discussing the contract now avoids upstreaming a request-owned pinned-arena prefetch architecture and then replacing it.

Relationship to existing work

• #4272 is merged: LOCAL_DISK/SSD prefetch-on-exist, with task registration, shared promotion execution and quotas. We propose a shared-DFS source extension, not another implementation of that feature.
• #4271 documents that implementation; #3417 discusses explicit prefetch triggers. The new trigger must reuse accepted gates, task bookkeeping, and capacity policy.
• SGLang #45128 defines queue-time prefetch/admission direction. SGLang should not manage native source leases or staging ownership.
• #4376 / #4377 remain the native ReadPlan discussion. This RFC does not introduce another ReadPlan core.
• #4780, #4927, and #4942 address related transfer drain, lease revalidation, and DFS host-read paths. Their applicable guarantees should be reused; they do not by themselves define queue-prefetch semantics.

Proposed API boundary

The downstream prototype exposes a batch trigger:

# Proposed spelling, called on a bounded background worker:
results = store.batch_prefetch_to_memory(keys)

We propose the following contract regardless of the final spelling:

• The native call may block on its worker until attempted promotions finish. It returns one result per submitted key, preserving input correspondence.
• A result distinguishes already-readable MEMORY/successful commit, missing source, unsupported capability, capacity/quota pressure, and failure using existing error conventions where possible.
• Success means MEMORY was readable when completion was observed. It does not promise residency until a later get.
• The backend adapter can use a local future/ticket tagged with the request incarnation. A new long-lived remote handle API is not required.
• Polling in SGLang must not issue a blocking promotion/start/end RPC on the scheduler thread.
• Capability detection is explicit. Old/unsupported masters or clients retain the ordinary lookup/get path; no speculative result may be treated as successful KV without a valid readable replica.

The existing prefetch-on-exist trigger remains useful for SSD. An explicit shared-DFS trigger should reuse the same accepted promotion infrastructure and access/tenant rules. We are not proposing a mandatory router protocol, new standalone prefetch service, or a second allocator.

Execution and publication

1. Select an eligible readable shared-DFS source using the existing query/access policy and exact object/source identity.
2. Read the source through the normal validated DFS path into bounded temporary staging. Keep every source/destination owner alive through actual I/O, including late completion after a reported error.
3. After staging is ready, register/revalidate the exact source and promotion task, allocate a MEMORY target, and execute the managed write/commit sequence. In particular, do not hand out a remote target write address across the potentially long DFS-read window.
4. Publish a normal MEMORY replica only after exact logical bytes, applicable checksum/version checks, and transfer completion succeed.
5. Release temporary staging and settle the task/quota exactly once when operations no longer access them. Failure must not publish a partially initialized MEMORY replica.
6. At request admission, acquire fresh ordinary load sessions and select the currently readable source. ReadPlan can consume that normal source without retaining a queue-time get session.

If source identity changed while reading/staging, publication must either prove the staged version still matches the registered object or fail safely. Reusing a bucket number or a target address is not proof of identity.

Target lifetime: deadline is not writer completion

The latest downstream prototype includes a correction for this specific failure:

A timed-out promotion allocation is reclaimed and reused, then the old writer resumes and overwrites the new object's allocation.

The proposed invariant is strict: once an executor can write a target, neither task TTL nor an RPC timeout permits target reuse until that writer is drained or otherwise fenced from the allocation. A failure path must also retain staging while native I/O can still access it.

The prototype reads DFS before target registration/allocation and conservatively retains an issued prefetch target until terminal notification. This prevents unsafe timeout reuse but has a known liveness cost: lost executors/notifications can strand capacity. We do not claim automatic recovery from that case. Any automatic reclamation must demonstrate owner-liveness/fencing or actual drain; an elapsed-time-only reaper is insufficient.

This is a merge gate and design question for the shared promotion machinery, not a reason to silently disable expiry checks for unrelated reads.

Resource, deduplication, and cancellation rules

• Use existing tenant/access checks, per-key promotion deduplication and in-flight/capacity gates. Concurrent SSD/DFS triggers must not create conflicting promotions of one object.
• Bound workers, submitted bytes, temporary staging, and outstanding tasks; memory pressure causes best-effort failure/fallback.
• Temporary staging is still needed. Promotion removes long-lived per-request queue staging, not Store MEMORY consumption or every host copy.
• Cancelling one SGLang request stops consuming its local result. It does not revoke a shared MEMORY replica or cancel another request's promotion.
• If no safe transport/I/O cancellation exists, stop further submission and retire owners after completion. Do not detach work that can access released memory.
• A promoted replica remains normally evictable after any applicable existing lease. Admission must reselect; READY is not an indefinite reservation.
• Metrics distinguish attempted/committed/failed promotion, temporary bytes and in-flight tasks. Backfill, foreground dual-write, and promotion must not be counted as the same operation.

Upstream implementation split

Independent backend work precedes this feature:

1. Bound DFS restore and preserve per-operation owners, duplicate-key correspondence, and deadline behavior.
2. Keep lease/source-identity checks, root/epoch fencing, and safe write staging/fallback.
3. Isolate DFS recovery/backfill from foreground executors, and expose agreed health/policy capabilities.
4. Continue native ReadPlan through #4377; add only required page-wise/range-reuse increments separately.
5. Submit the shared-DFS promotion increment on the accepted #4272 infrastructure with complete failure/target-lifetime tests.
6. Enable the SGLang queue/admission adapter only after the backend contract and supported release are available; forward-time preparation follows in a separate SGLang increment.

The old dedicated-arena/session-prefetch path is reference material, not a prerequisite to merge. Deployment-specific increases to staging capacity, read concurrency or fixed timeouts are not bundled into the feature.

Validation gates

These are required checks for the eventual upstream patch, not completed test claims:

• Existing MEMORY, DFS-only, missing object, mixed batch, duplicate key and partial failure results.
• Repeated/concurrent triggers, shared quota with SSD promotion, tenant/access separation, memory pressure and cancellation.
• Source removal/upsert, bucket/epoch changes and exact logical-byte/checksum validation.
• Pause a writer after target issuance, advance task expiry, and prove the target cannot be reused for another object before terminal drain/fencing.
• Timeout each DFS/read/write/commit stage, resume it late, and prove no use-after-free, cross-object write or duplicate settlement.
• Crash/lost notification behavior: document conservative retention versus a proven reclamation protocol.
• Evict MEMORY between promotion completion and admission; normal get/session selection remains correct.
• Matched end-to-end A/B: TTFT distribution, throughput, promotion success/failure, staging/Store-memory peak, and interference with foreground reads/writes.

Reference and implementation status

Prototype: lihenga/mooncake-das #115, inspected head 12d4eb24a88139f40c0c901b00f3c4dded63df45, consisting of the initial promotion change plus the target-lifetime correction. Consumer: lihenga/sglang-das #43, inspected head 234264a11f0b7393d066f9f53f733d2989cd20e2.

These are downstream prototypes, not an apply-ready current-main patch or a claim of final model/performance qualification. Historical test reports apply only to their recorded source revisions and scope. No new tests were run while preparing this RFC.

Feedback requested

1. Is extending the existing #4272 machinery with a shared-DFS source/explicit batch trigger the preferred integration point?
2. Can the issued-target lifetime invariant be expressed in current task ownership, and what proven reclamation behavior is acceptable after executor loss?
3. Does completion-as-observation plus fresh admission sessions fit the Store contract, without a queue-time residency reservation?
4. Should this increment be hosted as a follow-up to #3417, with this RFC tracking the DFS-specific safety and interface decisions?

Before submitting a new issue

• Related issues, RFCs, and the available implementation/design descriptions were reviewed.

AI assistance disclosure: Codex assisted with source/PR comparison and RFC drafting. Human review and the validation gates above remain required.

Originally created by @lihenga on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5037 ### Changes proposed Extend Mooncake's existing promotion machinery to support **shared-DFS objects promoted into store-managed MEMORY** for SGLang waiting-queue KV prefetch. This RFC fixes the backend contract and failure/lifetime rules before the implementation is upstreamed. Cross-project integration and PR sequence: [SGLang companion RFC #47031](https://github.com/sgl-project/sglang/issues/47031). The implementation is deliberately the **last feature phase**, after independent correctness, I/O, health/recovery, and ReadPlan work. Discussing the contract now avoids upstreaming a request-owned pinned-arena prefetch architecture and then replacing it. ## Relationship to existing work - [#4272](https://github.com/kvcache-ai/Mooncake/pull/4272) is merged: LOCAL_DISK/SSD prefetch-on-exist, with task registration, shared promotion execution and quotas. We propose a shared-DFS source extension, not another implementation of that feature. - [#4271](https://github.com/kvcache-ai/Mooncake/issues/4271) documents that implementation; [#3417](https://github.com/kvcache-ai/Mooncake/issues/3417) discusses explicit prefetch triggers. The new trigger must reuse accepted gates, task bookkeeping, and capacity policy. - [SGLang #45128](https://github.com/sgl-project/sglang/issues/45128) defines queue-time prefetch/admission direction. SGLang should not manage native source leases or staging ownership. - [#4376](https://github.com/kvcache-ai/Mooncake/issues/4376) / [#4377](https://github.com/kvcache-ai/Mooncake/pull/4377) remain the native ReadPlan discussion. This RFC does not introduce another ReadPlan core. - [#4780](https://github.com/kvcache-ai/Mooncake/pull/4780), [#4927](https://github.com/kvcache-ai/Mooncake/pull/4927), and [#4942](https://github.com/kvcache-ai/Mooncake/pull/4942) address related transfer drain, lease revalidation, and DFS host-read paths. Their applicable guarantees should be reused; they do not by themselves define queue-prefetch semantics. ## Proposed API boundary The downstream prototype exposes a batch trigger: ```python # Proposed spelling, called on a bounded background worker: results = store.batch_prefetch_to_memory(keys) ``` We propose the following contract regardless of the final spelling: - The native call may block on its worker until attempted promotions finish. It returns one result per submitted key, preserving input correspondence. - A result distinguishes already-readable MEMORY/successful commit, missing source, unsupported capability, capacity/quota pressure, and failure using existing error conventions where possible. - Success means MEMORY was readable when completion was observed. It does not promise residency until a later get. - The backend adapter can use a local future/ticket tagged with the request incarnation. A new long-lived remote handle API is not required. - Polling in SGLang must not issue a blocking promotion/start/end RPC on the scheduler thread. - Capability detection is explicit. Old/unsupported masters or clients retain the ordinary lookup/get path; no speculative result may be treated as successful KV without a valid readable replica. The existing prefetch-on-exist trigger remains useful for SSD. An explicit shared-DFS trigger should reuse the same accepted promotion infrastructure and access/tenant rules. We are not proposing a mandatory router protocol, new standalone prefetch service, or a second allocator. ## Execution and publication 1. Select an eligible readable shared-DFS source using the existing query/access policy and exact object/source identity. 2. Read the source through the normal validated DFS path into bounded temporary staging. Keep every source/destination owner alive through actual I/O, including late completion after a reported error. 3. After staging is ready, register/revalidate the exact source and promotion task, allocate a MEMORY target, and execute the managed write/commit sequence. In particular, do not hand out a remote target write address across the potentially long DFS-read window. 4. Publish a normal MEMORY replica only after exact logical bytes, applicable checksum/version checks, and transfer completion succeed. 5. Release temporary staging and settle the task/quota exactly once when operations no longer access them. Failure must not publish a partially initialized MEMORY replica. 6. At request admission, acquire fresh ordinary load sessions and select the currently readable source. ReadPlan can consume that normal source without retaining a queue-time get session. If source identity changed while reading/staging, publication must either prove the staged version still matches the registered object or fail safely. Reusing a bucket number or a target address is not proof of identity. ## Target lifetime: deadline is not writer completion The latest downstream prototype includes a correction for this specific failure: > A timed-out promotion allocation is reclaimed and reused, then the old writer resumes and overwrites the new object's allocation. The proposed invariant is strict: **once an executor can write a target, neither task TTL nor an RPC timeout permits target reuse until that writer is drained or otherwise fenced from the allocation.** A failure path must also retain staging while native I/O can still access it. The prototype reads DFS before target registration/allocation and conservatively retains an issued prefetch target until terminal notification. This prevents unsafe timeout reuse but has a known liveness cost: lost executors/notifications can strand capacity. We do not claim automatic recovery from that case. Any automatic reclamation must demonstrate owner-liveness/fencing or actual drain; an elapsed-time-only reaper is insufficient. This is a merge gate and design question for the shared promotion machinery, not a reason to silently disable expiry checks for unrelated reads. ## Resource, deduplication, and cancellation rules - Use existing tenant/access checks, per-key promotion deduplication and in-flight/capacity gates. Concurrent SSD/DFS triggers must not create conflicting promotions of one object. - Bound workers, submitted bytes, temporary staging, and outstanding tasks; memory pressure causes best-effort failure/fallback. - Temporary staging is still needed. Promotion removes long-lived **per-request queue staging**, not Store MEMORY consumption or every host copy. - Cancelling one SGLang request stops consuming its local result. It does not revoke a shared MEMORY replica or cancel another request's promotion. - If no safe transport/I/O cancellation exists, stop further submission and retire owners after completion. Do not detach work that can access released memory. - A promoted replica remains normally evictable after any applicable existing lease. Admission must reselect; READY is not an indefinite reservation. - Metrics distinguish attempted/committed/failed promotion, temporary bytes and in-flight tasks. Backfill, foreground dual-write, and promotion must not be counted as the same operation. ## Upstream implementation split Independent backend work precedes this feature: 1. Bound DFS restore and preserve per-operation owners, duplicate-key correspondence, and deadline behavior. 2. Keep lease/source-identity checks, root/epoch fencing, and safe write staging/fallback. 3. Isolate DFS recovery/backfill from foreground executors, and expose agreed health/policy capabilities. 4. Continue native ReadPlan through #4377; add only required page-wise/range-reuse increments separately. 5. Submit the shared-DFS promotion increment on the accepted #4272 infrastructure with complete failure/target-lifetime tests. 6. Enable the SGLang queue/admission adapter only after the backend contract and supported release are available; forward-time preparation follows in a separate SGLang increment. The old dedicated-arena/session-prefetch path is reference material, not a prerequisite to merge. Deployment-specific increases to staging capacity, read concurrency or fixed timeouts are not bundled into the feature. ## Validation gates These are required checks for the eventual upstream patch, not completed test claims: - Existing MEMORY, DFS-only, missing object, mixed batch, duplicate key and partial failure results. - Repeated/concurrent triggers, shared quota with SSD promotion, tenant/access separation, memory pressure and cancellation. - Source removal/upsert, bucket/epoch changes and exact logical-byte/checksum validation. - Pause a writer after target issuance, advance task expiry, and prove the target cannot be reused for another object before terminal drain/fencing. - Timeout each DFS/read/write/commit stage, resume it late, and prove no use-after-free, cross-object write or duplicate settlement. - Crash/lost notification behavior: document conservative retention versus a proven reclamation protocol. - Evict MEMORY between promotion completion and admission; normal get/session selection remains correct. - Matched end-to-end A/B: TTFT distribution, throughput, promotion success/failure, staging/Store-memory peak, and interference with foreground reads/writes. ## Reference and implementation status Prototype: [lihenga/mooncake-das #115](https://github.com/lihenga/mooncake-das/pull/115), inspected head **12d4eb24a88139f40c0c901b00f3c4dded63df45**, consisting of the initial promotion change plus the target-lifetime correction. Consumer: [lihenga/sglang-das #43](https://github.com/lihenga/sglang-das/pull/43), inspected head 234264a11f0b7393d066f9f53f733d2989cd20e2. These are downstream prototypes, not an apply-ready current-main patch or a claim of final model/performance qualification. Historical test reports apply only to their recorded source revisions and scope. No new tests were run while preparing this RFC. ## Feedback requested 1. Is extending the existing #4272 machinery with a shared-DFS source/explicit batch trigger the preferred integration point? 2. Can the issued-target lifetime invariant be expressed in current task ownership, and what proven reclamation behavior is acceptable after executor loss? 3. Does completion-as-observation plus fresh admission sessions fit the Store contract, without a queue-time residency reservation? 4. Should this increment be hosted as a follow-up to #3417, with this RFC tracking the DFS-specific safety and interface decisions? ### Before submitting a new issue - [x] Related issues, RFCs, and the available implementation/design descriptions were reviewed. AI assistance disclosure: Codex assisted with source/PR comparison and RFC drafting. Human review and the validation gates above remain required.

bsakaguchi commented 2026-10-29 21:24:43 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 28, 2026):

Thanks for opening this issue, @lihenga!

Field	Value
Issue	#5037
GitHub user ID	117809381
Reporter	@lihenga

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6465474274 --> @github-actions[bot] commented on GitHub (Oct 28, 2026): Thanks for opening this issue, @lihenga! | Field | Value | |-------|-------| | **Issue** | #5037 | | **GitHub user ID** | `117809381` | | **Reporter** | @lihenga | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue 2026-10-29 21:32:59 +00:00

[PR #1275] [MERGED] [doc] Restruct doc about vllm support. #2127

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1275

No description provided.