kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4803] [RFC]: Per-sequence state objects and two-stage hit resolution for hybrid/SWA models #1216

New issue

Open

opened 2026-10-29 21:24:04 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:24:04 +00:00

Owner

Originally created by @sususama on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4803

Changes proposed

Motivation

Mooncake Store models every cache entry as a paged KV block addressed by a content-addressed prefix chain. That covers full-attention models well. It does not cover models where part of the persistent per-sequence cache is not paged and not addressed by block position.

DeepSeek V4.1-Flash is one such model, and Mooncake already touches this family through the Engram work (#4083, #4692). In V4.1 only the first 20 layers keep a sliding-window KV, the window is 128 tokens plus 6 tokens of speculative-decode rollback, and the retained state is a fixed-length per-sequence slot rather than a page. The open-source DeepInfer release documents the layout precisely in doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf:

Property | Value -- | -- unit | one fixed-length slot per sequence (StateBlock), not a page persisted content | first 20 layers only persistence granularity | only at 128-token boundaries, and only after the last prefill chunk ring length | 134 (window 128 + rollback 6) on the inference side; 396 on the prefill side bytes per sequence | 20 × 134 × 528 B = 1,415,040 B (MXFP8) medium | host memory of the machine that produced it, plus a directory of which machine holds which key

The same shape appears in Gemma3n/Gemma4, Qwen4-Exp and the rest of the hybrid stack that the SGLang adapters in #4692 already cover.

The object model and the hit semantics are the two halves of one problem, which is why they are in one RFC:

Without a state object kind, the hit predicate in stage B has nothing to ask about.

Without the predicate, a present block with an absent state resolves to a false long hit, and the engine pays a full prefill for it.

Why the current model cannot express it

The key is wrong. A block object is keyed by a hash chain and means "the prefix up to block N". A state slot is valid for an exact aligned prefix length. Two prefixes sharing blocks 0..N-1 share paged KV but do not necessarily share state validity, because state is only written at aligned boundaries.

The value shape is wrong. It is per-sequence and fixed length, not block_size tokens of paged KV.

Partial validity is inherent. State can be absent while the corresponding block is present, and that is a normal outcome, not a transient failure to be retried.

The placement policy is different. ~1.4 MB per sequence is small enough that replica-in-host-memory plus a cross-machine pull is the right policy, which is nothing like bulk KV placement.

Hit length is decided by one signal only. Today the block chain decides both "how far" and "usable", so there is no way to report "the blocks are here but the tail is not reusable".

Proposed design

A. State object kind

Key = (content hash of the prefix at an aligned length, aligned length). Value = opaque fixed-length slot with a declared state_layout (per-layer byte offsets).

state_layout is registered per model/checkpoint like a schema, so Store can validate size and reject mismatched producers instead of serving garbage.

PUT/GET take a boundary_aligned_length and refuse non-aligned writes, matching the engine-side rule that state is only written on aligned boundaries.

A state object has no prefix-extension semantics: it can never answer a hit for a longer prefix, only avoid recomputation for the exact prefix it was written for.

B. Two-stage hit resolution

The block chain still decides the hit length.

State presence decides only whether the last block is usable.

When state is missing, stepping back one block lets the engine recompute a bounded amount instead of the whole prefix.

The engine-side counterpart is "replay only the last window tokens, with the attention window truncated inside the replay segment", which makes the recompute cost independent of context length. That stays engine work; this RFC is about the storage shape and the reported hit length.

C. Lifetime and placement

TTL may differ from block KV. The reference deployment uses 3 days for agent traffic and 7 days for chatbot traffic, while the bulk store runs a longer LRU-based retention; the two lifetimes are managed separately.

Eviction coupling between a state object and its block prefix is an open question below, not an assumption here.

The payload is small and node-local in the reference design, with a directory recording which machine holds which key. Whether that directory is the Master or a side structure is an open question.

What this unlocks

Without it, a hybrid/SWA model on Mooncake gets either (a) SWA bytes smuggled into a block object, which inflates payload and breaks block reuse, or (b) no prefix reuse at all for the layers that need it. With it, prefix reuse for these models costs ~1.4 MB per cached sequence and the router stops advertising matches the engine cannot use.

The asymmetry is the reason to bother: a false long hit costs a full prefill, a conservative short hit costs one block of recompute.

Relationship to existing work

#4083 / #4692: Engram tables are already a non-block object kind in Store, so the "second object kind" precedent exists.

#4242: the state object is small and contiguous, so it needs no scatter path; the transport side has no new requirement.

#131 discusses cache hit rate under cache-aware scheduling; that is a different cause (scheduling), but the router-side effect here is similar in spirit.

Phasing

Store: state object kind, layout registry, aligned PUT/GET validation.

Conductor: the second predicate in the prefix index and the adjusted hit length returned to routers.

Engine adapters: write state on aligned boundaries, replay a bounded suffix on a short hit.

Steps 1 and 2 are independently reviewable; they are one RFC because step 2 is meaningless without step 1, and the step-back granularity in step 2 depends on the alignment rule chosen in step 1.

Non-goals

Model-side logic (when to write, what to drop, how to recompute) stays in the engine.

No change to paged KV addressing or to the block hash recipe.

No decision here on where state replicas live; that is placement policy.

Not proposing a new transport; the existing path already covers a small contiguous object.

Open questions

Should state live in Store's object namespace, or in a separate lightweight directory that only maps (prefix, aligned length) to an owner?

Is a per-model state_layout registry acceptable, or should the value stay fully opaque with size validation only?

Eviction: should a state object be evicted together with its block prefix, or independently?

Should the step-back be one block or ceil(window / block_size) blocks? In the reference implementation state is stored at 128-token boundaries with a 128-token window, so one block is enough there; the general rule needs a definition.

Who owns the "window + rollback" constant — model config registered with the Store, or the engine passing it per request?

If state is absent but the engine could rebuild it cheaply from the block chain, should the Store report L1 and let the engine decide? (Pro: fewer conservative step-backs. Con: the router no longer knows the real cost.)

References

deepseek-ai/DeepInfer (Apache-2.0): deepinfer/worker/cache_engine.py (StateBlock, select_state_slice_to_store, load_state_from_store_to_prefill_cache) and doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf.

Searched existing issues: no current issue covers non-paged per-sequence state, and none covers hit-length resolution for models that need it.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @sususama on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4803 ### Changes proposed <h2>Motivation</h2><p>Mooncake Store models every cache entry as a paged KV block addressed by a content-addressed prefix chain. That covers full-attention models well. It does not cover models where part of the persistent per-sequence cache is <strong>not paged</strong> and <strong>not</strong> addressed by block position.</p><p>DeepSeek V4.1-Flash is one such model, and Mooncake already touches this family through the Engram work (#4083, #4692). In V4.1 only the first 20 layers keep a sliding-window KV, the window is 128 tokens plus 6 tokens of speculative-decode rollback, and the retained state is a fixed-length per-sequence slot rather than a page. The open-source DeepInfer release documents the layout precisely in <code>doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf</code>:</p> Property | Value -- | -- unit | one fixed-length slot per sequence (StateBlock), not a page persisted content | first 20 layers only persistence granularity | only at 128-token boundaries, and only after the last prefill chunk ring length | 134 (window 128 + rollback 6) on the inference side; 396 on the prefill side bytes per sequence | 20 × 134 × 528 B = 1,415,040 B (MXFP8) medium | host memory of the machine that produced it, plus a directory of which machine holds which key <p>The same shape appears in Gemma3n/Gemma4, Qwen4-Exp and the rest of the hybrid stack that the SGLang adapters in #4692 already cover.</p><p>The object model and the hit semantics are the two halves of one problem, which is why they are in one RFC:</p><ul><li>Without a state object kind, the hit predicate in stage B has nothing to ask about.</li></ul><ul><li>Without the predicate, a present block with an absent state resolves to a <strong>false long hit</strong>, and the engine pays a full prefill for it.</li></ul><h2>Why the current model cannot express it</h2><ol><li><strong>The key is wrong.</strong> A block object is keyed by a hash chain and means "the prefix up to block N". A state slot is valid for an exact aligned prefix length. Two prefixes sharing blocks 0..N-1 share paged KV but do not necessarily share state validity, because state is only written at aligned boundaries.</li></ol><ol><li><strong>The value shape is wrong.</strong> It is per-sequence and fixed length, not <code>block_size</code> tokens of paged KV.</li></ol><ol><li><strong>Partial validity is inherent.</strong> State can be absent while the corresponding block is present, and that is a normal outcome, not a transient failure to be retried.</li></ol><ol><li><strong>The placement policy is different.</strong> ~1.4 MB per sequence is small enough that replica-in-host-memory plus a cross-machine pull is the right policy, which is nothing like bulk KV placement.</li></ol><ol><li><strong>Hit length is decided by one signal only.</strong> Today the block chain decides both "how far" and "usable", so there is no way to report "the blocks are here but the tail is not reusable".</li></ol><h2>Proposed design</h2><h3>A. State object kind</h3><ul><li>Key = (content hash of the prefix at an aligned length, aligned length). Value = opaque fixed-length slot with a declared <code>state_layout</code> (per-layer byte offsets).</li></ul><ul><li><code>state_layout</code> is registered per model/checkpoint like a schema, so Store can validate size and reject mismatched producers instead of serving garbage.</li></ul><ul><li>PUT/GET take a <code>boundary_aligned_length</code> and refuse non-aligned writes, matching the engine-side rule that state is only written on aligned boundaries.</li></ul><ul><li>A state object has <strong>no prefix-extension semantics</strong>: it can never answer a hit for a longer prefix, only avoid recomputation for the exact prefix it was written for.</li></ul><h3>B. Two-stage hit resolution</h3><pre><code></code></pre><ul><li>The block chain still decides the hit length.</li></ul><ul><li>State presence decides only whether the last block is usable.</li></ul><ul><li>When state is missing, stepping back one block lets the engine recompute a bounded amount instead of the whole prefix.</li></ul><p>The engine-side counterpart is "replay only the last <code>window</code> tokens, with the attention window truncated inside the replay segment", which makes the recompute cost independent of context length. That stays engine work; this RFC is about the storage shape and the reported hit length.</p><h3>C. Lifetime and placement</h3><ul><li>TTL may differ from block KV. The reference deployment uses 3 days for agent traffic and 7 days for chatbot traffic, while the bulk store runs a longer LRU-based retention; the two lifetimes are managed separately.</li></ul><ul><li>Eviction coupling between a state object and its block prefix is an open question below, not an assumption here.</li></ul><ul><li>The payload is small and node-local in the reference design, with a directory recording which machine holds which key. Whether that directory is the Master or a side structure is an open question.</li></ul><h2>What this unlocks</h2><p>Without it, a hybrid/SWA model on Mooncake gets either (a) SWA bytes smuggled into a block object, which inflates payload and breaks block reuse, or (b) no prefix reuse at all for the layers that need it. With it, prefix reuse for these models costs ~1.4 MB per cached sequence and the router stops advertising matches the engine cannot use.</p><p>The asymmetry is the reason to bother: a false long hit costs a full prefill, a conservative short hit costs one block of recompute.</p><h2>Relationship to existing work</h2><ul><li>#4083 / #4692: Engram tables are already a non-block object kind in Store, so the "second object kind" precedent exists.</li></ul><ul><li>#4242: the state object is small and contiguous, so it needs no scatter path; the transport side has no new requirement.</li></ul><ul><li>#131 discusses cache hit rate under cache-aware scheduling; that is a different cause (scheduling), but the router-side effect here is similar in spirit.</li></ul><h2>Phasing</h2><ol><li><strong>Store</strong>: state object kind, layout registry, aligned PUT/GET validation.</li></ol><ol><li><strong>Conductor</strong>: the second predicate in the prefix index and the adjusted hit length returned to routers.</li></ol><ol><li><strong>Engine adapters</strong>: write state on aligned boundaries, replay a bounded suffix on a short hit.</li></ol><p>Steps 1 and 2 are independently reviewable; they are one RFC because step 2 is meaningless without step 1, and the step-back granularity in step 2 depends on the alignment rule chosen in step 1.</p><h2>Non-goals</h2><ul><li>Model-side logic (when to write, what to drop, how to recompute) stays in the engine.</li></ul><ul><li>No change to paged KV addressing or to the block hash recipe.</li></ul><ul><li>No decision here on where state replicas live; that is placement policy.</li></ul><ul><li>Not proposing a new transport; the existing path already covers a small contiguous object.</li></ul><h2>Open questions</h2><ol><li>Should state live in Store's object namespace, or in a separate lightweight directory that only maps (prefix, aligned length) to an owner?</li></ol><ol><li>Is a per-model <code>state_layout</code> registry acceptable, or should the value stay fully opaque with size validation only?</li></ol><ol><li>Eviction: should a state object be evicted together with its block prefix, or independently?</li></ol><ol><li>Should the step-back be one block or <code>ceil(window / block_size)</code> blocks? In the reference implementation state is stored at 128-token boundaries with a 128-token window, so one block is enough there; the general rule needs a definition.</li></ol><ol><li>Who owns the "window + rollback" constant — model config registered with the Store, or the engine passing it per request?</li></ol><ol><li>If state is absent but the engine could rebuild it cheaply from the block chain, should the Store report <code>L1</code> and let the engine decide? (Pro: fewer conservative step-backs. Con: the router no longer knows the real cost.)</li></ol><h2>References</h2><ul><li>deepseek-ai/DeepInfer (Apache-2.0): <code>deepinfer/worker/cache_engine.py</code> (<code>StateBlock</code>, <code>select_state_slice_to_store</code>, <code>load_state_from_store_to_prefill_cache</code>) and <code>doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf</code>.</li></ul><ul><li>Searched existing issues: no current issue covers non-paged per-sequence state, and none covers hit-length resolution for models that need it.</li></ul><sc-clipboard block-data="[{"blockId":"73gQKHG2RKqFPjpSOMqPMY","blockSubtree":{"block":{"73gQKHG2RKqFPjpSOMqPMY":{"role":"editor","value":{"id":"73gQKHG2RKqFPjpSOMqPMY","parentId":"","type":"header2","props":{"title":[["Motivation",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"ZKX58lNH4sDpnyvPsAvli7","blockSubtree":{"block":{"ZKX58lNH4sDpnyvPsAvli7":{"role":"editor","value":{"id":"ZKX58lNH4sDpnyvPsAvli7","parentId":"","type":"text","props":{"title":[["Mooncake Store models every cache entry as a paged KV block addressed by a content-addressed prefix chain. That covers full-attention models well. It does not cover models where part of the persistent per-sequence cache is ",null],["not paged",[["b"]]],[" and ",null],["not",[["b"]]],[" addressed by block position.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"aCVyOgq9pAvoVhssuC6k10","blockSubtree":{"block":{"aCVyOgq9pAvoVhssuC6k10":{"role":"editor","value":{"id":"aCVyOgq9pAvoVhssuC6k10","parentId":"","type":"text","props":{"title":[["DeepSeek V4.1-Flash is one such model, and Mooncake already touches this family through the Engram work (#4083, #4692). In V4.1 only the first 20 layers keep a sliding-window KV, the window is 128 tokens plus 6 tokens of speculative-decode rollback, and the retained state is a fixed-length per-sequence slot rather than a page. The open-source DeepInfer release documents the layout precisely in ",null],["doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf",[["c"]]],[":",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"C3dzVGGTjG9NyGAl6Bmw0l","blockSubtree":{"block":{"C3dzVGGTjG9NyGAl6Bmw0l":{"role":"editor","value":{"id":"C3dzVGGTjG9NyGAl6Bmw0l","parentId":"","type":"simple_table","props":{"tableColumnOrder":["ffewgMo5","YWZO3nM9"],"tableColumnWidthMap":{"ffewgMo5":120,"YWZO3nM9":120},"tableRowHeader":true},"children":["lgZgApM9Iam6KPHkHRfoTD","pNYI3X4wUwWSGCEhkMFOUf","hAr9EqLsHmkBgBFsi2PLmz","dHPmSRlesSfgax4rMziviR","MovXBzSvup1lzs84piYweQ","ZiF2lmZV4RQlMTwOBABsl9","J6Ckcok9T4L5TSx7BOzmbb"],"parentTable":"block","enabled":true}},"lgZgApM9Iam6KPHkHRfoTD":{"role":"editor","value":{"id":"lgZgApM9Iam6KPHkHRfoTD","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["TEpm9Gakg23vofECfAmpKu","USaC5Aci2HYJ3Ss0gOspsv"],"parentTable":"block","enabled":true}},"TEpm9Gakg23vofECfAmpKu":{"role":"editor","value":{"id":"TEpm9Gakg23vofECfAmpKu","parentId":"lgZgApM9Iam6KPHkHRfoTD","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["lewb9It8bETlnplt1TvFEl"],"parentTable":"block","enabled":true}},"lewb9It8bETlnplt1TvFEl":{"role":"editor","value":{"id":"lewb9It8bETlnplt1TvFEl","parentId":"TEpm9Gakg23vofECfAmpKu","type":"text","props":{"title":[["Property",null]]},"children":[],"parentTable":"block","enabled":true}},"USaC5Aci2HYJ3Ss0gOspsv":{"role":"editor","value":{"id":"USaC5Aci2HYJ3Ss0gOspsv","parentId":"lgZgApM9Iam6KPHkHRfoTD","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["4y47cW87yxwVqPwK7Py7m5"],"parentTable":"block","enabled":true}},"4y47cW87yxwVqPwK7Py7m5":{"role":"editor","value":{"id":"4y47cW87yxwVqPwK7Py7m5","parentId":"USaC5Aci2HYJ3Ss0gOspsv","type":"text","props":{"title":[["Value",null]]},"children":[],"parentTable":"block","enabled":true}},"pNYI3X4wUwWSGCEhkMFOUf":{"role":"editor","value":{"id":"pNYI3X4wUwWSGCEhkMFOUf","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["aQXQAjx15vYJMcqBEEYkY4","6XWaErUH1hXUbNqgd021E9"],"parentTable":"block","enabled":true}},"aQXQAjx15vYJMcqBEEYkY4":{"role":"editor","value":{"id":"aQXQAjx15vYJMcqBEEYkY4","parentId":"pNYI3X4wUwWSGCEhkMFOUf","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["VzV0V1hN55yPSGN7LNZsZI"],"parentTable":"block","enabled":true}},"VzV0V1hN55yPSGN7LNZsZI":{"role":"editor","value":{"id":"VzV0V1hN55yPSGN7LNZsZI","parentId":"aQXQAjx15vYJMcqBEEYkY4","type":"text","props":{"title":[["unit",null]]},"children":[],"parentTable":"block","enabled":true}},"6XWaErUH1hXUbNqgd021E9":{"role":"editor","value":{"id":"6XWaErUH1hXUbNqgd021E9","parentId":"pNYI3X4wUwWSGCEhkMFOUf","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["FpfTtzDeLGOCVOO4fXPINY"],"parentTable":"block","enabled":true}},"FpfTtzDeLGOCVOO4fXPINY":{"role":"editor","value":{"id":"FpfTtzDeLGOCVOO4fXPINY","parentId":"6XWaErUH1hXUbNqgd021E9","type":"text","props":{"title":[["one fixed-length slot per sequence (",null],["StateBlock",[["c"]]],["), not a page",null]]},"children":[],"parentTable":"block","enabled":true}},"hAr9EqLsHmkBgBFsi2PLmz":{"role":"editor","value":{"id":"hAr9EqLsHmkBgBFsi2PLmz","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["Z85iZslvKFswF1AwKQAJRT","7UaQCtbY9MmDJy1h8iam8l"],"parentTable":"block","enabled":true}},"Z85iZslvKFswF1AwKQAJRT":{"role":"editor","value":{"id":"Z85iZslvKFswF1AwKQAJRT","parentId":"hAr9EqLsHmkBgBFsi2PLmz","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["CqtcJkPf7rLVEfwXqzfaPQ"],"parentTable":"block","enabled":true}},"CqtcJkPf7rLVEfwXqzfaPQ":{"role":"editor","value":{"id":"CqtcJkPf7rLVEfwXqzfaPQ","parentId":"Z85iZslvKFswF1AwKQAJRT","type":"text","props":{"title":[["persisted content",null]]},"children":[],"parentTable":"block","enabled":true}},"7UaQCtbY9MmDJy1h8iam8l":{"role":"editor","value":{"id":"7UaQCtbY9MmDJy1h8iam8l","parentId":"hAr9EqLsHmkBgBFsi2PLmz","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["yEx7RGbQUqS9DIQXEv85ZN"],"parentTable":"block","enabled":true}},"yEx7RGbQUqS9DIQXEv85ZN":{"role":"editor","value":{"id":"yEx7RGbQUqS9DIQXEv85ZN","parentId":"7UaQCtbY9MmDJy1h8iam8l","type":"text","props":{"title":[["first 20 layers only",null]]},"children":[],"parentTable":"block","enabled":true}},"dHPmSRlesSfgax4rMziviR":{"role":"editor","value":{"id":"dHPmSRlesSfgax4rMziviR","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["0rMrpdTRU17fYuvlHO1Tqg","y2xMHcoKHJfTXX3OEZQoPM"],"parentTable":"block","enabled":true}},"0rMrpdTRU17fYuvlHO1Tqg":{"role":"editor","value":{"id":"0rMrpdTRU17fYuvlHO1Tqg","parentId":"dHPmSRlesSfgax4rMziviR","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["BJ7tVudMHSncMPl5ydEFeq"],"parentTable":"block","enabled":true}},"BJ7tVudMHSncMPl5ydEFeq":{"role":"editor","value":{"id":"BJ7tVudMHSncMPl5ydEFeq","parentId":"0rMrpdTRU17fYuvlHO1Tqg","type":"text","props":{"title":[["persistence granularity",null]]},"children":[],"parentTable":"block","enabled":true}},"y2xMHcoKHJfTXX3OEZQoPM":{"role":"editor","value":{"id":"y2xMHcoKHJfTXX3OEZQoPM","parentId":"dHPmSRlesSfgax4rMziviR","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["yYwKTgMPzSqqm4i2a08Cgf"],"parentTable":"block","enabled":true}},"yYwKTgMPzSqqm4i2a08Cgf":{"role":"editor","value":{"id":"yYwKTgMPzSqqm4i2a08Cgf","parentId":"y2xMHcoKHJfTXX3OEZQoPM","type":"text","props":{"title":[["only at 128-token boundaries, and only after the last prefill chunk",null]]},"children":[],"parentTable":"block","enabled":true}},"MovXBzSvup1lzs84piYweQ":{"role":"editor","value":{"id":"MovXBzSvup1lzs84piYweQ","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["zvoWh9XqBR68wCxsZf0iB1","08EHLJPcbaEDIYuZXFIjD8"],"parentTable":"block","enabled":true}},"zvoWh9XqBR68wCxsZf0iB1":{"role":"editor","value":{"id":"zvoWh9XqBR68wCxsZf0iB1","parentId":"MovXBzSvup1lzs84piYweQ","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["jSN45Aost9TH1iY4HtOSbC"],"parentTable":"block","enabled":true}},"jSN45Aost9TH1iY4HtOSbC":{"role":"editor","value":{"id":"jSN45Aost9TH1iY4HtOSbC","parentId":"zvoWh9XqBR68wCxsZf0iB1","type":"text","props":{"title":[["ring length",null]]},"children":[],"parentTable":"block","enabled":true}},"08EHLJPcbaEDIYuZXFIjD8":{"role":"editor","value":{"id":"08EHLJPcbaEDIYuZXFIjD8","parentId":"MovXBzSvup1lzs84piYweQ","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["vnrJnHiO0jG0saS3Z7pgIc"],"parentTable":"block","enabled":true}},"vnrJnHiO0jG0saS3Z7pgIc":{"role":"editor","value":{"id":"vnrJnHiO0jG0saS3Z7pgIc","parentId":"08EHLJPcbaEDIYuZXFIjD8","type":"text","props":{"title":[["134 (window 128 + rollback 6) on the inference side; 396 on the prefill side",null]]},"children":[],"parentTable":"block","enabled":true}},"ZiF2lmZV4RQlMTwOBABsl9":{"role":"editor","value":{"id":"ZiF2lmZV4RQlMTwOBABsl9","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["zATkLzIc3sRaTnsvyPEBOl","4D4H4xZoob5ocXyIw7DbAU"],"parentTable":"block","enabled":true}},"zATkLzIc3sRaTnsvyPEBOl":{"role":"editor","value":{"id":"zATkLzIc3sRaTnsvyPEBOl","parentId":"ZiF2lmZV4RQlMTwOBABsl9","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["ujUTuQi6IgpTRILMEByYJk"],"parentTable":"block","enabled":true}},"ujUTuQi6IgpTRILMEByYJk":{"role":"editor","value":{"id":"ujUTuQi6IgpTRILMEByYJk","parentId":"zATkLzIc3sRaTnsvyPEBOl","type":"text","props":{"title":[["bytes per sequence",null]]},"children":[],"parentTable":"block","enabled":true}},"4D4H4xZoob5ocXyIw7DbAU":{"role":"editor","value":{"id":"4D4H4xZoob5ocXyIw7DbAU","parentId":"ZiF2lmZV4RQlMTwOBABsl9","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["YDQIOOKR6xnXjQN6h7DFh4"],"parentTable":"block","enabled":true}},"YDQIOOKR6xnXjQN6h7DFh4":{"role":"editor","value":{"id":"YDQIOOKR6xnXjQN6h7DFh4","parentId":"4D4H4xZoob5ocXyIw7DbAU","type":"text","props":{"title":[["20 × 134 × 528 B = 1,415,040 B (MXFP8)",null]]},"children":[],"parentTable":"block","enabled":true}},"J6Ckcok9T4L5TSx7BOzmbb":{"role":"editor","value":{"id":"J6Ckcok9T4L5TSx7BOzmbb","parentId":"C3dzVGGTjG9NyGAl6Bmw0l","type":"simple_table_row","props":{},"children":["LaFDipZugLqVjwavFscNn8","dEerS1ZgwYAJGMze16nr2A"],"parentTable":"block","enabled":true}},"LaFDipZugLqVjwavFscNn8":{"role":"editor","value":{"id":"LaFDipZugLqVjwavFscNn8","parentId":"J6Ckcok9T4L5TSx7BOzmbb","type":"simple_table_cell","props":{"columnKey":"ffewgMo5"},"children":["f8vNOkRrtQLisNIaOWK02A"],"parentTable":"block","enabled":true}},"f8vNOkRrtQLisNIaOWK02A":{"role":"editor","value":{"id":"f8vNOkRrtQLisNIaOWK02A","parentId":"LaFDipZugLqVjwavFscNn8","type":"text","props":{"title":[["medium",null]]},"children":[],"parentTable":"block","enabled":true}},"dEerS1ZgwYAJGMze16nr2A":{"role":"editor","value":{"id":"dEerS1ZgwYAJGMze16nr2A","parentId":"J6Ckcok9T4L5TSx7BOzmbb","type":"simple_table_cell","props":{"columnKey":"YWZO3nM9"},"children":["ZENAc8G9bvceh8HTTfcuAA"],"parentTable":"block","enabled":true}},"ZENAc8G9bvceh8HTTfcuAA":{"role":"editor","value":{"id":"ZENAc8G9bvceh8HTTfcuAA","parentId":"dEerS1ZgwYAJGMze16nr2A","type":"text","props":{"title":[["host memory of the machine that produced it, plus a directory of which machine holds which key",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"ue3zxBTnLdd3c6KOYYrXHn","blockSubtree":{"block":{"ue3zxBTnLdd3c6KOYYrXHn":{"role":"editor","value":{"id":"ue3zxBTnLdd3c6KOYYrXHn","parentId":"","type":"text","props":{"title":[["The same shape appears in Gemma3n/Gemma4, Qwen4-Exp and the rest of the hybrid stack that the SGLang adapters in #4692 already cover.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"5ZMslE7n2jE0hXIrgU1krU","blockSubtree":{"block":{"5ZMslE7n2jE0hXIrgU1krU":{"role":"editor","value":{"id":"5ZMslE7n2jE0hXIrgU1krU","parentId":"","type":"text","props":{"title":[["The object model and the hit semantics are the two halves of one problem, which is why they are in one RFC:",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"rI3t6egbaUFcE1D8THmYaY","blockSubtree":{"block":{"rI3t6egbaUFcE1D8THmYaY":{"role":"editor","value":{"id":"rI3t6egbaUFcE1D8THmYaY","parentId":"","type":"bulleted_list","props":{"title":[["Without a state object kind, the hit predicate in stage B has nothing to ask about.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"6QMjWYJyW4ooO0wUENRzo3","blockSubtree":{"block":{"6QMjWYJyW4ooO0wUENRzo3":{"role":"editor","value":{"id":"6QMjWYJyW4ooO0wUENRzo3","parentId":"","type":"bulleted_list","props":{"title":[["Without the predicate, a present block with an absent state resolves to a ",null],["false long hit",[["b"]]],[", and the engine pays a full prefill for it.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"sDZj8IIpERi8uiI370Qvvv","blockSubtree":{"block":{"sDZj8IIpERi8uiI370Qvvv":{"role":"editor","value":{"id":"sDZj8IIpERi8uiI370Qvvv","parentId":"","type":"header2","props":{"title":[["Why the current model cannot express it",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"XGCdrCaYRpq52TbIaiFLbr","blockSubtree":{"block":{"XGCdrCaYRpq52TbIaiFLbr":{"role":"editor","value":{"id":"XGCdrCaYRpq52TbIaiFLbr","parentId":"","type":"numbered_list","props":{"title":[["The key is wrong.",[["b"]]],[" A block object is keyed by a hash chain and means \"the prefix up to block N\". A state slot is valid for an exact aligned prefix length. Two prefixes sharing blocks 0..N-1 share paged KV but do not necessarily share state validity, because state is only written at aligned boundaries.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"hrtfJGWWuRfkkwAXKHf6QE","blockSubtree":{"block":{"hrtfJGWWuRfkkwAXKHf6QE":{"role":"editor","value":{"id":"hrtfJGWWuRfkkwAXKHf6QE","parentId":"","type":"numbered_list","props":{"title":[["The value shape is wrong.",[["b"]]],[" It is per-sequence and fixed length, not ",null],["block_size",[["c"]]],[" tokens of paged KV.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"kT8kvgHRbWQI0yZF7rsSVH","blockSubtree":{"block":{"kT8kvgHRbWQI0yZF7rsSVH":{"role":"editor","value":{"id":"kT8kvgHRbWQI0yZF7rsSVH","parentId":"","type":"numbered_list","props":{"title":[["Partial validity is inherent.",[["b"]]],[" State can be absent while the corresponding block is present, and that is a normal outcome, not a transient failure to be retried.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"W6XBZWG83MLbRt6XBPqHpk","blockSubtree":{"block":{"W6XBZWG83MLbRt6XBPqHpk":{"role":"editor","value":{"id":"W6XBZWG83MLbRt6XBPqHpk","parentId":"","type":"numbered_list","props":{"title":[["The placement policy is different.",[["b"]]],[" ~1.4 MB per sequence is small enough that replica-in-host-memory plus a cross-machine pull is the right policy, which is nothing like bulk KV placement.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"1d5YHqQRsid3OG9ZfyfgwX","blockSubtree":{"block":{"1d5YHqQRsid3OG9ZfyfgwX":{"role":"editor","value":{"id":"1d5YHqQRsid3OG9ZfyfgwX","parentId":"","type":"numbered_list","props":{"title":[["Hit length is decided by one signal only.",[["b"]]],[" Today the block chain decides both \"how far\" and \"usable\", so there is no way to report \"the blocks are here but the tail is not reusable\".",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"4nll3NLYkIObs0YtznHhyV","blockSubtree":{"block":{"4nll3NLYkIObs0YtznHhyV":{"role":"editor","value":{"id":"4nll3NLYkIObs0YtznHhyV","parentId":"","type":"header2","props":{"title":[["Proposed design",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"tmGYFomJ3DljO8xAemO4N1","blockSubtree":{"block":{"tmGYFomJ3DljO8xAemO4N1":{"role":"editor","value":{"id":"tmGYFomJ3DljO8xAemO4N1","parentId":"","type":"header3","props":{"title":[["A. State object kind",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"g4QERZ5JFUEo1tmlGhb3Fy","blockSubtree":{"block":{"g4QERZ5JFUEo1tmlGhb3Fy":{"role":"editor","value":{"id":"g4QERZ5JFUEo1tmlGhb3Fy","parentId":"","type":"bulleted_list","props":{"title":[["Key = (content hash of the prefix at an aligned length, aligned length). Value = opaque fixed-length slot with a declared ",null],["state_layout",[["c"]]],[" (per-layer byte offsets).",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"gDHJ6j4Ymd5EVl0pITGaWS","blockSubtree":{"block":{"gDHJ6j4Ymd5EVl0pITGaWS":{"role":"editor","value":{"id":"gDHJ6j4Ymd5EVl0pITGaWS","parentId":"","type":"bulleted_list","props":{"title":[["state_layout",[["c"]]],[" is registered per model/checkpoint like a schema, so Store can validate size and reject mismatched producers instead of serving garbage.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"DI8DTiuzxCFr9050hhriQN","blockSubtree":{"block":{"DI8DTiuzxCFr9050hhriQN":{"role":"editor","value":{"id":"DI8DTiuzxCFr9050hhriQN","parentId":"","type":"bulleted_list","props":{"title":[["PUT/GET take a ",null],["boundary_aligned_length",[["c"]]],[" and refuse non-aligned writes, matching the engine-side rule that state is only written on aligned boundaries.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"tELdKRV1TbhbZIR9FhkICs","blockSubtree":{"block":{"tELdKRV1TbhbZIR9FhkICs":{"role":"editor","value":{"id":"tELdKRV1TbhbZIR9FhkICs","parentId":"","type":"bulleted_list","props":{"title":[["A state object has ",null],["no prefix-extension semantics",[["b"]]],[": it can never answer a hit for a longer prefix, only avoid recomputation for the exact prefix it was written for.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"3NlLKqo7sFNH3f76LLlyHl","blockSubtree":{"block":{"3NlLKqo7sFNH3f76LLlyHl":{"role":"editor","value":{"id":"3NlLKqo7sFNH3f76LLlyHl","parentId":"","type":"header3","props":{"title":[["B. Two-stage hit resolution",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"6b0P7hqfjqHQ1zTK1M3K6w","blockSubtree":{"block":{"6b0P7hqfjqHQ1zTK1M3K6w":{"role":"editor","value":{"id":"6b0P7hqfjqHQ1zTK1M3K6w","parentId":"","type":"code","props":{"title":[["max_hit <= L - (window + max_rollback) # leave room to replay\nL1 = block_chain_hit_length(...) # multiple of block_size\nif state_present_in([L1 - window, L1]):\n hit = L1\nelse:\n hit = L1 - block_size # step back one block"]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"c93YxDryI0uihKtwVfx23t","blockSubtree":{"block":{"c93YxDryI0uihKtwVfx23t":{"role":"editor","value":{"id":"c93YxDryI0uihKtwVfx23t","parentId":"","type":"bulleted_list","props":{"title":[["The block chain still decides the hit length.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"B9txIFgruxr2W0a90HbkVi","blockSubtree":{"block":{"B9txIFgruxr2W0a90HbkVi":{"role":"editor","value":{"id":"B9txIFgruxr2W0a90HbkVi","parentId":"","type":"bulleted_list","props":{"title":[["State presence decides only whether the last block is usable.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"gxz8PMLmkORcNRqO81C7pt","blockSubtree":{"block":{"gxz8PMLmkORcNRqO81C7pt":{"role":"editor","value":{"id":"gxz8PMLmkORcNRqO81C7pt","parentId":"","type":"bulleted_list","props":{"title":[["When state is missing, stepping back one block lets the engine recompute a bounded amount instead of the whole prefix.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"9jxA3abIA9BgSuNKzikCtI","blockSubtree":{"block":{"9jxA3abIA9BgSuNKzikCtI":{"role":"editor","value":{"id":"9jxA3abIA9BgSuNKzikCtI","parentId":"","type":"text","props":{"title":[["The engine-side counterpart is \"replay only the last ",null],["window",[["c"]]],[" tokens, with the attention window truncated inside the replay segment\", which makes the recompute cost independent of context length. That stays engine work; this RFC is about the storage shape and the reported hit length.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"TO4qXBJvDf9ALnNgjaGDnN","blockSubtree":{"block":{"TO4qXBJvDf9ALnNgjaGDnN":{"role":"editor","value":{"id":"TO4qXBJvDf9ALnNgjaGDnN","parentId":"","type":"header3","props":{"title":[["C. Lifetime and placement",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"TN5tbmUR6jy5jSuxO8kSAE","blockSubtree":{"block":{"TN5tbmUR6jy5jSuxO8kSAE":{"role":"editor","value":{"id":"TN5tbmUR6jy5jSuxO8kSAE","parentId":"","type":"bulleted_list","props":{"title":[["TTL may differ from block KV. The reference deployment uses 3 days for agent traffic and 7 days for chatbot traffic, while the bulk store runs a longer LRU-based retention; the two lifetimes are managed separately.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"nrQ2ahvc4KeweQ39A8F7No","blockSubtree":{"block":{"nrQ2ahvc4KeweQ39A8F7No":{"role":"editor","value":{"id":"nrQ2ahvc4KeweQ39A8F7No","parentId":"","type":"bulleted_list","props":{"title":[["Eviction coupling between a state object and its block prefix is an open question below, not an assumption here.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"r0Zk16jTPQN5N5Pr5wG0lK","blockSubtree":{"block":{"r0Zk16jTPQN5N5Pr5wG0lK":{"role":"editor","value":{"id":"r0Zk16jTPQN5N5Pr5wG0lK","parentId":"","type":"bulleted_list","props":{"title":[["The payload is small and node-local in the reference design, with a directory recording which machine holds which key. Whether that directory is the Master or a side structure is an open question.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"fIF77DL46WjL9bYUGQaoRu","blockSubtree":{"block":{"fIF77DL46WjL9bYUGQaoRu":{"role":"editor","value":{"id":"fIF77DL46WjL9bYUGQaoRu","parentId":"","type":"header2","props":{"title":[["What this unlocks",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"KHLC0TCavmljcr0yeibTWR","blockSubtree":{"block":{"KHLC0TCavmljcr0yeibTWR":{"role":"editor","value":{"id":"KHLC0TCavmljcr0yeibTWR","parentId":"","type":"text","props":{"title":[["Without it, a hybrid/SWA model on Mooncake gets either (a) SWA bytes smuggled into a block object, which inflates payload and breaks block reuse, or (b) no prefix reuse at all for the layers that need it. With it, prefix reuse for these models costs ~1.4 MB per cached sequence and the router stops advertising matches the engine cannot use.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"QZDVkJWmDHkrgZnp0hadYs","blockSubtree":{"block":{"QZDVkJWmDHkrgZnp0hadYs":{"role":"editor","value":{"id":"QZDVkJWmDHkrgZnp0hadYs","parentId":"","type":"text","props":{"title":[["The asymmetry is the reason to bother: a false long hit costs a full prefill, a conservative short hit costs one block of recompute.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"Rm4rahbmN3yfo9a8A2vbK0","blockSubtree":{"block":{"Rm4rahbmN3yfo9a8A2vbK0":{"role":"editor","value":{"id":"Rm4rahbmN3yfo9a8A2vbK0","parentId":"","type":"header2","props":{"title":[["Relationship to existing work",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"rB389ZRE7yKLTcxLvSD8oe","blockSubtree":{"block":{"rB389ZRE7yKLTcxLvSD8oe":{"role":"editor","value":{"id":"rB389ZRE7yKLTcxLvSD8oe","parentId":"","type":"bulleted_list","props":{"title":[["#4083 / #4692: Engram tables are already a non-block object kind in Store, so the \"second object kind\" precedent exists.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"VfQNqtJuNtOU72xreytQwO","blockSubtree":{"block":{"VfQNqtJuNtOU72xreytQwO":{"role":"editor","value":{"id":"VfQNqtJuNtOU72xreytQwO","parentId":"","type":"bulleted_list","props":{"title":[["#4242: the state object is small and contiguous, so it needs no scatter path; the transport side has no new requirement.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"1b0q02lmeZHSFJcn4dBsLh","blockSubtree":{"block":{"1b0q02lmeZHSFJcn4dBsLh":{"role":"editor","value":{"id":"1b0q02lmeZHSFJcn4dBsLh","parentId":"","type":"bulleted_list","props":{"title":[["#131 discusses cache hit rate under cache-aware scheduling; that is a different cause (scheduling), but the router-side effect here is similar in spirit.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"3f7w9qvGxRW4yEQ5I28kor","blockSubtree":{"block":{"3f7w9qvGxRW4yEQ5I28kor":{"role":"editor","value":{"id":"3f7w9qvGxRW4yEQ5I28kor","parentId":"","type":"header2","props":{"title":[["Phasing",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"h8XAsW5lN6nSPesC6w6iP1","blockSubtree":{"block":{"h8XAsW5lN6nSPesC6w6iP1":{"role":"editor","value":{"id":"h8XAsW5lN6nSPesC6w6iP1","parentId":"","type":"numbered_list","props":{"title":[["Store",[["b"]]],[": state object kind, layout registry, aligned PUT/GET validation.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"gzXeLOmuuDeMVhKLUh9uHn","blockSubtree":{"block":{"gzXeLOmuuDeMVhKLUh9uHn":{"role":"editor","value":{"id":"gzXeLOmuuDeMVhKLUh9uHn","parentId":"","type":"numbered_list","props":{"title":[["Conductor",[["b"]]],[": the second predicate in the prefix index and the adjusted hit length returned to routers.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"m4ORC3P1172F6cJVx1HpL7","blockSubtree":{"block":{"m4ORC3P1172F6cJVx1HpL7":{"role":"editor","value":{"id":"m4ORC3P1172F6cJVx1HpL7","parentId":"","type":"numbered_list","props":{"title":[["Engine adapters",[["b"]]],[": write state on aligned boundaries, replay a bounded suffix on a short hit.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"peKghnOXjqLJVhUJFtc4Cp","blockSubtree":{"block":{"peKghnOXjqLJVhUJFtc4Cp":{"role":"editor","value":{"id":"peKghnOXjqLJVhUJFtc4Cp","parentId":"","type":"text","props":{"title":[["Steps 1 and 2 are independently reviewable; they are one RFC because step 2 is meaningless without step 1, and the step-back granularity in step 2 depends on the alignment rule chosen in step 1.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"zkjxfWiJtikzvaE54R13ZS","blockSubtree":{"block":{"zkjxfWiJtikzvaE54R13ZS":{"role":"editor","value":{"id":"zkjxfWiJtikzvaE54R13ZS","parentId":"","type":"header2","props":{"title":[["Non-goals",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"5nF7CKjhmJsLUC1iMuRtxa","blockSubtree":{"block":{"5nF7CKjhmJsLUC1iMuRtxa":{"role":"editor","value":{"id":"5nF7CKjhmJsLUC1iMuRtxa","parentId":"","type":"bulleted_list","props":{"title":[["Model-side logic (when to write, what to drop, how to recompute) stays in the engine.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"gRTzosHSGgS8jGGT80wfB8","blockSubtree":{"block":{"gRTzosHSGgS8jGGT80wfB8":{"role":"editor","value":{"id":"gRTzosHSGgS8jGGT80wfB8","parentId":"","type":"bulleted_list","props":{"title":[["No change to paged KV addressing or to the block hash recipe.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"yWU7FP4BxUtVYnRA4cT8NL","blockSubtree":{"block":{"yWU7FP4BxUtVYnRA4cT8NL":{"role":"editor","value":{"id":"yWU7FP4BxUtVYnRA4cT8NL","parentId":"","type":"bulleted_list","props":{"title":[["No decision here on where state replicas live; that is placement policy.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"HkeJGyKsmIcl2aylI1c9Kb","blockSubtree":{"block":{"HkeJGyKsmIcl2aylI1c9Kb":{"role":"editor","value":{"id":"HkeJGyKsmIcl2aylI1c9Kb","parentId":"","type":"bulleted_list","props":{"title":[["Not proposing a new transport; the existing path already covers a small contiguous object.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"LnSi3Ut6tgdTJU5r1Aewdx","blockSubtree":{"block":{"LnSi3Ut6tgdTJU5r1Aewdx":{"role":"editor","value":{"id":"LnSi3Ut6tgdTJU5r1Aewdx","parentId":"","type":"header2","props":{"title":[["Open questions",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"zPLDC4M4TtZCfEngcu7CN4","blockSubtree":{"block":{"zPLDC4M4TtZCfEngcu7CN4":{"role":"editor","value":{"id":"zPLDC4M4TtZCfEngcu7CN4","parentId":"","type":"numbered_list","props":{"title":[["Should state live in Store's object namespace, or in a separate lightweight directory that only maps (prefix, aligned length) to an owner?",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"FNfUIhTg54t87aBS5JASFe","blockSubtree":{"block":{"FNfUIhTg54t87aBS5JASFe":{"role":"editor","value":{"id":"FNfUIhTg54t87aBS5JASFe","parentId":"","type":"numbered_list","props":{"title":[["Is a per-model ",null],["state_layout",[["c"]]],[" registry acceptable, or should the value stay fully opaque with size validation only?",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"fKvbAuJWXl3ZYsQFG7MFD1","blockSubtree":{"block":{"fKvbAuJWXl3ZYsQFG7MFD1":{"role":"editor","value":{"id":"fKvbAuJWXl3ZYsQFG7MFD1","parentId":"","type":"numbered_list","props":{"title":[["Eviction: should a state object be evicted together with its block prefix, or independently?",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"zGAhnAHOU5ELE9XHbxCDBm","blockSubtree":{"block":{"zGAhnAHOU5ELE9XHbxCDBm":{"role":"editor","value":{"id":"zGAhnAHOU5ELE9XHbxCDBm","parentId":"","type":"numbered_list","props":{"title":[["Should the step-back be one block or ",null],["ceil(window / block_size)",[["c"]]],[" blocks? In the reference implementation state is stored at 128-token boundaries with a 128-token window, so one block is enough there; the general rule needs a definition.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"KGguvG1oNqxB8AwVpbki2A","blockSubtree":{"block":{"KGguvG1oNqxB8AwVpbki2A":{"role":"editor","value":{"id":"KGguvG1oNqxB8AwVpbki2A","parentId":"","type":"numbered_list","props":{"title":[["Who owns the \"window + rollback\" constant — model config registered with the Store, or the engine passing it per request?",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"fSgLrmKZWbk1JiiRL4hW2o","blockSubtree":{"block":{"fSgLrmKZWbk1JiiRL4hW2o":{"role":"editor","value":{"id":"fSgLrmKZWbk1JiiRL4hW2o","parentId":"","type":"numbered_list","props":{"title":[["If state is absent but the engine could rebuild it cheaply from the block chain, should the Store report ",null],["L1",[["c"]]],[" and let the engine decide? (Pro: fewer conservative step-backs. Con: the router no longer knows the real cost.)",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"D3DvqlEdnEvfjW5mB2qApZ","blockSubtree":{"block":{"D3DvqlEdnEvfjW5mB2qApZ":{"role":"editor","value":{"id":"D3DvqlEdnEvfjW5mB2qApZ","parentId":"","type":"header2","props":{"title":[["References",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"SGbf3wJavvdH7Jjon5b94g","blockSubtree":{"block":{"SGbf3wJavvdH7Jjon5b94g":{"role":"editor","value":{"id":"SGbf3wJavvdH7Jjon5b94g","parentId":"","type":"bulleted_list","props":{"title":[["deepseek-ai/DeepInfer (Apache-2.0): ",null],["deepinfer/worker/cache_engine.py",[["c"]]],[" (",null],["StateBlock",[["c"]]],[", ",null],["select_state_slice_to_store",[["c"]]],[", ",null],["load_state_from_store_to_prefill_cache",[["c"]]],[") and ",null],["doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf",[["c"]]],[".",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"YR8fOa0920PXxMf4NSCu9Q","blockSubtree":{"block":{"YR8fOa0920PXxMf4NSCu9Q":{"role":"editor","value":{"id":"YR8fOa0920PXxMf4NSCu9Q","parentId":"","type":"bulleted_list","props":{"title":[["Searched existing issues: no current issue covers non-paged per-sequence state, and none covers hit-length resolution for models that need it.",null]]},"children":[],"parentTable":"block","enabled":true}}}}}]"></sc-clipboard> ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:24:04 +00:00

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 20, 2026):

Thanks for opening this issue, @sususama!

Field	Value
Issue	#4803
GitHub user ID	`52159774`
Reporter	@sususama

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 20, 2026): Thanks for opening this issue, @sususama! | Field | Value | |-------|-------| | **Issue** | #4803 | | **GitHub user ID** | `52159774` | | **Reporter** | @sususama | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 20, 2026):

The state-validity and routing contract is worth separating from a new Store object kind. Store already accepts arbitrary byte slices under a string key (ObjectKey, Put/Get), so a fixed-size state payload does not by itself require changing that layer. A first adapter could use a separate key namespace containing the checkpoint/layout fingerprint, prefix hash and exact aligned boundary, validate the payload size, and publish availability only after the write completes.

For the lookup result, I would keep block_hit_length separate from state availability and the engine's fallback cost. An engine that supports bounded replay can recover differently from one that requires exact reconstruction; stepping back one block is not a safe general rule for every hybrid/SWA model. That capability and the replay boundary need to come from the engine rather than be inferred from an object's presence.

One source correction for section C: the reported 3/7-day retention is for Block KV. The SWA state in the reference deployment stays in host memory for about 180 seconds. Independent lifetimes are useful here, but those numbers should not become a state-object TTL default. I would pin the adapter's state layout and add hit/missing/expired/mismatched-layout contract tests before deciding whether the Master needs a model-specific layout registry.

@he-yufeng commented on GitHub (Oct 20, 2026): The state-validity and routing contract is worth separating from a new Store object kind. Store already accepts arbitrary byte slices under a string key ([ObjectKey](https://github.com/kvcache-ai/Mooncake/blob/5ea6cf0d95d3a69b4ba196d28bf1f96e28ec5e10/mooncake-store/include/types.h#L200), [Put/Get](https://github.com/kvcache-ai/Mooncake/blob/5ea6cf0d95d3a69b4ba196d28bf1f96e28ec5e10/mooncake-store/include/client_service.h#L197-L246)), so a fixed-size state payload does not by itself require changing that layer. A first adapter could use a separate key namespace containing the checkpoint/layout fingerprint, prefix hash and exact aligned boundary, validate the payload size, and publish availability only after the write completes. For the lookup result, I would keep `block_hit_length` separate from state availability and the engine's fallback cost. An engine that supports bounded replay can recover differently from one that requires exact reconstruction; stepping back one block is not a safe general rule for every hybrid/SWA model. That capability and the replay boundary need to come from the engine rather than be inferred from an object's presence. One source correction for section C: the reported 3/7-day retention is for Block KV. The SWA state in the reference deployment stays in host memory for about 180 seconds. Independent lifetimes are useful here, but those numbers should not become a state-object TTL default. I would pin the adapter's state layout and add hit/missing/expired/mismatched-layout contract tests before deciding whether the Master needs a model-specific layout registry.

bsakaguchi commented

2026-10-29 21:24:05 +00:00

Author

Owner

@sususama commented on GitHub (Oct 21, 2026):

Thanks, this is a useful correction and I'll fold both points in — and I'll write the adapter.

Section A — agreed, no new object kind. ObjectKey is a std::string and Put/Get already take arbitrary byte payloads, so I'll rewrite this as an adapter contract rather than a Store change: a separate key namespace carrying {checkpoint/layout fingerprint, prefix hash at the aligned boundary}, payload-size validation on both sides, and availability published only after the write completes. The state_layout registry becomes an adapter-side pinned constant in v1.

Section B — agreed, and this is the better shape. Store should report facts, not infer policy. Concretely: return block_hit_length and state_available(aligned_length) as separate fields and let the engine combine them with its own replay capability. I'll drop "step back one block" as a Store rule. One note on why both halves are needed: an engine that declares bounded-replay support still cannot act on it if the lookup never reports the state as missing.

Section C — you're right, I had this wrong. The 3/7-day figures are Block KV retention and should not appear as a state TTL default; I'll remove them. For the record, the open release does not pin a constant here: state capacity is a slot pool sized by DEEPINFER_NUM_STATE_SLOTS_RATIO, so residency is driven by pool pressure and request lifetime rather than by time. If the ~180 s figure comes from something citable I'd rather reference it than paraphrase it.

Contract tests — agreed, and I'd start there rather than argue about the registry: hit / missing / expired / layout-mismatch / boundary-misaligned against a pinned layout.

One question before I open anything: should the adapter live on the EngramStore-style path, or as a new lightweight connector under mooncake-integration? Default if you would rather not spend time on this: a new lightweight Python module on the store-facing side (python/mooncake/store/), following structured_object_store.py, because the adapter needs no GPU row gather and put / get / is_exist / get_size are already exposed. I'll proceed that way if I don't hear otherwise; it is easy to move before anything depends on it.

@sususama commented on GitHub (Oct 21, 2026): Thanks, this is a useful correction and I'll fold both points in — and I'll write the adapter. **Section A — agreed, no new object kind.** `ObjectKey` is a `std::string` and Put/Get already take arbitrary byte payloads, so I'll rewrite this as an adapter contract rather than a Store change: a separate key namespace carrying `{checkpoint/layout fingerprint, prefix hash at the aligned boundary}`, payload-size validation on both sides, and availability published only after the write completes. The `state_layout` registry becomes an adapter-side pinned constant in v1. **Section B — agreed, and this is the better shape.** Store should report facts, not infer policy. Concretely: return `block_hit_length` and `state_available(aligned_length)` as separate fields and let the engine combine them with its own replay capability. I'll drop "step back one block" as a Store rule. One note on why both halves are needed: an engine that declares bounded-replay support still cannot act on it if the lookup never reports the state as missing. **Section C — you're right, I had this wrong.** The 3/7-day figures are Block KV retention and should not appear as a state TTL default; I'll remove them. For the record, the open release does not pin a constant here: state capacity is a slot pool sized by `DEEPINFER_NUM_STATE_SLOTS_RATIO`, so residency is driven by pool pressure and request lifetime rather than by time. If the ~180 s figure comes from something citable I'd rather reference it than paraphrase it. **Contract tests** — agreed, and I'd start there rather than argue about the registry: hit / missing / expired / layout-mismatch / boundary-misaligned against a pinned layout. One question before I open anything: should the adapter live on the `EngramStore`-style path, or as a new lightweight connector under `mooncake-integration`? **Default if you would rather not spend time on this:** a new lightweight Python module on the store-facing side (`python/mooncake/store/`), following `structured_object_store.py`, because the adapter needs no GPU row gather and `put` / `get` / `is_exist` / `get_size` are already exposed. I'll proceed that way if I don't hear otherwise; it is easy to move before anything depends on it.

bsakaguchi commented

2026-10-29 21:24:06 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 23, 2026):

Confirmed, go with the default: a lightweight module under python/mooncake/store/ following structured_object_store.py. mooncake-integration is the C++/allocator-level surface, and this adapter is exactly the kind of byte-payload contract that belongs on the store-facing Python side. Nothing about it needs the GPU row-gather path.

Two things worth pinning before the first PR lands, both cheap now and expensive to retrofit:

Version the layout fingerprint itself, not just the layout. state_layout as an adapter-side constant is right for v1, but put a scheme/version byte (or a short semver string) into the fingerprint so a v2 layout change reads as a clean miss instead of a collision.
Reserve the key namespace explicitly. A documented prefix (something like state/ plus the fingerprint) keeps adapter keys out of the way of engine-written Block KV keys, and the contract tests should include a namespace-collision case, not only hit/miss/expired/layout-mismatch/boundary-misaligned.

On the TTL point: agreed, leave time-based figures out of the state path entirely. Pool-pressure-driven residency via the slot ratio is the accurate description, and the adapter should treat expiry as an observed fact from is_exist/get_size, never as a duration it assumes.

Looking forward to the contract tests first, as you proposed.

@he-yufeng commented on GitHub (Oct 23, 2026): Confirmed, go with the default: a lightweight module under `python/mooncake/store/` following `structured_object_store.py`. `mooncake-integration` is the C++/allocator-level surface, and this adapter is exactly the kind of byte-payload contract that belongs on the store-facing Python side. Nothing about it needs the GPU row-gather path. Two things worth pinning before the first PR lands, both cheap now and expensive to retrofit: 1. Version the layout fingerprint itself, not just the layout. `state_layout` as an adapter-side constant is right for v1, but put a scheme/version byte (or a short semver string) into the fingerprint so a v2 layout change reads as a clean miss instead of a collision. 2. Reserve the key namespace explicitly. A documented prefix (something like `state/` plus the fingerprint) keeps adapter keys out of the way of engine-written Block KV keys, and the contract tests should include a namespace-collision case, not only hit/miss/expired/layout-mismatch/boundary-misaligned. On the TTL point: agreed, leave time-based figures out of the state path entirely. Pool-pressure-driven residency via the slot ratio is the accurate description, and the adapter should treat expiry as an observed fact from `is_exist`/`get_size`, never as a duration it assumes. Looking forward to the contract tests first, as you proposed.

bsakaguchi commented

2026-10-29 21:24:06 +00:00

Author

Owner

@whn09 commented on GitHub (Oct 23, 2026):

An engine-side consumer for this contract now exists: deepseek-ai/DeepInfer#216 (draft) adds Mooncake Store as an L3 tier under DeepInfer's host prefix cache (#144 there), for DeepSeek-V4.1-Flash. Notes from it that may help the adapter and the contract tests:

How it maps to Sections A/B today (plain MooncakeDistributedStore, until the adapter lands):

State objects: state/<fp>/<prefix chain hash at the 128-aligned boundary>. One object per boundary, holding exactly the persisted SWA region DeepInfer stores at that position (state_size_to_disk, taken with select_state_slice_to_store). Boundaries are the last few 128-aligned positions of each stored prompt.
Block KV: a separate namespace, dsv4kv/<fp>/g<G>/<chain hash of the last block>. Each object holds G consecutive 128-token blocks, all layers per block.
<fp>: v1-<sha256[:16]> over the checkpoint config.json plus page bytes, state bytes, block size and the env knobs that change the state ring layout. The version is in the fingerprint itself, as suggested, so a layout change is a clean miss.
Hit resolution follows Section B: batch_is_exist on block keys gives the block run; batch_is_exist on state keys at candidate boundaries gives state availability. The engine then picks the deepest boundary that it can also replay from (it must recompute ≥ 140 trailing tokens) and fetches with batch_get_into_multi_buffers. The Store never infers policy.
Expiry: an object that is gone between exists and get (we treat -704 / size mismatch as a miss) falls back to the host-only lookup. There is a unit test for exactly that case.

Observations from B200, TCP, one host, wheel 0.3.13.post1 that may matter for defaults:

Per-object overhead dominates for single 114 KB blocks. batch_put_from reaches 0.17 GB/s with one block per object, but 1.7 GB/s with 4 blocks per object (the loopback limit). Hence G=4 (512 tokens) for block objects. State objects are larger and single.
The Store rejects protocol=shm (unsupported_protocol). protocol=efa on the same host gave TRANSFER_FAIL (-800) for every put; I haven't dug into that yet.

Once python/mooncake/store/'s adapter lands, DeepInfer can switch its state path to it directly. Happy to run the contract tests against real V4.1 states, and to share layout numbers, if that is useful.

中文

这个约定现在已经有了引擎侧的使用方：deepseek-ai/DeepInfer#216（Draft）在 DeepInfer 的主机前缀缓存（那边的 #144）下面加了一层 Mooncake Store 作为 L3，面向 DeepSeek-V4.1-Flash。下面几点或许对 adapter 和契约测试有帮助。

目前和 Section A/B 的对应关系（adapter 落地之前，直接用 MooncakeDistributedStore）：

state 对象： state/<fp>/<128 对齐边界处的前缀链哈希>。每个边界一个对象，内容正好是 DeepInfer 在该位置保存的 SWA 区域（state_size_to_disk，用 select_state_slice_to_store 取出）。边界取每个已保存 prompt 的最后几个 128 对齐位置。
Block KV： 用单独的命名空间，dsv4kv/<fp>/g<G>/<最后一个 block 的链哈希>。每个对象包含 G 个连续的 128-token block，每个 block 含所有层。
<fp>： v1-<sha256 前 16 位>，由 checkpoint 的 config.json 加上页字节数、state 字节数、block 大小，以及会改变 state 环形布局的环境变量算出。按建议把版本号放进了指纹本身，布局一变就自然查不到。
命中判定遵循 Section B： 对 block key 做 batch_is_exist 得到连续 block 的长度，对候选边界的 state key 做 batch_is_exist 得到 state 是否可用；然后由引擎挑出最深的、自己也能重放的边界（末尾必须重算至少 140 个 token），再用 batch_get_into_multi_buffers 读取。Store 本身不做任何策略推断。
过期： exists 和 get 之间对象消失时（-704 或大小不符都按未命中处理），退回只用主机池的查找。这种情况有专门的单元测试。

在 B200 上的观察（TCP，单机，wheel 0.3.13.post1），可能和默认参数有关：

对单个 114 KB 的 block 来说，每个对象的固定开销占了大头。 每对象 1 个 block 时 batch_put_from 只有 0.17 GB/s，每对象 4 个 block 能到 1.7 GB/s（回环带宽上限），所以 block 对象取 G=4（512 token）。state 对象本身较大，单独存放。
Store 不接受 protocol=shm（unsupported_protocol）；同机用 protocol=efa 时每次 put 都返回 TRANSFER_FAIL (-800)，我还没来得及深入查。

等 python/mooncake/store/ 的 adapter 落地后，DeepInfer 可以直接把 state 路径切过去。如果有用，我很乐意用真实的 V4.1 state 跑契约测试，也可以提供布局相关的数据。

@whn09 commented on GitHub (Oct 23, 2026): An engine-side consumer for this contract now exists: **deepseek-ai/DeepInfer#216** (draft) adds Mooncake Store as an L3 tier under DeepInfer's host prefix cache (#144 there), for DeepSeek-V4.1-Flash. Notes from it that may help the adapter and the contract tests: **How it maps to Sections A/B today** (plain `MooncakeDistributedStore`, until the adapter lands): - **State objects:** `state/<fp>/<prefix chain hash at the 128-aligned boundary>`. One object per boundary, holding exactly the persisted SWA region DeepInfer stores at that position (`state_size_to_disk`, taken with `select_state_slice_to_store`). Boundaries are the last few 128-aligned positions of each stored prompt. - **Block KV:** a separate namespace, `dsv4kv/<fp>/g<G>/<chain hash of the last block>`. Each object holds G consecutive 128-token blocks, all layers per block. - **`<fp>`:** `v1-<sha256[:16]>` over the checkpoint `config.json` plus page bytes, state bytes, block size and the env knobs that change the state ring layout. The version is in the fingerprint itself, as suggested, so a layout change is a clean miss. - **Hit resolution follows Section B:** `batch_is_exist` on block keys gives the block run; `batch_is_exist` on state keys at candidate boundaries gives state availability. The engine then picks the deepest boundary that it can also replay from (it must recompute ≥ 140 trailing tokens) and fetches with `batch_get_into_multi_buffers`. The Store never infers policy. - **Expiry:** an object that is gone between `exists` and `get` (we treat `-704` / size mismatch as a miss) falls back to the host-only lookup. There is a unit test for exactly that case. **Observations from B200, TCP, one host, wheel 0.3.13.post1** that may matter for defaults: - **Per-object overhead dominates for single 114 KB blocks.** `batch_put_from` reaches 0.17 GB/s with one block per object, but 1.7 GB/s with 4 blocks per object (the loopback limit). Hence G=4 (512 tokens) for block objects. State objects are larger and single. - The Store rejects `protocol=shm` (`unsupported_protocol`). `protocol=efa` on the same host gave `TRANSFER_FAIL (-800)` for every put; I haven't dug into that yet. Once `python/mooncake/store/`'s adapter lands, DeepInfer can switch its state path to it directly. Happy to run the contract tests against real V4.1 states, and to share layout numbers, if that is useful. <details><summary>中文</summary> 这个约定现在已经有了引擎侧的使用方：**deepseek-ai/DeepInfer#216**（Draft）在 DeepInfer 的主机前缀缓存（那边的 #144）下面加了一层 Mooncake Store 作为 L3，面向 DeepSeek-V4.1-Flash。下面几点或许对 adapter 和契约测试有帮助。 **目前和 Section A/B 的对应关系**（adapter 落地之前，直接用 `MooncakeDistributedStore`）： - **state 对象：** `state/<fp>/<128 对齐边界处的前缀链哈希>`。每个边界一个对象，内容正好是 DeepInfer 在该位置保存的 SWA 区域（`state_size_to_disk`，用 `select_state_slice_to_store` 取出）。边界取每个已保存 prompt 的最后几个 128 对齐位置。 - **Block KV：** 用单独的命名空间，`dsv4kv/<fp>/g<G>/<最后一个 block 的链哈希>`。每个对象包含 G 个连续的 128-token block，每个 block 含所有层。 - **`<fp>`：** `v1-<sha256 前 16 位>`，由 checkpoint 的 `config.json` 加上页字节数、state 字节数、block 大小，以及会改变 state 环形布局的环境变量算出。按建议把版本号放进了指纹本身，布局一变就自然查不到。 - **命中判定遵循 Section B：** 对 block key 做 `batch_is_exist` 得到连续 block 的长度，对候选边界的 state key 做 `batch_is_exist` 得到 state 是否可用；然后由引擎挑出最深的、自己也能重放的边界（末尾必须重算至少 140 个 token），再用 `batch_get_into_multi_buffers` 读取。Store 本身不做任何策略推断。 - **过期：** `exists` 和 `get` 之间对象消失时（`-704` 或大小不符都按未命中处理），退回只用主机池的查找。这种情况有专门的单元测试。 **在 B200 上的观察**（TCP，单机，wheel 0.3.13.post1），可能和默认参数有关： - **对单个 114 KB 的 block 来说，每个对象的固定开销占了大头。** 每对象 1 个 block 时 `batch_put_from` 只有 0.17 GB/s，每对象 4 个 block 能到 1.7 GB/s（回环带宽上限），所以 block 对象取 G=4（512 token）。state 对象本身较大，单独存放。 - Store 不接受 `protocol=shm`（`unsupported_protocol`）；同机用 `protocol=efa` 时每次 put 都返回 `TRANSFER_FAIL (-800)`，我还没来得及深入查。等 `python/mooncake/store/` 的 adapter 落地后，DeepInfer 可以直接把 state 路径切过去。如果有用，我很乐意用真实的 V4.1 state 跑契约测试，也可以提供布局相关的数据。 </details>

bsakaguchi referenced this issue

2026-10-29 21:32:38 +00:00

[PR #1216] [MERGED] [bugfix] fix bfloat16 for get_tensor #2080