[GH-ISSUE #4757] [Bug]: [Store] BatchGet returns TRANSFER_FAIL at the 60 s wait while RDMA READs are still in flight; caller reuses the destination GPU buffers and gets corrupted KV #1212

New issue

Open

opened 2026-10-29 21:24:00 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 21:24:00 +00:00

Owner

Copy link

Originally created by @ThisFat on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4757

Bug Report

Environment

Mooncake	mooncake-transfer-engine 0.3.13.post1 (PyPI wheel). Log line transfer_task.cpp:891 matches tag v0.3.13.post1 (commit 719735896c86), i.e. the non-event-driven polling branch of wait_for_completion()
vLLM	0.30.0, built-in MooncakeStoreConnector; {"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lookup_async":true}}
Model / serving	GLM-5.3-Flash (hybrid; kv cache group sizes [640, 4, 640, 640, 640, 640]), TP=8, --max-model-len 262144, --enable-prefix-caching, --enable-chunked-prefill, --max-num-batched-tokens 8192, --prefix-match-unit 128
Nodes	3 nodes (A, B, C), each 8× NVIDIA H20-3e, 2× Xeon Platinum 8558P, 2 TiB RAM
OS / drivers	Ubuntu 22.04.5, kernel 5.15.0-187-generic, NVIDIA driver 610.43.02 with nvidia_peermem loaded, MLNX_OFED 24.10-5.1.6.1
Python stack	Python 3.12.13, torch 2.13.0+cu130
Network	RoCE v2, one ConnectX NIC per node for Mooncake (mlx5_0, 400 Gb/s NDR), netdev MTU 9000 / active MTU 4096, GID index 3
Mooncake topology	mooncake_master on A: --default_kv_lease_ttl=60s --eviction_high_watermark_ratio=0.85 --eviction_ratio=0.02. One mooncake_client Store per node: --protocol=rdma --device_names=mlx5_0 --global_segment_size=1099511627776 --local_buffer_size=8589934592 --metadata_server=P2PHANDSHAKE. vLLM on each node uses the store as a client. Pool ~84% used, eviction continuously active
Env vars	No MC_* set (defaults: MC_SLICE_TIMEOUT=-1, MC_RETRY_CNT=9)

Description

TransferEngineOperationState::wait_for_completion() gives up after a hard-coded 60 s, sets TRANSFER_FAIL and returns, without cancelling or draining the batch. Client::BatchGet then returns to the caller. vLLM's MooncakeStoreConnector treats the failure as "these blocks are invalid": the scheduler reschedules the request and the GPU KV blocks are freed and reallocated (to other requests, or to the same request for recompute). The RDMA READs of the timed-out batch that are still queued or in flight keep the old local destination addresses. When they complete later, they write stale KV into GPU blocks that now hold other data.

This is the read-side counterpart of #4491 (timed-out put overwrites the next object).

Expected: after BatchGet returns an error for a key, Mooncake no longer writes into that key's destination buffer, or the API tells the caller the buffer is not yet safe to reuse.

Actual: TRANSFER_FAIL is returned while transfers targeting the buffer may still complete. The batch's freeBatchID() fails silently, and nothing in the logs shows that transfers were left behind.

Observed behavior

1. After node C's Store joined the pool (2 → 3 TiB), all three vLLM processes started logging bursts of:

   E.... transfer_task.cpp:891] Failed to complete transfers after 60000 milliseconds for batch ...
   E.... client_service.cpp:...] Transfer failed for key: <key> with error: -800
   W.... client_service.cpp:...] lease_expired_before_data_transfer_completed key=<key>
   E.... real_client.cpp:...] BatchGet failed for key '<key>': LEASE_EXPIRED   (or TRANSFER_FAIL)
   WARNING [worker.py:1471] Failed to get N Mooncake keys from sub-batch (...)
   WARNING [scheduler.py:3240] Recovered from KV load failure: 1 request(s) rescheduled (N tokens affected).
   

   Over ~3.3 days: node A 8,825 timed-out batches, node B 5,849, node C 35,166. Nodes A and B had zero of these in the preceding 3 days with a 1- and then 2-Store pool (including ~18 h of A↔B cross-node reads). Load does not explain it: a day peaking at ~1.2 TB/h of load_get had none; a later day peaking at ~0.33 TB/h still had >1,000 timed-out batches per hour on node A.

2. No RDMA-level errors in the same logs: no Process failed for slice, no Cannot redispatch slice, no BatchID cannot be freed. LEASE_EXPIRED outnumbers TRANSFER_FAIL about 3:1, so many of these transfers did finish, just after 60 s.

3. Output corruption correlates with the timeouts. Over two days we matched every Recovered from KV load failure event (tokens > 0) to gateway requests on the same node (arrived before the event, first token after it, prompt ≥ affected tokens). An output counts as broken if it is garbled from the first token or runs away (hits max_tokens or ends in endless repetition):

   Group (prompt ≥ 40k tokens)	Requests	Broken outputs
   Rescheduled after a KV load failure	169	6 (3.55%), garbled from the first token
   Same node, prefill overlapping the 60 s before a timeout, not rescheduled	588	5 (0.85%), normal start, runaway later
   Same nodes, other times	46,365	79 (0.17%)
   Nodes without Mooncake	89,196	56 (0.06%)

   The second row was never touched by the scheduler's recovery path, yet breaks ~5× more often than baseline. That points to memory being overwritten, not to a scheduling bug. Example: one request (49,713-token prompt, clean conversation history) waited 81.7 s for its first token. About 60 s after it arrived, the log on that node shows Failed to get 65 Mooncake keys → rescheduled (49152 tokens affected), and the response was garbage from the first token.

4. Mitigation in progress: we stopped node C's Store (back to 2 Stores). The timeout and reschedule counters on A and B have not increased since.

Steps to reproduce

A. How it happened in production (reproduces for us, but not on demand)

1. Start mooncake_master on node A with the flags above, and one mooncake_client Store (1 TiB, RDMA) on each of A, B, C.
2. Start vLLM 0.30.0 on each node with the MooncakeStoreConnector config above (kv_load_failure_policy=recompute, lookup_async=true).
3. Serve multi-turn agent traffic with long prompts (40k–140k tokens) so most requests get external prefix hits read from remote Stores, with the pool near the eviction watermark.
4. Watch the vLLM log for Failed to complete transfers after 60000 milliseconds followed by Recovered from KV load failure. Compare the responses of the rescheduled requests, and of other requests on the same node in the same minute, against the same prompts on a node without Mooncake.

We do not yet know what makes reads stall for 60 s on our fabric (see question 4), so we cannot trigger the stall on demand.

B. Minimal reproduction of the corruption (derived from the code; we have not run it)

The corruption does not depend on why the transfer is slow, only on a READ completing after wait_for_completion() has returned. A deterministic test needs:

1. Two clients and a Store; register a destination buffer D on the reading client.
2. Put key K with payload pattern P1.
3. Make the READ for K take longer than 60 s. Any of these should work: delay completion in the RDMA/transport layer in a test build (similar to the reproduction being prepared for #4491); or block RoCE traffic between the two nodes for ~65 s (switch ACL or port shutdown) and then restore it, so the queued/redispatched slices complete late.
4. Call BatchGet / batch_get_into_multi_buffers for K into D. Expect TRANSFER_FAIL (-800) after 60 s.
5. Right after it returns, fill D with pattern P2 (this is what vLLM does when it reuses the block).
6. Restore the transfer path and wait a few seconds.
7. Check D. If any part of it reads P1 again, the late READ overwrote a buffer the caller had already reused.

Code path (tag v0.3.13.post1)

• mooncake-store/src/transfer_task.cpp, TransferEngineOperationState::wait_for_completion(): timeout_milliseconds = 60 * 1000. On timeout the polling branch calls set_result_internal(ErrorCode::TRANSFER_FAIL) and returns. The comment in check_task_status() says it waits for all tasks to reach a terminal state so the caller does not see completion while background transfers are still running; the timeout branch bypasses that.
• mooncake-store/include/transfer_task.h:226: ~TransferEngineOperationState() { engine_.freeBatchID(batch_id_); } ignores the return value.
• mooncake-transfer-engine/src/multi_transport.cpp:108, MultiTransport::freeBatchID(): returns BatchBusy for unfinished tasks without logging.
• mooncake-store/src/client_service.cpp, Client::BatchGet: waits on each future, then returns; keys that succeeded after the lease get LEASE_EXPIRED.
• RDMA: failed slices are redispatched (up to MC_RETRY_CNT, default 9) to the same dest_addr; MC_SLICE_TIMEOUT is disabled by default.

Related issues (searched before filing)

• #4491 : same mechanism, put direction. Confirmed by maintainers on main.
• RFC #3793 / draft PR #3792: describes this Get/BatchGet hazard; not merged.
• #3508 (merged to main 2026-09-28): defers reclaiming timed-out BatchDesc. It protects the engine's metadata, but as far as we can tell does not stop late completions from writing into the caller's buffers.
• #4580: makes the 60 s timeout configurable; does not address late completions.
• #4153 / #3929: garbled output on the same model, caused by duplicate keys in batch gets (fixed in main 2026-09-11, not in 0.3.13.post1). We think that is a separate mechanism, because our broken outputs cluster around the 60 s timeouts. It may explain part of the remaining gap outside timeout windows (0.17% vs 0.06%); we have not verified this.

Questions

1. Can you confirm the Get/BatchGet direction is affected the same way as #4491 on current main?
2. Until RFC #3793 lands, how should a caller know when destination buffers are safe to reuse after a timeout TRANSFER_FAIL? For example, a distinct error code for "timed out, transfers may still be in flight", or an option to block until the batch drains.
3. Will #3792 (or a follow-up) cover Client::BatchGet / batch_get_into_multi_buffers, not only the scatter API?
4. Why would reads stall for 60 s with no RDMA completion errors? NIC counters per 10 min on all three nodes: roce_adp_retrans ~850–1,400, implied_nak_seq_err ~6,500–8,300, roce_adp_retrans_to 1–2. rx_prio0_buf_discard is in the tens of millions since boot. The numbers are similar across nodes.

We can share more log excerpts and our analysis scripts.

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

Originally created by @ThisFat on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4757 ### Bug Report ## Environment | | | |---|---| | Mooncake | `mooncake-transfer-engine` 0.3.13.post1 (PyPI wheel). Log line `transfer_task.cpp:891` matches tag `v0.3.13.post1` (commit `719735896c86`), i.e. the non-event-driven polling branch of `wait_for_completion()` | | vLLM | 0.30.0, built-in `MooncakeStoreConnector`; `{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"lookup_async":true}}` | | Model / serving | GLM-5.3-Flash (hybrid; `kv cache group sizes [640, 4, 640, 640, 640, 640]`), TP=8, `--max-model-len 262144`, `--enable-prefix-caching`, `--enable-chunked-prefill`, `--max-num-batched-tokens 8192`, `--prefix-match-unit 128` | | Nodes | 3 nodes (A, B, C), each 8× NVIDIA H20-3e, 2× Xeon Platinum 8558P, 2 TiB RAM | | OS / drivers | Ubuntu 22.04.5, kernel 5.15.0-187-generic, NVIDIA driver 610.43.02 with `nvidia_peermem` loaded, MLNX_OFED 24.10-5.1.6.1 | | Python stack | Python 3.12.13, torch 2.13.0+cu130 | | Network | RoCE v2, one ConnectX NIC per node for Mooncake (`mlx5_0`, 400 Gb/s NDR), netdev MTU 9000 / active MTU 4096, GID index 3 | | Mooncake topology | `mooncake_master` on A: `--default_kv_lease_ttl=60s --eviction_high_watermark_ratio=0.85 --eviction_ratio=0.02`. One `mooncake_client` Store per node: `--protocol=rdma --device_names=mlx5_0 --global_segment_size=1099511627776 --local_buffer_size=8589934592 --metadata_server=P2PHANDSHAKE`. vLLM on each node uses the store as a client. Pool ~84% used, eviction continuously active | | Env vars | No `MC_*` set (defaults: `MC_SLICE_TIMEOUT=-1`, `MC_RETRY_CNT=9`) | ### Description `TransferEngineOperationState::wait_for_completion()` gives up after a hard-coded 60 s, sets `TRANSFER_FAIL` and returns, without cancelling or draining the batch. `Client::BatchGet` then returns to the caller. vLLM's `MooncakeStoreConnector` treats the failure as "these blocks are invalid": the scheduler reschedules the request and the GPU KV blocks are freed and reallocated (to other requests, or to the same request for recompute). The RDMA READs of the timed-out batch that are still queued or in flight keep the old local destination addresses. When they complete later, they write stale KV into GPU blocks that now hold other data. This is the read-side counterpart of #4491 (timed-out put overwrites the next object). **Expected:** after `BatchGet` returns an error for a key, Mooncake no longer writes into that key's destination buffer, or the API tells the caller the buffer is not yet safe to reuse. **Actual:** `TRANSFER_FAIL` is returned while transfers targeting the buffer may still complete. The batch's `freeBatchID()` fails silently, and nothing in the logs shows that transfers were left behind. ### Observed behavior 1. After node C's Store joined the pool (2 → 3 TiB), all three vLLM processes started logging bursts of: ``` E.... transfer_task.cpp:891] Failed to complete transfers after 60000 milliseconds for batch ... E.... client_service.cpp:...] Transfer failed for key: <key> with error: -800 W.... client_service.cpp:...] lease_expired_before_data_transfer_completed key=<key> E.... real_client.cpp:...] BatchGet failed for key '<key>': LEASE_EXPIRED (or TRANSFER_FAIL) WARNING [worker.py:1471] Failed to get N Mooncake keys from sub-batch (...) WARNING [scheduler.py:3240] Recovered from KV load failure: 1 request(s) rescheduled (N tokens affected). ``` Over ~3.3 days: node A 8,825 timed-out batches, node B 5,849, node C 35,166. Nodes A and B had **zero** of these in the preceding 3 days with a 1- and then 2-Store pool (including ~18 h of A↔B cross-node reads). Load does not explain it: a day peaking at ~1.2 TB/h of `load_get` had none; a later day peaking at ~0.33 TB/h still had >1,000 timed-out batches per hour on node A. 2. No RDMA-level errors in the same logs: no `Process failed for slice`, no `Cannot redispatch slice`, no `BatchID cannot be freed`. `LEASE_EXPIRED` outnumbers `TRANSFER_FAIL` about 3:1, so many of these transfers did finish, just after 60 s. 3. Output corruption correlates with the timeouts. Over two days we matched every `Recovered from KV load failure` event (tokens > 0) to gateway requests on the same node (arrived before the event, first token after it, prompt ≥ affected tokens). An output counts as broken if it is garbled from the first token or runs away (hits `max_tokens` or ends in endless repetition): | Group (prompt ≥ 40k tokens) | Requests | Broken outputs | |---|---|---| | Rescheduled after a KV load failure | 169 | 6 (3.55%), garbled **from the first token** | | Same node, prefill overlapping the 60 s before a timeout, not rescheduled | 588 | 5 (0.85%), normal start, runaway later | | Same nodes, other times | 46,365 | 79 (0.17%) | | Nodes without Mooncake | 89,196 | 56 (0.06%) | The second row was never touched by the scheduler's recovery path, yet breaks ~5× more often than baseline. That points to memory being overwritten, not to a scheduling bug. Example: one request (49,713-token prompt, clean conversation history) waited 81.7 s for its first token. About 60 s after it arrived, the log on that node shows `Failed to get 65 Mooncake keys` → `rescheduled (49152 tokens affected)`, and the response was garbage from the first token. 4. Mitigation in progress: we stopped node C's Store (back to 2 Stores). The timeout and reschedule counters on A and B have not increased since. ### Steps to reproduce **A. How it happened in production (reproduces for us, but not on demand)** 1. Start `mooncake_master` on node A with the flags above, and one `mooncake_client` Store (1 TiB, RDMA) on each of A, B, C. 2. Start vLLM 0.30.0 on each node with the `MooncakeStoreConnector` config above (`kv_load_failure_policy=recompute`, `lookup_async=true`). 3. Serve multi-turn agent traffic with long prompts (40k–140k tokens) so most requests get external prefix hits read from remote Stores, with the pool near the eviction watermark. 4. Watch the vLLM log for `Failed to complete transfers after 60000 milliseconds` followed by `Recovered from KV load failure`. Compare the responses of the rescheduled requests, and of other requests on the same node in the same minute, against the same prompts on a node without Mooncake. We do not yet know what makes reads stall for 60 s on our fabric (see question 4), so we cannot trigger the stall on demand. **B. Minimal reproduction of the corruption (derived from the code; we have not run it)** The corruption does not depend on why the transfer is slow, only on a READ completing after `wait_for_completion()` has returned. A deterministic test needs: 1. Two clients and a Store; register a destination buffer `D` on the reading client. 2. `Put` key `K` with payload pattern `P1`. 3. Make the READ for `K` take longer than 60 s. Any of these should work: delay completion in the RDMA/transport layer in a test build (similar to the reproduction being prepared for #4491); or block RoCE traffic between the two nodes for ~65 s (switch ACL or port shutdown) and then restore it, so the queued/redispatched slices complete late. 4. Call `BatchGet` / `batch_get_into_multi_buffers` for `K` into `D`. Expect `TRANSFER_FAIL` (-800) after 60 s. 5. Right after it returns, fill `D` with pattern `P2` (this is what vLLM does when it reuses the block). 6. Restore the transfer path and wait a few seconds. 7. Check `D`. If any part of it reads `P1` again, the late READ overwrote a buffer the caller had already reused. ### Code path (tag v0.3.13.post1) - `mooncake-store/src/transfer_task.cpp`, `TransferEngineOperationState::wait_for_completion()`: `timeout_milliseconds = 60 * 1000`. On timeout the polling branch calls `set_result_internal(ErrorCode::TRANSFER_FAIL)` and returns. The comment in `check_task_status()` says it waits for all tasks to reach a terminal state so the caller does not see completion while background transfers are still running; the timeout branch bypasses that. - `mooncake-store/include/transfer_task.h:226`: `~TransferEngineOperationState() { engine_.freeBatchID(batch_id_); }` ignores the return value. - `mooncake-transfer-engine/src/multi_transport.cpp:108`, `MultiTransport::freeBatchID()`: returns `BatchBusy` for unfinished tasks without logging. - `mooncake-store/src/client_service.cpp`, `Client::BatchGet`: waits on each future, then returns; keys that succeeded after the lease get `LEASE_EXPIRED`. - RDMA: failed slices are redispatched (up to `MC_RETRY_CNT`, default 9) to the same `dest_addr`; `MC_SLICE_TIMEOUT` is disabled by default. ### Related issues (searched before filing) - #4491 : same mechanism, put direction. Confirmed by maintainers on main. - RFC #3793 / draft PR #3792: describes this Get/BatchGet hazard; not merged. - #3508 (merged to main 2026-09-28): defers reclaiming timed-out `BatchDesc`. It protects the engine's metadata, but as far as we can tell does not stop late completions from writing into the caller's buffers. - #4580: makes the 60 s timeout configurable; does not address late completions. - #4153 / #3929: garbled output on the same model, caused by duplicate keys in batch gets (fixed in main 2026-09-11, not in 0.3.13.post1). We think that is a separate mechanism, because our broken outputs cluster around the 60 s timeouts. It may explain part of the remaining gap outside timeout windows (0.17% vs 0.06%); we have not verified this. ### Questions 1. Can you confirm the Get/BatchGet direction is affected the same way as #4491 on current main? 2. Until RFC #3793 lands, how should a caller know when destination buffers are safe to reuse after a timeout `TRANSFER_FAIL`? For example, a distinct error code for "timed out, transfers may still be in flight", or an option to block until the batch drains. 3. Will #3792 (or a follow-up) cover `Client::BatchGet` / `batch_get_into_multi_buffers`, not only the scatter API? 4. Why would reads stall for 60 s with no RDMA completion errors? NIC counters per 10 min on all three nodes: `roce_adp_retrans` ~850–1,400, `implied_nak_seq_err` ~6,500–8,300, `roce_adp_retrans_to` 1–2. `rx_prio0_buf_discard` is in the tens of millions since boot. The numbers are similar across nodes. We can share more log excerpts and our analysis scripts. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label 2026-10-29 21:24:00 +00:00

bsakaguchi commented 2026-10-29 21:24:01 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Thanks for opening this issue, @ThisFat!

Field	Value
Issue	#4757
GitHub user ID	134714568
Reporter	@ThisFat

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6258688128 --> @github-actions[bot] commented on GitHub (Oct 19, 2026): Thanks for opening this issue, @ThisFat! | Field | Value | |-------|-------| | **Issue** | #4757 | | **GitHub user ID** | `134714568` | | **Reporter** | @ThisFat | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:24:01 +00:00

Author

Owner

Copy link

@he-yufeng commented on GitHub (Oct 19, 2026):

Thanks for the unusually well-instrumented report — the matched cohort table (rescheduled 3.55% vs same-node baseline 0.17%, and the overlapping-prefill row at 0.85% that the scheduler never touched) is about as close to proof of stray writes as black-box evidence gets.

This is the read-side twin of #4491, which @apex-mochen is already working on for the engine half (dropping unposted slices of a timed-out batch so they can't land later). The read side needs the same discipline plus one extra step: when wait_for_completion() gives up at the hard-coded 60 s, the batch is abandoned without cancel/drain, freeBatchID() fails silently, and the caller gets no signal that the destination buffers are still being written. TRANSFER_FAIL is treated by callers (vLLM's connector here) as "buffers free to reuse", which is precisely the wrong contract while RDMA READs are still queued.

For the fix shape I'd expect the engine half of #4491 to cover most of this: on timeout, drain or cancel the batch's remaining slices and only return failure once no worker can still touch the destinations; until that lands, a stopgap worth considering is logging the leftover-slice count when freeBatchID() fails so the silent part at least becomes visible. #4580 (configurable wait timeout) reduces how often the path triggers but does not change the contract — the stray-write window exists at any timeout value.

Tagging this for the transfer-engine owners; it belongs to the same fix arc as #4491 and #3792.

<!-- gh-comment-id:6261600383 --> @he-yufeng commented on GitHub (Oct 19, 2026): Thanks for the unusually well-instrumented report — the matched cohort table (rescheduled 3.55% vs same-node baseline 0.17%, and the overlapping-prefill row at 0.85% that the scheduler never touched) is about as close to proof of stray writes as black-box evidence gets. This is the read-side twin of #4491, which @apex-mochen is already working on for the engine half (dropping unposted slices of a timed-out batch so they can't land later). The read side needs the same discipline plus one extra step: when `wait_for_completion()` gives up at the hard-coded 60 s, the batch is abandoned without cancel/drain, `freeBatchID()` fails silently, and the caller gets no signal that the destination buffers are still being written. `TRANSFER_FAIL` is treated by callers (vLLM's connector here) as "buffers free to reuse", which is precisely the wrong contract while RDMA READs are still queued. For the fix shape I'd expect the engine half of #4491 to cover most of this: on timeout, drain or cancel the batch's remaining slices and only return failure once no worker can still touch the destinations; until that lands, a stopgap worth considering is logging the leftover-slice count when `freeBatchID()` fails so the silent part at least becomes visible. #4580 (configurable wait timeout) reduces how often the path triggers but does not change the contract — the stray-write window exists at any timeout value. Tagging this for the transfer-engine owners; it belongs to the same fix arc as #4491 and #3792.

bsakaguchi referenced this issue 2026-10-29 21:32:36 +00:00

[PR #1212] [CLOSED] [Store] feat: introduce tiered backend #2075

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1212

No description provided.