[GH-ISSUE #4825] [Bug]: Hard-coded 60s transfer timeout in mooncake-store causes -800 (TRANSFER_FAIL) and -707 (LEASE_EXPIRED) on large batch_get — please make it configurable #1230

New issue

Open

opened 2026-10-29 21:24:12 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:24:12 +00:00

Owner

Originally created by @LYZ2024 on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4825

Bug Report

Summary

TransferEngineOperationState::wait_for_completion() in
mooncake-store/src/transfer_task.cpp (line ~830) uses a compile-time
constant for the batch transfer timeout:

// 60 seconds
constexpr int64_t timeout_milliseconds = 60 * 1000;
When a batch_get transfers a large number of keys (KV-cache blocks in our
PD-disaggregated LLM serving setup), any batch that takes longer than 60s is
failed as a whole, and the failure cascades:

Keys still in flight when the deadline hits → ErrorCode::TRANSFER_FAIL (-800) (transfer_task.cpp:892: "Failed to complete transfers after 60000 milliseconds for batch N").
Keys whose transfer did finish → the post-transfer lease re-check in client_service.cpp (BatchGet final scan, ~line 1790) finds the read lease (granted as lease_ttl_ms from the master at query time) already expired → ErrorCode::LEASE_EXPIRED (-707) ("lease_expired_before_data_transfer_completed").
The caller (vLLM-ascend mooncake_backend.py) then reports Failed to get N keys out of M. error_codes=[-800, -707] and the whole batch of KV data must be recomputed — the pool hit is wasted entirely.
So the 60s constant doesn't just fail slow transfers; it also burns the
master-granted read lease window, making -707 a systematic companion of
-800 whenever batch transfer time approaches the constant.


Why this is hard to work around
It is constexpr: no environment variable, config file entry, or RPC option can change it at runtime.
MC_TRANSFER_TIMEOUT (documented in the transfer-engine docs, read in mooncake-integration/transfer_engine/transfer_engine_py.cpp) only affects the P2P transfer-engine path, not the store's wait_for_completion() — the built libmooncake_store.so doesn't even contain that string. Users who hit -800 on the store path have no supported knob at all.
Slow-but-healthy transfers are a normal condition on simulated / bandwidth-limited / shared networks; 60s is not generous for batches of tens of thousands of keys.
What we observed
vLLM PD-disaggregation with AscendStoreConnector (backend=mooncake), DeepSeek-V4-Flash, ~10k-token prompts → batches of ~3k-20k keys.
Every repeated request (pool hit) failed with error_codes=[-800, -707] after exactly 60000 ms, while the master still held all keys.
Binary-patching the two mov x1, #0xea60 instructions in wait_for_completion (the timeout comparison and the error-log value) from 60000 to a larger value eliminated all -800/-707 errors; transfers that previously timed out now complete in 70-100s.
Request
Please expose this timeout as a runtime-configurable parameter, e.g.:

an environment variable consistent with the existing convention (MC_STORE_TRANSFER_TIMEOUT_MS or similar), or
a field in the store client / master config, or
at minimum, derive it from the lease TTL so the transfer window can never be shorter than the lease it must fit inside.
Something like:


static const int64_t timeout_milliseconds = [] {
    if (const char* v = std::getenv("MC_STORE_TRANSFER_TIMEOUT_MS")) {
        return std::max<int64_t>(1000, atoll(v));
    }
    return int64_t(60) * 1000;
}();
Happy to open a PR if the maintainers agree on the knob name/placement.

Environment
mooncake-store (git tree matching deployed libmooncake_store.so, transfer_task.cpp:830/:892, client_service.cpp lease scan)
Consumer: vllm-ascend MultiConnector (MooncakeConnector P2P + AscendStoreConnector with mooncake backend), aarch64
Batch sizes: 1.7k-20k keys per batch_get

### Before submitting...

- [ ] Ensure you searched for relevant issues and read the [documentation]

Originally created by @LYZ2024 on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4825 ### Bug Report ## Summary `TransferEngineOperationState::wait_for_completion()` in `mooncake-store/src/transfer_task.cpp` (line ~830) uses a **compile-time constant** for the batch transfer timeout: ```cpp // 60 seconds constexpr int64_t timeout_milliseconds = 60 * 1000; When a batch_get transfers a large number of keys (KV-cache blocks in our PD-disaggregated LLM serving setup), any batch that takes longer than 60s is failed as a whole, and the failure cascades: Keys still in flight when the deadline hits → ErrorCode::TRANSFER_FAIL (-800) (transfer_task.cpp:892: "Failed to complete transfers after 60000 milliseconds for batch N"). Keys whose transfer did finish → the post-transfer lease re-check in client_service.cpp (BatchGet final scan, ~line 1790) finds the read lease (granted as lease_ttl_ms from the master at query time) already expired → ErrorCode::LEASE_EXPIRED (-707) ("lease_expired_before_data_transfer_completed"). The caller (vLLM-ascend mooncake_backend.py) then reports Failed to get N keys out of M. error_codes=[-800, -707] and the whole batch of KV data must be recomputed — the pool hit is wasted entirely. So the 60s constant doesn't just fail slow transfers; it also burns the master-granted read lease window, making -707 a systematic companion of -800 whenever batch transfer time approaches the constant. Why this is hard to work around It is constexpr: no environment variable, config file entry, or RPC option can change it at runtime. MC_TRANSFER_TIMEOUT (documented in the transfer-engine docs, read in mooncake-integration/transfer_engine/transfer_engine_py.cpp) only affects the P2P transfer-engine path, not the store's wait_for_completion() — the built libmooncake_store.so doesn't even contain that string. Users who hit -800 on the store path have no supported knob at all. Slow-but-healthy transfers are a normal condition on simulated / bandwidth-limited / shared networks; 60s is not generous for batches of tens of thousands of keys. What we observed vLLM PD-disaggregation with AscendStoreConnector (backend=mooncake), DeepSeek-V4-Flash, ~10k-token prompts → batches of ~3k-20k keys. Every repeated request (pool hit) failed with error_codes=[-800, -707] after exactly 60000 ms, while the master still held all keys. Binary-patching the two mov x1, #0xea60 instructions in wait_for_completion (the timeout comparison and the error-log value) from 60000 to a larger value eliminated all -800/-707 errors; transfers that previously timed out now complete in 70-100s. Request Please expose this timeout as a runtime-configurable parameter, e.g.: an environment variable consistent with the existing convention (MC_STORE_TRANSFER_TIMEOUT_MS or similar), or a field in the store client / master config, or at minimum, derive it from the lease TTL so the transfer window can never be shorter than the lease it must fit inside. Something like: static const int64_t timeout_milliseconds = [] { if (const char* v = std::getenv("MC_STORE_TRANSFER_TIMEOUT_MS")) { return std::max<int64_t>(1000, atoll(v)); } return int64_t(60) * 1000; }(); Happy to open a PR if the maintainers agree on the knob name/placement. Environment mooncake-store (git tree matching deployed libmooncake_store.so, transfer_task.cpp:830/:892, client_service.cpp lease scan) Consumer: vllm-ascend MultiConnector (MooncakeConnector P2P + AscendStoreConnector with mooncake backend), aarch64 Batch sizes: 1.7k-20k keys per batch_get ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:24:12 +00:00

bsakaguchi commented

2026-10-29 21:24:12 +00:00

Author

Owner

@LYZ2024 commented on GitHub (Oct 21, 2026):

(EngineCore pid=27737) ERROR 07-23 11:54:45 [mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:45 [kv_transfer.py:753] Failed to load blocks. failed_count=111, failed_blocks={0, 7526, 7527, 7528, 7529, 7530, 7531, 7532, 7533, 7534, 7535, 7536, 7537, 7538, 7539, 7540, 7541, 7542, 7543, 7544, 7545, 7546, 7547, 7548, 7549, 7550, 7551, 7552, 7553, 7554, 7555, 7556, 7557, 7558, 7559, 7560, 7561, 7562, 7563, 7564, 7565, 7566, 7567, 7568, 7569, 7570, 7571, 7572, 7573, 7574, 7575, 7576, 7577, 7578, 7579, 7580, 7581, 7582, 7583, 7584, 7585, 7586, 7587, 7588, 7589, 7590, 7591, 7592, 7593, 7594, 7595, 7596, 7597, 7598, 7599, 7600, 7601, 7602, 7603, 7604, 7605, 7606, 7607, 7608, 7609, 7610, 7611, 7612, 7613, 7614, 7615, 7616, 7617, 7618, 7619, 7620, 7621, 7622, 7623, 7624, 7625, 7626, 7627, 7628, 7629, 7630, 7631, 7632, 7633, 7634, 7635}. Check block availability and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] EngineCore encountered a fatal error.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] Traceback (most recent call last):
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1158, in run_engine_core
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core.run_busy_loop()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1199, in run_busy_loop
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._process_engine_step()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1238, in _process_engine_step
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] outputs, model_executed = self.step_fn()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 557, in step_with_batch_queue
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core_outputs = self.scheduler.update_from_output(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] failed_kv_load_req_ids = self._handle_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2299, in _handle_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._update_requests_with_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2208, in _update_requests_with_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] ValueError: too many values to unpack (expected 1)
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] outputs = await engine_core.get_output_async()
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 1030, in get_output_async
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] raise self._format_exception(outputs) from None
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

@LYZ2024 commented on GitHub (Oct 21, 2026): (EngineCore pid=27737) ERROR 07-23 11:54:45 [mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state. (EngineCore pid=27737) ERROR 07-23 11:54:45 [kv_transfer.py:753] Failed to load blocks. failed_count=111, failed_blocks={0, 7526, 7527, 7528, 7529, 7530, 7531, 7532, 7533, 7534, 7535, 7536, 7537, 7538, 7539, 7540, 7541, 7542, 7543, 7544, 7545, 7546, 7547, 7548, 7549, 7550, 7551, 7552, 7553, 7554, 7555, 7556, 7557, 7558, 7559, 7560, 7561, 7562, 7563, 7564, 7565, 7566, 7567, 7568, 7569, 7570, 7571, 7572, 7573, 7574, 7575, 7576, 7577, 7578, 7579, 7580, 7581, 7582, 7583, 7584, 7585, 7586, 7587, 7588, 7589, 7590, 7591, 7592, 7593, 7594, 7595, 7596, 7597, 7598, 7599, 7600, 7601, 7602, 7603, 7604, 7605, 7606, 7607, 7608, 7609, 7610, 7611, 7612, 7613, 7614, 7615, 7616, 7617, 7618, 7619, 7620, 7621, 7622, 7623, 7624, 7625, 7626, 7627, 7628, 7629, 7630, 7631, 7632, 7633, 7634, 7635}. Check block availability and memory state. (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] EngineCore encountered a fatal error. (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] Traceback (most recent call last): (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1158, in run_engine_core (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core.run_busy_loop() (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1199, in run_busy_loop (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._process_engine_step() (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1238, in _process_engine_step (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] outputs, model_executed = self.step_fn() (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 557, in step_with_batch_queue (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core_outputs = self.scheduler.update_from_output( (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] failed_kv_load_req_ids = self._handle_invalid_blocks( (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2299, in _handle_invalid_blocks (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._update_requests_with_invalid_blocks( (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2208, in _update_requests_with_invalid_blocks (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id) (EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] ValueError: too many values to unpack (expected 1) (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] AsyncLLM output_handler failed. (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] Traceback (most recent call last): (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] outputs = await engine_core.get_output_async() (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 1030, in get_output_async (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] raise self._format_exception(outputs) from None (APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

bsakaguchi commented

2026-10-29 21:24:13 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 21, 2026):

Thanks for opening this issue, @LYZ2024!

Field	Value
Issue	#4825
GitHub user ID	`180744123`
Reporter	@LYZ2024

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 21, 2026): Thanks for opening this issue, @LYZ2024! | Field | Value | |-------|-------| | **Issue** | #4825 | | **GitHub user ID** | `180744123` | | **Reporter** | @LYZ2024 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.