mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #3076] [mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state. #791
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#791
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @LYZ2024 on GitHub (Jul 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3076
Bug Report
(EngineCore pid=27737) ERROR 07-23 11:54:45 [mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:45 [kv_transfer.py:753] Failed to load blocks. failed_count=111, failed_blocks={0, 7526, 7527, 7528, 7529, 7530, 7531, 7532, 7533, 7534, 7535, 7536, 7537, 7538, 7539, 7540, 7541, 7542, 7543, 7544, 7545, 7546, 7547, 7548, 7549, 7550, 7551, 7552, 7553, 7554, 7555, 7556, 7557, 7558, 7559, 7560, 7561, 7562, 7563, 7564, 7565, 7566, 7567, 7568, 7569, 7570, 7571, 7572, 7573, 7574, 7575, 7576, 7577, 7578, 7579, 7580, 7581, 7582, 7583, 7584, 7585, 7586, 7587, 7588, 7589, 7590, 7591, 7592, 7593, 7594, 7595, 7596, 7597, 7598, 7599, 7600, 7601, 7602, 7603, 7604, 7605, 7606, 7607, 7608, 7609, 7610, 7611, 7612, 7613, 7614, 7615, 7616, 7617, 7618, 7619, 7620, 7621, 7622, 7623, 7624, 7625, 7626, 7627, 7628, 7629, 7630, 7631, 7632, 7633, 7634, 7635}. Check block availability and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] EngineCore encountered a fatal error.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] Traceback (most recent call last):
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1158, in run_engine_core
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core.run_busy_loop()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1199, in run_busy_loop
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._process_engine_step()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1238, in _process_engine_step
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] outputs, model_executed = self.step_fn()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 557, in step_with_batch_queue
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core_outputs = self.scheduler.update_from_output(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] failed_kv_load_req_ids = self._handle_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2299, in _handle_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._update_requests_with_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2208, in _update_requests_with_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] ValueError: too many values to unpack (expected 1)
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] outputs = await engine_core.get_output_async()
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 1030, in get_output_async
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] raise self._format_exception(outputs) from None
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
mooncake version:latest
vllm version: 0.22.1+empty
1.3 Kvcache pooling
P node
export MOONCAKE_CONFIG_PATH="/path/to/mooncake.json"
export PYTHONHASHSEED=0
vllm serve /path/to/DeepSeek-V4-Flash/ --max_model_len 65536 --no-enable-prefix-caching --block_size 2 --max-num-batched-tokens 4096 --served-model-name qwen --gpu-memory-utilization 0.95 --async-scheduling --max-num-seqs 64 --port 8555 -tp 1 --additional_config '{"enable_cpu_binding": "True"}' --load_format safetensors --compilation-config '{"cudagraph_capture_sizes":[8], "cudagraph_mode": "FULL_DECODE_ONLY"}' --kv-transfer-config '{"kv_connector": "MultiConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "connectors": [ { "kv_connector": "MooncakeConnector", "kv_role": "kv_producer", "kv_port": "30200", "kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector", "kv_connector_extra_config": { "prefill": {"dp_size": 1, "tp_size": 1}, "decode": {"dp_size": 1, "tp_size": 1} } }, { "kv_connector": "AscendStoreConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "lookup_rpc_port": "0", "backend": "mooncake" } } ] } }' --enforce-eager
D node
export MOONCAKE_CONFIG_PATH="/path/to/mooncake.json"
export PYTHONHASHSEED=0
vllm serve /path/to/DeepSeek-V4-Flash/ --max_model_len 65536 --no-enable-prefix-caching --block_size 2 --max-num-batched-tokens 4096 --served-model-name qwen --gpu-memory-utilization 0.95 --async-scheduling --max-num-seqs 64 --port 8555 -tp 1 --additional_config '{"enable_cpu_binding": "True"}' --load_format safetensors --compilation-config '{"cudagraph_capture_sizes":[8], "cudagraph_mode": "FULL_DECODE_ONLY"}' --enforce-eager --kv-transfer-config '{"kv_connector": "MultiConnector", "kv_role": "kv_consumer", "kv_connector_extra_config": { "connectors": [ { "kv_connector": "MooncakeConnector", "kv_role": "kv_consumer", "kv_port": "30300", "kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector", "kv_connector_extra_config": { "prefill": {"dp_size": 1, "tp_size": 1}, "decode": {"dp_size": 1, "tp_size": 1} } }, { "kv_connector": "AscendStoreConnector", "kv_role": "kv_consumer", "kv_connector_extra_config": { "lookup_rpc_port": "0", "backend": "mooncake" } } ] } }'
ascend_for_lingqu/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py --host xx.xx.xx.xx --port 1025 --prefiller-hosts xx.xx.xx.xx --prefiller-port 8555 --decoder-hosts xx.xx.xx.xx --decoder-ports 8555
mooncake.json:
{
"metadata_server": "P2PHANDSHAKE",
"protocol": "ascend",
"device_name": "",
"use_ascend_direct": true,
"alloc_in_same_node": true,
"master_server_address": "xx.xx.xx.xx:50088",
"global_segment_size": 30000000000,
"local_buffer_size": 21474836480,
"default_kv_lease_ttl": 300000,
"default_kv_soft_pin_ttl": 300000,
"transfer_timeout": 10
}
mooncake_master --port 50088 --default_kv_lease_ttl 300000 --default_kv_soft_pin_ttl 300000
curl -X POST http://xx.xx.xx.xx:1025/v1/completions -H "Content-Type: application/json" -d '{
"prompt": "Something longer than 9000 tokens",
"max_tokens": 4
}'
Before submitting...
@github-actions[bot] commented on GitHub (Jul 23, 2026):
Thanks for opening this issue, @LYZ2024!
180744123A maintainer will triage this when possible. To help us respond faster, please include:
Useful links: Documentation · Contributing guide
@he-yufeng commented on GitHub (Jul 23, 2026):
-707 is LEASE_EXPIRED (mooncake-store/include/types.h): the master still knew the keys, but their leases ran out before the data transfer completed. All 6722 keys failing with the same code looks less like eviction and more like the whole batch being fetched after its lease window.
Two things worth checking: what default_kv_lease_ttl is set to on the master (milliseconds), and how much wall time passes between the prefill side putting those blocks and the decode side fetching them. If decode can start later than the TTL, either raise the TTL or refresh the lease before the fetch.
Your Mooncake version/commit and the TTL you run with would help confirm.
@LYZ2024 commented on GitHub (Jul 24, 2026):
--default_kv_lease_ttl 300000 --default_kv_soft_pin_ttl 300000 already exist
@github-actions[bot] commented on GitHub (Oct 22, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Oct 29, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.