[GH-ISSUE #2799] [Bug]: [SSD] DSV4 enable SSD offload, Batch get key failed, with the error reported as INVALID_KEY. #727

New issue

Closed

opened 2026-10-29 21:18:46 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:18:46 +00:00

Owner

Originally created by @huangdong2022 on GitHub (Jul 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2799

Bug Report

Environment

mooncake version（main） : 8b884bcd13

The DSV4 model, with SSD enabled, features 4 machines with PD separation, 1P1D, where both P and D are dual-machine setups.

vllm script (P shell)

unset ftp_proxy
unset https_proxy
unset http_proxy
rm -rf ~/ascend/log
rm -rf /opt/XXX/Test/vLLM/mooncake-ssd/logs/*
#!/bin/bash
TIMESTAMP=$(date +"%Y%m%d%H%M")

#export HCCL_INTRA_PCIE_ENABLE=1
#export HCCL_INTRA_ROCE_ENABLE=0

MASTER_IP_ADDRESS="x.xxx.xx.xx"
IP_ADDRESS="x.xxx.xx.xx"
SERVICE_PORT=10000
LOG_FILE="/opt/XXX/Test/vLLM/mooncake-ssd/logs/deepseekv4_start_prefill_${TIMESTAMP}.log"

NETWORK_INTERFACE="eth0"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
PREFILL_DATA_PARALLEL_RANK=0

PREFILL_DATA_PARALLEL_SIZE=2
PREFILL_TENSOR_PARALLEL_SIZE=8

MODEL_PATH="/opt/Dsv4-flash-bf16"
MODEL_NAME="DeepSeek-V4-Flash"

# # mooncake
export MOONCAKE_CONFIG_PATH="/opt/XXX/Test/vLLM/mooncake-ssd/configs/mooncake.json"
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH}
export PYTHONHASHSEED=0
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000

# # jemalloc
export LD_PRELOAD=/usr/lib64/libjemalloc.so.2:$LD_PRELOAD
# # AIV
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1200
export HCCL_CONNECT_TIMEOUT=1200

export HCCL_IF_IP=$IP_ADDRESS
export GLOO_SOCKET_IFNAME=$NETWORK_INTERFACE
export TP_SOCKET_IFNAME=$NETWORK_INTERFACE
export HCCL_SOCKET_IFNAME=$NETWORK_INTERFACE
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024

#export VLLM_VERSION=0.20.2
export VLLM_ASCEND_APPLY_DSV4_PATCH=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export USE_MULTI_GROUPS_KV_CACHE=1
#export USE_MULTI_BLOCK_POOL=1

vllm serve "$MODEL_PATH" \
--host "$IP_ADDRESS" \
--port "$SERVICE_PORT" \
--data-parallel-size "$PREFILL_DATA_PARALLEL_SIZE" \
--data-parallel-rank "$PREFILL_DATA_PARALLEL_RANK" \
--data-parallel-address "$MASTER_IP_ADDRESS" \
--data-parallel-rpc-port 12321 \
--tensor-parallel-size "$PREFILL_TENSOR_PARALLEL_SIZE" \
--enable-expert-parallel \
--served-model-name "$MODEL_NAME" \
--seed 1024 \
--max-model-len 204800 \
--max-num-batched-tokens 4096 \
--max-num-seqs 16 \
--block-size 128 \
--enforce-eager \
--async-scheduling \
--no-disable-hybrid-kv-cache-manager \
--no-enable-prefix-caching \
--trust-remote-code \
--gpu_memory_utilization 0.95 \
--speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
--safetensors-load-strategy 'prefetch' \
--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true}' \
--kv-transfer-config \
    '{"kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
        "connectors": [{
	    "kv_connector": "MooncakeHybridConnector",
	    "kv_role": "kv_producer",
	    "kv_port": "30000",
	    "kv_connector_extra_config": {
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 8
                },
                "decode": {
                        "dp_size": 2,
                        "tp_size": 8
                }
	    }
        },
	{
	    "kv_connector": "AscendStoreConnector",
	    "kv_role": "kv_producer",
	    "kv_connector_extra_config": {
                "backend": "mooncake",
		"lookup_rpc_port": "0"
            }
	}]
    }
    }' > "$LOG_FILE" 2>&1 &

mooncake master

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH}

mooncake_master \
  --rpc_port=39486 \
  --eviction_high_watermark_ratio=0.7 \
  --eviction_ratio=0.2 \
  --default_kv_lease_ttl=11000 \
  --enable_offload=true \
  --offload_on_evict=true

Error log

E20260708 12:57:39.994860 11520 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.995028 11521 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.995077 11521 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995029 11520 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995108 11520 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995096 11521 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995134 11520 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:39.995139 11521 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:39.995927 11522 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.995949 11522 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995958 11522 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.995970 11522 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:39.997578 11523 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.997612 11523 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.997623 11523 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.997637 11523 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:39.999794 11524 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35
E20260708 12:57:39.999840 11524 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.999853 11524 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:39.999871 11524 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:40.002527 11525 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@e51be75a0bbf08480449ff7935fb607d4ce723933968881f455263e518e1fada
E20260708 12:57:40.002549 11525 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:40.002559 11525 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:40.002570 11525 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:40.005353 10732 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@7b6197044193fe1b8d283f97eb7af9266ed01a5c00be92bb0d1da7c62b015b37
E20260708 12:57:40.005435 10732 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:40.005450 10732 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:40.005486 10732 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY
E20260708 12:57:40.006532 11311 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@0c1d79561339734b8aec9688f742b63df90e1e595f6d5fe3772765ed02485d2b
E20260708 12:57:40.006592 11311 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY
E20260708 12:57:40.006604 11311 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY

Possible reasons

BatchOffload Notifies Master Before Committing Local Index → INVALID_KEY on Get

Symptoms

Under high concurrency between offload and get, many requests fail with Failed to get key (res: -400, i.e. INVALID_KEY). BatchLoad / SSD read failures appear alongside request-level load failures. Master-side SSD usage is well below capacity and EvictDiskReplica=0, so the issue is not caused by disk exhaustion.

Root Cause

In BucketStorageBackend::BatchOffload, after a successful disk write the code used to call complete_handler first (which triggers NotifyOffloadSuccess, registering a LOCAL_DISK replica on the Master and allowing MEMORY eviction), and only then commit the client-side object_bucket_map_.

During that window, concurrent get requests that take the SSD read path cannot find the storage key in the local index → Key not found → INVALID_KEY. This is a publish-before-commit consistency bug: Master visibility precedes the client local index. It is unrelated to SSD capacity.

Incorrect Order (Before Fix)

WriteBucket → NotifyOffloadSuccess → [race window] → commit object_bucket_map_

Fix (`storage_backend.cpp`)

WriteBucket → commit object_bucket_map_ → NotifyOffloadSuccess

If NotifyOffloadSuccess fails: RollbackCommittedBucket rolls back the local index and CleanupOrphanedBucket removes orphan on-disk files, avoiding either "Client can read but Master does not know" or the reverse ghost-replica case.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @huangdong2022 on GitHub (Jul 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2799 ### Bug Report # Environment mooncake version（main） : 8b884bcd131009f5b02a981abcb83cfdb5e21736 The DSV4 model, with SSD enabled, features 4 machines with PD separation, 1P1D, where both P and D are dual-machine setups. # vllm script (P shell) ```shell unset ftp_proxy unset https_proxy unset http_proxy rm -rf ~/ascend/log rm -rf /opt/XXX/Test/vLLM/mooncake-ssd/logs/* #!/bin/bash TIMESTAMP=$(date +"%Y%m%d%H%M") #export HCCL_INTRA_PCIE_ENABLE=1 #export HCCL_INTRA_ROCE_ENABLE=0 MASTER_IP_ADDRESS="x.xxx.xx.xx" IP_ADDRESS="x.xxx.xx.xx" SERVICE_PORT=10000 LOG_FILE="/opt/XXX/Test/vLLM/mooncake-ssd/logs/deepseekv4_start_prefill_${TIMESTAMP}.log" NETWORK_INTERFACE="eth0" export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 PREFILL_DATA_PARALLEL_RANK=0 PREFILL_DATA_PARALLEL_SIZE=2 PREFILL_TENSOR_PARALLEL_SIZE=8 MODEL_PATH="/opt/Dsv4-flash-bf16" MODEL_NAME="DeepSeek-V4-Flash" # # mooncake export MOONCAKE_CONFIG_PATH="/opt/XXX/Test/vLLM/mooncake-ssd/configs/mooncake.json" export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH} export PYTHONHASHSEED=0 export HCCL_INTRA_ROCE_ENABLE=1 export ASCEND_CONNECT_TIMEOUT=10000 export ASCEND_TRANSFER_TIMEOUT=10000 # # jemalloc export LD_PRELOAD=/usr/lib64/libjemalloc.so.2:$LD_PRELOAD # # AIV export HCCL_OP_EXPANSION_MODE="AIV" export TASK_QUEUE_ENABLE=1 export VLLM_RPC_TIMEOUT=3600000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000 export HCCL_EXEC_TIMEOUT=1200 export HCCL_CONNECT_TIMEOUT=1200 export HCCL_IF_IP=$IP_ADDRESS export GLOO_SOCKET_IFNAME=$NETWORK_INTERFACE export TP_SOCKET_IFNAME=$NETWORK_INTERFACE export HCCL_SOCKET_IFNAME=$NETWORK_INTERFACE export OMP_PROC_BIND=false export OMP_NUM_THREADS=10 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=1024 #export VLLM_VERSION=0.20.2 export VLLM_ASCEND_APPLY_DSV4_PATCH=1 export VLLM_ASCEND_ENABLE_FLASHCOMM1=1 #export USE_MULTI_GROUPS_KV_CACHE=1 #export USE_MULTI_BLOCK_POOL=1 vllm serve "$MODEL_PATH" \ --host "$IP_ADDRESS" \ --port "$SERVICE_PORT" \ --data-parallel-size "$PREFILL_DATA_PARALLEL_SIZE" \ --data-parallel-rank "$PREFILL_DATA_PARALLEL_RANK" \ --data-parallel-address "$MASTER_IP_ADDRESS" \ --data-parallel-rpc-port 12321 \ --tensor-parallel-size "$PREFILL_TENSOR_PARALLEL_SIZE" \ --enable-expert-parallel \ --served-model-name "$MODEL_NAME" \ --seed 1024 \ --max-model-len 204800 \ --max-num-batched-tokens 4096 \ --max-num-seqs 16 \ --block-size 128 \ --enforce-eager \ --async-scheduling \ --no-disable-hybrid-kv-cache-manager \ --no-enable-prefix-caching \ --trust-remote-code \ --gpu_memory_utilization 0.95 \ --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \ --safetensors-load-strategy 'prefetch' \ --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \ --tokenizer-mode deepseek_v4 \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_v4 \ --additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true}' \ --kv-transfer-config \ '{"kv_connector": "MultiConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "connectors": [{ "kv_connector": "MooncakeHybridConnector", "kv_role": "kv_producer", "kv_port": "30000", "kv_connector_extra_config": { "prefill": { "dp_size": 2, "tp_size": 8 }, "decode": { "dp_size": 2, "tp_size": 8 } } }, { "kv_connector": "AscendStoreConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "backend": "mooncake", "lookup_rpc_port": "0" } }] } }' > "$LOG_FILE" 2>&1 & ``` # mooncake master ```shell export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:${LD_LIBRARY_PATH} mooncake_master \ --rpc_port=39486 \ --eviction_high_watermark_ratio=0.7 \ --eviction_ratio=0.2 \ --default_kv_lease_ttl=11000 \ --enable_offload=true \ --offload_on_evict=true ``` # Error log ```text E20260708 12:57:39.994860 11520 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35 E20260708 12:57:39.995028 11521 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35 E20260708 12:57:39.995077 11521 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995029 11520 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995108 11520 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995096 11521 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995134 11520 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:39.995139 11521 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:39.995927 11522 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35 E20260708 12:57:39.995949 11522 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995958 11522 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.995970 11522 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:39.997578 11523 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35 E20260708 12:57:39.997612 11523 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.997623 11523 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.997637 11523 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:39.999794 11524 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@16e8aff8f59dd4f5396cff3500306159171b8eb1745e182f9f3f7ccd94827d35 E20260708 12:57:39.999840 11524 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.999853 11524 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:39.999871 11524 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:40.002527 11525 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@e51be75a0bbf08480449ff7935fb607d4ce723933968881f455263e518e1fada E20260708 12:57:40.002549 11525 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:40.002559 11525 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:40.002570 11525 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:40.005353 10732 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@7b6197044193fe1b8d283f97eb7af9266ed01a5c00be92bb0d1da7c62b015b37 E20260708 12:57:40.005435 10732 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:40.005450 10732 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:40.005486 10732 real_client.cpp:5434] Batch get offload object failed,err_code = INVALID_KEY E20260708 12:57:40.006532 11311 storage_backend.cpp:1424] Key not found: Dsv4-flash-bf16@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c4@0c1d79561339734b8aec9688f742b63df90e1e595f6d5fe3772765ed02485d2b E20260708 12:57:40.006592 11311 file_storage.cpp:690] Batch load object failed,err_code = INVALID_KEY E20260708 12:57:40.006604 11311 file_storage.cpp:311] Batch load object failed,err_code = INVALID_KEY ``` # Possible reasons `BatchOffload` Notifies Master Before Committing Local Index → `INVALID_KEY` on Get ## Symptoms Under high concurrency between offload and get, many requests fail with `Failed to get key` (`res: -400`, i.e. `INVALID_KEY`). `BatchLoad` / SSD read failures appear alongside request-level load failures. Master-side SSD usage is well below capacity and `EvictDiskReplica=0`, so the issue is not caused by disk exhaustion. ## Root Cause In `BucketStorageBackend::BatchOffload`, after a successful disk write the code used to call `complete_handler` first (which triggers `NotifyOffloadSuccess`, registering a `LOCAL_DISK` replica on the Master and allowing MEMORY eviction), and only then commit the client-side `object_bucket_map_`. During that window, concurrent get requests that take the SSD read path cannot find the storage key in the local index → `Key not found` → `INVALID_KEY`. This is a **publish-before-commit** consistency bug: Master visibility precedes the client local index. It is unrelated to SSD capacity. ## Incorrect Order (Before Fix) ``` WriteBucket → NotifyOffloadSuccess → [race window] → commit object_bucket_map_ ``` ## Fix (`storage_backend.cpp`) ``` WriteBucket → commit object_bucket_map_ → NotifyOffloadSuccess ``` If `NotifyOffloadSuccess` fails: `RollbackCommittedBucket` rolls back the local index and `CleanupOrphanedBucket` removes orphan on-disk files, avoiding either "Client can read but Master does not know" or the reverse ghost-replica case. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:46 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:47 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 8, 2026):

Thanks for opening this issue, @huangdong2022!

Field	Value
Issue	#2799
GitHub user ID	`161736910`
Reporter	@huangdong2022

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 8, 2026): Thanks for opening this issue, @huangdong2022! | Field | Value | |-------|-------| | **Issue** | #2799 | | **GitHub user ID** | `161736910` | | **Reporter** | @huangdong2022 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.