kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1729] [Bug]: SSD Offload fails in Metadata Server mode (RDMA writes memory successfully but no data to SSD) #501

New issue

Closed

opened 2026-10-29 21:15:31 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:15:31 +00:00

Owner

Originally created by @Colors-111 on GitHub (Mar 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1729

Originally assigned to: @ykwd, @zhangzuo21 on GitHub.

Bug Report

When using Mooncake Store with SSD offload enabled in non-P2PHANDSHAKE mode (using Metadata Service), BatchQuerySegmentSlices fails with INVALID_KEY error because the transport_endpoint_ comparison logic doesn't handle multi-NIC scenarios correctly.

Tested qwen3-32b with two H20 instances Hicache enabled within a Docker container.

E0323 09:34:39.531296  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.534227  3922 file_storage.cpp:481] Key not found: bdd996f1823dd51621ef94f44416cef32cd909907772e311a686cc196189b3ab_3_v
E0323 09:34:39.534245  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.535080  3922 file_storage.cpp:481] Key not found: 496c6bc85b0a8c92559dabfc8ddc1674d460492d03d2a10da230c70d167d34f5_0_k
E0323 09:34:39.535097  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.535931  3922 file_storage.cpp:481] Key not found: 0e1e4ba2445bcdabae622eac574fa961f34b15759784875a329235d2d5e8e849_1_v
E0323 09:34:39.535948  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.536794  3922 file_storage.cpp:481] Key not found: a07ad9abf9c2a7c6d8032a2defbc18f450c094a51bd2c8deeae2c436aa0594de_3_v
E0323 09:34:39.536810  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.537624  3922 file_storage.cpp:481] Key not found: e3344c975b754a108c5ab69c9358e5ec87733d0f9e9a11efc700ad45ce924db6_0_v
E0323 09:34:39.537642  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.539505  3922 file_storage.cpp:481] Key not found: 990c28883ad808f1caed10251278e6d535262f9cdc8f8bc0e1759d2aee67b3bc_0_k
E0323 09:34:39.539521  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.542181  3922 file_storage.cpp:481] Key not found: 5ffdd98eca9dece994827d20ac2e1e9043ebeb47f873e0e9dc39dd23553de878_3_v
E0323 09:34:39.542197  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.545236  3922 file_storage.cpp:481] Key not found: 8f1e1de78ddebaec444b51013390661c90b0dd614b39f2788e8e8d628701036b_0_k
E0323 09:34:39.545253  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.547999  3922 file_storage.cpp:481] Key not found: beaeb76f98aa9f64346259fe19e32e60e1f719f117dcab2b373788d5b07608aa_0_v
E0323 09:34:39.548015  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.550714  3922 file_storage.cpp:481] Key not found: 6328fe9393612dd8cca9efa31e127ce6b9d3b37cda6902415366fe01ec9dd158_0_k
E0323 09:34:39.550730  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.551554  3922 file_storage.cpp:481] Key not found: b11c692360b133956251adf255327e23b5d319a2231214483720f0ca6917119a_2_k
E0323 09:34:39.551571  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY
E0323 09:34:39.552387  3922 file_storage.cpp:481] Key not found: 212180d83919ee636abf0851ab29e91a151a3debb21a113ae336aabef21d9a7c_2_v
E0323 09:34:39.552403  3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY

Environment:

Mooncake version: Latest (Release: v0.3.10)
Deployment:
1 Master + 1 RealClient on machine A
1 RealClient on machine B
Protocol: RDMA
Metadata mode: Non-P2PHANDSHAKE (using HTTP metadata server)

Steps to Reproduce

master:


mooncake_master \
--eviction_high_watermark_ratio=0.95 \
--rpc_address ${MASTER_IP} \
--rpc_port 50051 \
--rpc_thread_num=32 \
--enable_offload=true \
--enable_http_metadata_server=true \
--http_metadata_server_port=8080 \
--enable_metric_reporting=true \
--metrics_port 9003

client:


export MASTER_IP="MASTER_IP"
export NODE_IP="NODE_IP"

export MOONCAKE_LOCAL_HOSTNAME="${NODE_IP}"
export MOONCAKE_TE_META_DATA_SERVER="http://${MASTER_IP}:8080/metadata"
export MOONCAKE_MASTER="${MASTER_IP}:50051"
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="mlx5_0,mlx5_1"

export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/data0/mooncake_offload
export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
export MOONCAKE_BUCKET_MAX_TOTAL_SIZE=$((200 * 1024 * 1024 * 1024))  # 200 GB
export MOONCAKE_BUCKET_EVICTION_POLICY=lru
export MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS=2

mooncake_client \
--master_server_address="${MOONCAKE_MASTER}" \
--host="${NODE_IP}" \
--port=50052 \
--protocol="rdma" \
--device_names="mlx5_0,mlx5_1" \
--global_segment_size="100GB" \
--enable_offload="true" \
--metadata_server="${MOONCAKE_TE_META_DATA_SERVER}"

sglang:

--model /data0/models/Qwen3-32B-FP8 \
--port 30000 \
--tp 4 \
--enable-hierarchical-cache \
--schedule-policy lpm \
--hicache-storage-prefetch-policy timeout \
--hicache-storage-backend mooncake \
--hicache-size 100 \
--hicache-storage-backend-extra-config '{
  "master_server_address": "master_ip:50051",
  "local_hostname": "local_hostname",
  "metadata_server": "http://master_ip:8080/metadata",
  "global_segment_size": "0",
  "protocol": "rdma",
  "device_name": "mlx5_0,mlx5_1"
}'

then run benchmark:

    --backend sglang \
    --model /data0/models/Qwen3-235B-A22B-FP8 \
    --tokenizer /data0/models/Qwen3-235B-A22B-FP8 \
    --base-url http://localhost:30000 \
    --dataset-name random \
    --dataset-path /data0/models/ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 288 \
    --max-concurrency 20 \
    --warmup-requests 16 \
    --random-input-len 16384 \
    --random-output-len 1138 \
    --random-range-ratio 1.0

Expected Behavior

SSD offload should work correctly, with objects being written to the SSD storage path.

Actual Behavior

RealClient logs show:

E0323 10:45:04.495340 file_storage.cpp:496] Key not found: xxx
E0323 10:45:04.495361 file_storage.cpp:357] BatchQuerySlices failed with error: INVALID_KEY

No files are written to the SSD offload directory.

Root Cause Analysis

The Problem

my guess：
Metadata Server mode forces segment.te_endpoint = --host IP (business IP), which is inconsistent with the actual RDMA NIC IP. SSD Offload relies on te_endpoint to verify local Replica Key —— mismatch leads to "Key not found", thus Offload aborts. P2PHANDSHAKE mode automatically sets segment.te_endpoint = RDMA NIC actual IP, so the verification passes.

In file_storage.cpp: BatchQuerySegmentSlices compares addresses incorrectly:

if (memory_descriptor.buffer_descriptor.transport_endpoint_ ==
    client_->GetTransportEndpoint()) {

Why It Fails

Mode	transport_endpoint_	GetTransportEndpoint()	Result
P2P_HANDSHAKE	RDMA IP:PORT (e.g., IP1:Port1)	RDMA IP:PORT	Match
Non-P2P_HANDSHAKE	local_hostname_ (e.g., IP2:Port2)	RDMA IP:PORT	No Match

In non-P2PHANDSHAKE mode:

transport_endpoint_ = local_hostname_ = user-specified --host parameter
GetTransportEndpoint() = actual RDMA NIC IP:PORT
These are different IP addresses in multi-NIC environments, causing the comparison to fail.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Colors-111 on GitHub (Mar 23, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1729 Originally assigned to: @ykwd, @zhangzuo21 on GitHub. ### Bug Report When using Mooncake Store with SSD offload enabled in non-P2PHANDSHAKE mode (using Metadata Service), BatchQuerySegmentSlices fails with INVALID_KEY error because the transport_endpoint_ comparison logic doesn't handle multi-NIC scenarios correctly. Tested qwen3-32b with two H20 instances Hicache enabled within a Docker container. ```E0323 09:34:39.531279 3922 file_storage.cpp:481] Key not found: 2a14b0d2e63eaa94e95c7b5301911cf43e6b0b694b39ef6b04cc6fc4530624f7_1_k E0323 09:34:39.531296 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.534227 3922 file_storage.cpp:481] Key not found: bdd996f1823dd51621ef94f44416cef32cd909907772e311a686cc196189b3ab_3_v E0323 09:34:39.534245 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.535080 3922 file_storage.cpp:481] Key not found: 496c6bc85b0a8c92559dabfc8ddc1674d460492d03d2a10da230c70d167d34f5_0_k E0323 09:34:39.535097 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.535931 3922 file_storage.cpp:481] Key not found: 0e1e4ba2445bcdabae622eac574fa961f34b15759784875a329235d2d5e8e849_1_v E0323 09:34:39.535948 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.536794 3922 file_storage.cpp:481] Key not found: a07ad9abf9c2a7c6d8032a2defbc18f450c094a51bd2c8deeae2c436aa0594de_3_v E0323 09:34:39.536810 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.537624 3922 file_storage.cpp:481] Key not found: e3344c975b754a108c5ab69c9358e5ec87733d0f9e9a11efc700ad45ce924db6_0_v E0323 09:34:39.537642 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.539505 3922 file_storage.cpp:481] Key not found: 990c28883ad808f1caed10251278e6d535262f9cdc8f8bc0e1759d2aee67b3bc_0_k E0323 09:34:39.539521 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.542181 3922 file_storage.cpp:481] Key not found: 5ffdd98eca9dece994827d20ac2e1e9043ebeb47f873e0e9dc39dd23553de878_3_v E0323 09:34:39.542197 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.545236 3922 file_storage.cpp:481] Key not found: 8f1e1de78ddebaec444b51013390661c90b0dd614b39f2788e8e8d628701036b_0_k E0323 09:34:39.545253 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.547999 3922 file_storage.cpp:481] Key not found: beaeb76f98aa9f64346259fe19e32e60e1f719f117dcab2b373788d5b07608aa_0_v E0323 09:34:39.548015 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.550714 3922 file_storage.cpp:481] Key not found: 6328fe9393612dd8cca9efa31e127ce6b9d3b37cda6902415366fe01ec9dd158_0_k E0323 09:34:39.550730 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.551554 3922 file_storage.cpp:481] Key not found: b11c692360b133956251adf255327e23b5d319a2231214483720f0ca6917119a_2_k E0323 09:34:39.551571 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY E0323 09:34:39.552387 3922 file_storage.cpp:481] Key not found: 212180d83919ee636abf0851ab29e91a151a3debb21a113ae336aabef21d9a7c_2_v E0323 09:34:39.552403 3922 file_storage.cpp:352] BatchQuerySlices failed with error: INVALID_KEY ``` ### Environment: - Mooncake version: Latest (Release: v0.3.10) - Deployment: 1 Master + 1 RealClient on machine A 1 RealClient on machine B - Protocol: RDMA - Metadata mode: Non-P2PHANDSHAKE (using HTTP metadata server) ### Steps to Reproduce master: ```export MASTER_IP="this node IP" mooncake_master \ --eviction_high_watermark_ratio=0.95 \ --rpc_address ${MASTER_IP} \ --rpc_port 50051 \ --rpc_thread_num=32 \ --enable_offload=true \ --enable_http_metadata_server=true \ --http_metadata_server_port=8080 \ --enable_metric_reporting=true \ --metrics_port 9003 ``` client: ```mkdir -p /data0/mooncake_offload export MASTER_IP="MASTER_IP" export NODE_IP="NODE_IP" export MOONCAKE_LOCAL_HOSTNAME="${NODE_IP}" export MOONCAKE_TE_META_DATA_SERVER="http://${MASTER_IP}:8080/metadata" export MOONCAKE_MASTER="${MASTER_IP}:50051" export MOONCAKE_PROTOCOL="rdma" export MOONCAKE_DEVICE="mlx5_0,mlx5_1" export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/data0/mooncake_offload export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend export MOONCAKE_BUCKET_MAX_TOTAL_SIZE=$((200 * 1024 * 1024 * 1024)) # 200 GB export MOONCAKE_BUCKET_EVICTION_POLICY=lru export MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS=2 mooncake_client \ --master_server_address="${MOONCAKE_MASTER}" \ --host="${NODE_IP}" \ --port=50052 \ --protocol="rdma" \ --device_names="mlx5_0,mlx5_1" \ --global_segment_size="100GB" \ --enable_offload="true" \ --metadata_server="${MOONCAKE_TE_META_DATA_SERVER}" ``` sglang: ```python -m sglang.launch_server \ --model /data0/models/Qwen3-32B-FP8 \ --port 30000 \ --tp 4 \ --enable-hierarchical-cache \ --schedule-policy lpm \ --hicache-storage-prefetch-policy timeout \ --hicache-storage-backend mooncake \ --hicache-size 100 \ --hicache-storage-backend-extra-config '{ "master_server_address": "master_ip:50051", "local_hostname": "local_hostname", "metadata_server": "http://master_ip:8080/metadata", "global_segment_size": "0", "protocol": "rdma", "device_name": "mlx5_0,mlx5_1" }' ``` then run benchmark: ```python3 -m sglang.bench_serving \ --backend sglang \ --model /data0/models/Qwen3-235B-A22B-FP8 \ --tokenizer /data0/models/Qwen3-235B-A22B-FP8 \ --base-url http://localhost:30000 \ --dataset-name random \ --dataset-path /data0/models/ShareGPT_V3_unfiltered_cleaned_split.json \ --num-prompts 288 \ --max-concurrency 20 \ --warmup-requests 16 \ --random-input-len 16384 \ --random-output-len 1138 \ --random-range-ratio 1.0 ``` ### Expected Behavior SSD offload should work correctly, with objects being written to the SSD storage path. ### Actual Behavior RealClient logs show: ```I0323 10:45:04.495337 file_storage.cpp:474] BatchQuerySegmentSlices: key=xxx, found memory replica at endpoint=10.15.56.196:12862, client_->GetTransportEndpoint()=10.154.0.5:15846 E0323 10:45:04.495340 file_storage.cpp:496] Key not found: xxx E0323 10:45:04.495361 file_storage.cpp:357] BatchQuerySlices failed with error: INVALID_KEY ``` No files are written to the SSD offload directory. ### Root Cause Analysis #### The Problem my guess： Metadata Server mode forces segment.te_endpoint = --host IP (business IP), which is inconsistent with the actual RDMA NIC IP. SSD Offload relies on te_endpoint to verify local Replica Key —— mismatch leads to "Key not found", thus Offload aborts. P2PHANDSHAKE mode automatically sets segment.te_endpoint = RDMA NIC actual IP, so the verification passes. In `file_storage.cpp: BatchQuerySegmentSlices` compares addresses incorrectly: ``` if (memory_descriptor.buffer_descriptor.transport_endpoint_ == client_->GetTransportEndpoint()) { ``` #### Why It Fails | Mode | transport_endpoint_ | GetTransportEndpoint() | Result | |-------------------|----------------------------------------|------------------------|----------| | P2P_HANDSHAKE | RDMA IP:PORT (e.g., IP1:Port1) | RDMA IP:PORT | Match | | Non-P2P_HANDSHAKE | local_hostname_ (e.g., IP2:Port2) | RDMA IP:PORT | No Match | In non-P2PHANDSHAKE mode: - transport_endpoint_ = local_hostname_ = user-specified --host parameter - GetTransportEndpoint() = actual RDMA NIC IP:PORT These are different IP addresses in multi-NIC environments, causing the comparison to fail. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:15:31 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:15:32 +00:00

Author

Owner

@Colors-111 commented on GitHub (Mar 23, 2026):

@ykwd @zhangzuo21 ,
I just submitted an issue regarding the SSD Offload failure in Metadata Server mode.
Could you please take a look when you have time?
Thanks a lot for your help!

@Colors-111 commented on GitHub (Mar 23, 2026): @ykwd @zhangzuo21 , I just submitted an issue regarding the SSD Offload failure in Metadata Server mode. Could you please take a look when you have time? Thanks a lot for your help!

bsakaguchi commented

2026-10-29 21:15:32 +00:00

Author

Owner

@zhangzuo21 commented on GitHub (Mar 24, 2026):

Thanks for your report and analysis! I'll dive into the problem these days.

@zhangzuo21 commented on GitHub (Mar 24, 2026): Thanks for your report and analysis! I'll dive into the problem these days.

bsakaguchi commented

2026-10-29 21:15:32 +00:00

Author

Owner

@zhangzuo21 commented on GitHub (Mar 30, 2026):

Hi @Colors-111, this has been fixed in PR #1771 . Thanks for the detailed report and root cause analysis!

@zhangzuo21 commented on GitHub (Mar 30, 2026): Hi @Colors-111, this has been fixed in PR #1771 . Thanks for the detailed report and root cause analysis!

bsakaguchi referenced this issue

2026-10-29 21:22:21 +00:00

[GH-ISSUE #4181] [RFC]: [Store] The client heartbeat path must share no lock or thread with data-plane operations #1053

bsakaguchi referenced this issue

2026-10-29 21:28:52 +00:00

[PR #501] [MERGED] [Store] High Availability V2: Client Failover #1609