kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2011] [Bug]: SGLang HiCache + Mooncake SSD offload: no disk read-back after eviction, `Expected DiskDescriptor` during prefetch #557

New issue

Closed

opened 2026-10-29 21:16:23 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:16:23 +00:00

Owner

Originally created by @shengjia-al on GitHub (Apr 29, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2011

Bug Report

Describe the bug

I am testing SGLang HiCache with Mooncake SSD/file offload. The write-through path works: Mooncake creates .bucket / .meta files and master_allocated_file_size_bytes grows. However, after forcing Mooncake eviction, I cannot observe successful read-back from the offload file backend.

All Mooncake client batch_get_into logs still show:

read store: 0us, with memory key count: N, offload key count: 0

At the same time, SGLang prefetch reports:

Batch RPC call failed: Expected DiskDescriptor
Prefetch operation ... failed to retrieve page ...

This was reproduced with the SSD offload environment variables documented by Mooncake, including MOONCAKE_OFFLOAD_USE_URING=1.

Expected behavior: after memory entries are evicted and the data exists in the Mooncake file backend, SGLang HiCache prefetch should be able to read those entries back from SSD/file offload, or the required extra configuration should be documented.

Reproduction

Model:

model path: /xuntui-test/models/Qwen3-1.7B
model size: 1.7B
quantization: none
TP/DP/PP: TP=1, DP=1, PP=1
PD disaggregation: disabled
GPU used for this test: CUDA_VISIBLE_DEVICES=2

Start Mooncake master:

mooncake_master \
  --rpc_port=53851 \
  --enable_offload=true \
  --enable_http_metadata_server=true \
  --http_metadata_server_host=0.0.0.0 \
  --http_metadata_server_port=8468 \
  --metrics_port=9384 \
  --root_fs_dir=/local_nvme/mooncake_offload_qwen1p7b_offload_read_official_hwm70_20260429_123335 \
  --eviction_high_watermark_ratio=0.70 \
  --eviction_ratio=0.20 \
  --logtostderr

Start Mooncake client:

MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=$(realpath -e /local_nvme/mooncake_offload_qwen1p7b_offload_read_official_hwm70_20260429_123335) \
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=$((2 * 1024 * 1024 * 1024)) \
MOONCAKE_OFFLOAD_USE_URING=1 \
MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend \
mooncake_client \
  --host=192.168.1.9 \
  --global_segment_size=32GB \
  --master_server_address=192.168.1.9:53851 \
  --protocol=rdma \
  --device_names=erdma_1 \
  --enable_offload=true \
  --port=53852 \
  --metadata_server=P2PHANDSHAKE \
  --logtostderr

Start SGLang:

CUDA_VISIBLE_DEVICES=2 \
MOONCAKE_PROTOCOL=rdma \
MOONCAKE_LOCAL_HOSTNAME=192.168.1.9 \
MOONCAKE_DEVICE=erdma_1 \
MOONCAKE_GLOBAL_SEGMENT_SIZE=0 \
MOONCAKE_STANDALONE_STORAGE=1 \
MOONCAKE_CLIENT=192.168.1.9:53852 \
python -m sglang.launch_server \
  --model-path /xuntui-test/models/Qwen3-1.7B \
  --tp-size 1 \
  --host 0.0.0.0 \
  --port 33801 \
  --nccl-port 29981 \
  --trust-remote-code \
  --mem-fraction-static 0.40 \
  --max-total-tokens 65536 \
  --page-size 64 \
  --admin-api-key abcdefg \
  --enable-hierarchical-cache \
  --hicache-size 8 \
  --hicache-storage-prefetch-policy wait_complete \
  --hicache-mem-layout page_first_direct \
  --hicache-storage-backend mooncake

Run workload:

python /home/junda/test_tcp/bench_multiturn.py \
  --host 127.0.0.1 \
  --port 33801 \
  --model-path /xuntui-test/models/Qwen3-1.7B \
  --num-clients 12 \
  --request-length 4096 \
  --output-length 8 \
  --num-rounds 5 \
  --request-rate 2 \
  --disable-auto-run \
  --max-parallel 2 \
  --enable-round-barrier \
  --seed 2 \
  --tag official_hwm70_wait_complete_c12_r5_rate2 \
  --log-file bench_official_c12_r5_rate2.jsonl

Observed benchmark result:

Total requests: 60
Average TTFT: 0.69s
P99 TTFT: 2.29s
Average latency: 0.88s
Request throughput: 1.37 req/s
Cache Hit Rate: 0.316255

Mooncake metrics after the run:

master_allocated_bytes 23880798208
master_total_capacity_bytes 34359738368
master_allocated_file_size_bytes 27715964928
master_key_count 7553
master_successful_evictions_total 1
master_attempted_evictions_total 1

Filesystem evidence:

offload dir size: 52G
file count: 7759

But no offload reads were observed:

grep -E "offload key count: [1-9]" client.log | wc -l
# 0

grep -E "read store: [1-9]" client.log | wc -l
# 0

Representative logs:

Time taken for batch_get_into: 42872us, read store: 0us, with memory key count: 128, offload key count: 0
Time taken for batch_get_into: 88695us, read store: 0us, with memory key count: 256, offload key count: 0

E0429 12:36:23.474697 dummy_client.cpp:189] Batch RPC call failed: Expected DiskDescriptor
[2026-04-29 12:36:23] Prefetch operation ... failed to retrieve page ...

Error counters:

Expected DiskDescriptor: 4
failed to retrieve page: 2
BatchPut failed: 0
Write page to storage failed: 0
Request failed: 0

Control run: if I do not force eviction and only increase --global_segment_size to avoid high-watermark pressure, the same SGLang + Mooncake write-through path is stable:

Cache hit rate: 0.349545
BatchPut failed: 0
Expected DiskDescriptor: 0
Prefetch failed: 0
master_successful_evictions_total: 0

This suggests the issue appears when entering the eviction / offload read-back path, not during basic Mooncake connection or write-through file creation.

Environment

Output of python -m sglang.check_env:

Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3: NVIDIA L20
GPU 0,1,2,3 Compute Capability: 8.9
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 580.126.09
PyTorch: 2.9.1+cu128
sglang: 0.5.9
sgl_kernel: 0.3.21
flashinfer_python: 0.6.3
flashinfer_cubin: 0.6.3
flashinfer_jit_cache: Module Not Found
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.4.4
aiohttp: 3.13.5
fastapi: 0.136.1
vllm: Module Not Found
xgrammar: 0.1.27
ulimit soft: 65535

Additional environment:

OS: Ubuntu 24.04.4 LTS
Kernel: Linux 6.8.0-106-generic x86_64
NVIDIA driver: 580.126.09
nvidia-smi CUDA version: 13.0
torch CUDA version: 12.8
mooncake-transfer-engine: 0.3.10.post2
sglang: 0.5.9
torch: 2.9.1+cu128
sglang-kernel: 0.3.21
vllm: not installed
Docker image: not used; this is a bare-metal/venv test

GPU:

4 x NVIDIA L20, 46068 MiB each

RDMA:

device: erdma_1
transport: eRDMA
fw_ver: 0.2.0
state: PORT_ACTIVE
link_layer: Ethernet
host IP used by Mooncake: 192.168.1.9

Disk:

/local_nvme is ext4 on /dev/nvme0n1p3
capacity: 295G
available after cleanup: 130G

Mooncake client startup confirms file backend and RDMA:

Storage root directory is: /local_nvme/mooncake_offload_...
Disk eviction enabled: 1
Using specified RDMA devices: erdma_1
RDMA device: erdma_1, ... c0:a8:01:09
[SharedUringRing] thread-local ring initialised queue_depth=32
Bucket backend max_total_size not set; using 90% of disk capacity: 285001585459 bytes
Starting real client service on 192.168.1.9:53852

One startup warning:

io_uring_register_buffers failed ... falling back to non-fixed-buffer I/O
I/O will use regular (non-fixed-buffer) io_uring, which is correct but slightly less optimal

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @shengjia-al on GitHub (Apr 29, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2011 ### Bug Report ## Describe the bug I am testing SGLang HiCache with Mooncake SSD/file offload. The write-through path works: Mooncake creates `.bucket` / `.meta` files and `master_allocated_file_size_bytes` grows. However, after forcing Mooncake eviction, I cannot observe successful read-back from the offload file backend. All Mooncake client `batch_get_into` logs still show: ```text read store: 0us, with memory key count: N, offload key count: 0 ``` At the same time, SGLang prefetch reports: ```text Batch RPC call failed: Expected DiskDescriptor Prefetch operation ... failed to retrieve page ... ``` This was reproduced with the SSD offload environment variables documented by Mooncake, including `MOONCAKE_OFFLOAD_USE_URING=1`. Expected behavior: after memory entries are evicted and the data exists in the Mooncake file backend, SGLang HiCache prefetch should be able to read those entries back from SSD/file offload, or the required extra configuration should be documented. ## Reproduction Model: ```text model path: /xuntui-test/models/Qwen3-1.7B model size: 1.7B quantization: none TP/DP/PP: TP=1, DP=1, PP=1 PD disaggregation: disabled GPU used for this test: CUDA_VISIBLE_DEVICES=2 ``` Start Mooncake master: ```bash mooncake_master \ --rpc_port=53851 \ --enable_offload=true \ --enable_http_metadata_server=true \ --http_metadata_server_host=0.0.0.0 \ --http_metadata_server_port=8468 \ --metrics_port=9384 \ --root_fs_dir=/local_nvme/mooncake_offload_qwen1p7b_offload_read_official_hwm70_20260429_123335 \ --eviction_high_watermark_ratio=0.70 \ --eviction_ratio=0.20 \ --logtostderr ``` Start Mooncake client: ```bash MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=$(realpath -e /local_nvme/mooncake_offload_qwen1p7b_offload_read_official_hwm70_20260429_123335) \ MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=$((2 * 1024 * 1024 * 1024)) \ MOONCAKE_OFFLOAD_USE_URING=1 \ MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend \ mooncake_client \ --host=192.168.1.9 \ --global_segment_size=32GB \ --master_server_address=192.168.1.9:53851 \ --protocol=rdma \ --device_names=erdma_1 \ --enable_offload=true \ --port=53852 \ --metadata_server=P2PHANDSHAKE \ --logtostderr ``` Start SGLang: ```bash CUDA_VISIBLE_DEVICES=2 \ MOONCAKE_PROTOCOL=rdma \ MOONCAKE_LOCAL_HOSTNAME=192.168.1.9 \ MOONCAKE_DEVICE=erdma_1 \ MOONCAKE_GLOBAL_SEGMENT_SIZE=0 \ MOONCAKE_STANDALONE_STORAGE=1 \ MOONCAKE_CLIENT=192.168.1.9:53852 \ python -m sglang.launch_server \ --model-path /xuntui-test/models/Qwen3-1.7B \ --tp-size 1 \ --host 0.0.0.0 \ --port 33801 \ --nccl-port 29981 \ --trust-remote-code \ --mem-fraction-static 0.40 \ --max-total-tokens 65536 \ --page-size 64 \ --admin-api-key abcdefg \ --enable-hierarchical-cache \ --hicache-size 8 \ --hicache-storage-prefetch-policy wait_complete \ --hicache-mem-layout page_first_direct \ --hicache-storage-backend mooncake ``` Run workload: ```bash python /home/junda/test_tcp/bench_multiturn.py \ --host 127.0.0.1 \ --port 33801 \ --model-path /xuntui-test/models/Qwen3-1.7B \ --num-clients 12 \ --request-length 4096 \ --output-length 8 \ --num-rounds 5 \ --request-rate 2 \ --disable-auto-run \ --max-parallel 2 \ --enable-round-barrier \ --seed 2 \ --tag official_hwm70_wait_complete_c12_r5_rate2 \ --log-file bench_official_c12_r5_rate2.jsonl ``` Observed benchmark result: ```text Total requests: 60 Average TTFT: 0.69s P99 TTFT: 2.29s Average latency: 0.88s Request throughput: 1.37 req/s Cache Hit Rate: 0.316255 ``` Mooncake metrics after the run: ```text master_allocated_bytes 23880798208 master_total_capacity_bytes 34359738368 master_allocated_file_size_bytes 27715964928 master_key_count 7553 master_successful_evictions_total 1 master_attempted_evictions_total 1 ``` Filesystem evidence: ```text offload dir size: 52G file count: 7759 ``` But no offload reads were observed: ```bash grep -E "offload key count: [1-9]" client.log | wc -l # 0 grep -E "read store: [1-9]" client.log | wc -l # 0 ``` Representative logs: ```text Time taken for batch_get_into: 42872us, read store: 0us, with memory key count: 128, offload key count: 0 Time taken for batch_get_into: 88695us, read store: 0us, with memory key count: 256, offload key count: 0 E0429 12:36:23.474697 dummy_client.cpp:189] Batch RPC call failed: Expected DiskDescriptor [2026-04-29 12:36:23] Prefetch operation ... failed to retrieve page ... ``` Error counters: ```text Expected DiskDescriptor: 4 failed to retrieve page: 2 BatchPut failed: 0 Write page to storage failed: 0 Request failed: 0 ``` Control run: if I do not force eviction and only increase `--global_segment_size` to avoid high-watermark pressure, the same SGLang + Mooncake write-through path is stable: ```text Cache hit rate: 0.349545 BatchPut failed: 0 Expected DiskDescriptor: 0 Prefetch failed: 0 master_successful_evictions_total: 0 ``` This suggests the issue appears when entering the eviction / offload read-back path, not during basic Mooncake connection or write-through file creation. ## Environment Output of `python -m sglang.check_env`: ```text Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3: NVIDIA L20 GPU 0,1,2,3 Compute Capability: 8.9 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.8, V12.8.93 CUDA Driver Version: 580.126.09 PyTorch: 2.9.1+cu128 sglang: 0.5.9 sgl_kernel: 0.3.21 flashinfer_python: 0.6.3 flashinfer_cubin: 0.6.3 flashinfer_jit_cache: Module Not Found triton: 3.5.1 transformers: 4.57.1 torchao: 0.9.0 numpy: 2.4.4 aiohttp: 3.13.5 fastapi: 0.136.1 vllm: Module Not Found xgrammar: 0.1.27 ulimit soft: 65535 ``` Additional environment: ```text OS: Ubuntu 24.04.4 LTS Kernel: Linux 6.8.0-106-generic x86_64 NVIDIA driver: 580.126.09 nvidia-smi CUDA version: 13.0 torch CUDA version: 12.8 mooncake-transfer-engine: 0.3.10.post2 sglang: 0.5.9 torch: 2.9.1+cu128 sglang-kernel: 0.3.21 vllm: not installed Docker image: not used; this is a bare-metal/venv test ``` GPU: ```text 4 x NVIDIA L20, 46068 MiB each ``` RDMA: ```text device: erdma_1 transport: eRDMA fw_ver: 0.2.0 state: PORT_ACTIVE link_layer: Ethernet host IP used by Mooncake: 192.168.1.9 ``` Disk: ```text /local_nvme is ext4 on /dev/nvme0n1p3 capacity: 295G available after cleanup: 130G ``` Mooncake client startup confirms file backend and RDMA: ```text Storage root directory is: /local_nvme/mooncake_offload_... Disk eviction enabled: 1 Using specified RDMA devices: erdma_1 RDMA device: erdma_1, ... c0:a8:01:09 [SharedUringRing] thread-local ring initialised queue_depth=32 Bucket backend max_total_size not set; using 90% of disk capacity: 285001585459 bytes Starting real client service on 192.168.1.9:53852 ``` One startup warning: ```text io_uring_register_buffers failed ... falling back to non-fixed-buffer I/O I/O will use regular (non-fixed-buffer) io_uring, which is correct but slightly less optimal ``` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:16:23 +00:00

closed this issue
added the
stale

auto-closed

bug
labels

bsakaguchi commented

2026-10-29 21:16:24 +00:00

Author

Owner

@LujhCoconut commented on GitHub (Apr 29, 2026):

This bug appears to be the same issue addressed in my PR, which provides a more detailed description in PR(#2004). I guess this will be fixed once this PR is merged.

@LujhCoconut commented on GitHub (Apr 29, 2026): This bug appears to be the same issue addressed in my [PR](https://github.com/kvcache-ai/Mooncake/pull/2004), which provides a more detailed description in PR(#2004). I guess this will be fixed once this PR is merged.

bsakaguchi commented

2026-10-29 21:16:24 +00:00

Author

Owner

@shengjia-al commented on GitHub (Apr 29, 2026):

这个 bug 似乎和我之前提交的PR中描述的问题一样，PR（ #2004 ）里有更详细的描述。我猜这个 PR 合并后这个问题就能解决了。

Thanks for pointing this out!

I just read through your PR description. Although I'm not familiar with the underlying C++ implementation, the behavior you described matches my experience exactly. In my case with SGLang, once the memory is full and data is evicted to SSD, the system completely fails to read it back (showing offload key count: 0 and throwing Expected DiskDescriptor).

It seems like we hit the same SSD read-back roadblock, just from different serving frameworks.

Thank you for working on this fix! I will keep an eye on PR #2004 and look forward to testing it once it gets merged

@shengjia-al commented on GitHub (Apr 29, 2026): > [这个 bug 似乎和我之前提交的PR](https://github.com/kvcache-ai/Mooncake/pull/2004)中描述的问题一样，PR（ [#2004](https://github.com/kvcache-ai/Mooncake/pull/2004) ）里有更详细的描述。我猜这个 PR 合并后这个问题就能解决了。 Thanks for pointing this out! I just read through your PR description. Although I'm not familiar with the underlying C++ implementation, the behavior you described matches my experience exactly. In my case with SGLang, once the memory is full and data is evicted to SSD, the system completely fails to read it back (showing offload key count: 0 and throwing Expected DiskDescriptor). It seems like we hit the same SSD read-back roadblock, just from different serving frameworks. Thank you for working on this fix! I will keep an eye on PR #2004 and look forward to testing it once it gets merged

bsakaguchi commented

2026-10-29 21:16:24 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 29, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 29, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:16:24 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 6, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Aug 6, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:29:11 +00:00

[PR #557] [MERGED] Optimize slice handling to accelerate the large batch transfer operation #1649

bsakaguchi referenced this issue

2026-10-29 21:29:15 +00:00

[PR #564] [MERGED] add support for asynchronous batch transfer to accelerate transfer operation #1657