[GH-ISSUE #2156] [Bug]: Assertion `sizes[i] <= kMaxSliceSize' failed when using SSD offload #594

New issue

Closed

opened 2026-10-29 21:16:56 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 21:16:56 +00:00

Owner

Copy link

Originally created by @TrafalgarZZZ on GitHub (May 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2156

Originally assigned to: @LujhCoconut on GitHub.

Bug Report

Bug Description

I'm following the Mooncake SSD Offloading Benchmark doc to verify Mooncake store's KV cache offloading feature. It consistently fails with the following error:

mooncake_client: /home/runner/work/Mooncake/Mooncake/mooncake-store/src/file_storage.cpp:598: tl::expected<std::shared_ptr<mooncake::FileStorage::AllocatedBatch>, mooncake::ErrorCode> mooncake::FileStorage::AllocateBatch(const std::vector<std::__cxx11::basic_string<char> >&, const std::vector<long int>&): Assertion `sizes[i] <= kMaxSliceSize' failed.

Steps to Reproduce

• sglang: v0.5.12
• mooncake-transfer-engine: 0.3.10.post2

Here's my steps following the "Mooncake SSD Offloading Benchmark" doc:

1. Start mooncake master

mooncake_master \
    -enable_offload=true \
    -http_metadata_server_port=8081 \
    -metrics_port=9004 \
    -logtostderr

2. Start mooncake client

MOONCAKE_OFFLOAD_FILE_STORAGE_PATH="/mnt/nvme/mooncake_store/" \
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=21474836480 \
MOONCAKE_OFFLOAD_USE_URING=1 \
mooncake_client \
    --host=127.0.0.1 \
    --global_segment_size=20GB \
    --master_server_address=localhost:50051 \
    --metadata_server=P2PHANDSHAKE \
    --protocol=rdma \
    --enable_offload=true \
    --port=50052 \
    --logtostderr

3. Start SGLang server (with HiCache enabled & set hicache storage backend to mooncake)

MOONCAKE_MASTER="127.0.0.1:50051" \
MOONCAKE_GLOBAL_SEGMENT_SIZE=0 \
MOONCAKE_PROTOCOL="rdma" \
python3 -m sglang.launch_server --model-path /mount/model/Qwen3.6-35B-A3B/  --port 30000 --host 0.0.0.0 --enable-metrics --tp=2 --mem-fraction-static 0.30 --page-size 64 --mamba-scheduler-strategy extra_buffer --enable-hierarchical-cache --hicache-write-policy write_through --hicache-ratio 2 --hicache-storage-prefetch-policy wait_complete --hicache-storage-backend mooncake --hicache-mem-layout page_first_direct --attention-backend triton

4. Benchmark the SGLang server with multi-turn benchmark script

python3 benchmark/hicache/bench_multiturn.py --host localhost --port 30000 --model-path /mount/model/Qwen3.6-35B-A3B/ --disable-random-sample --output-length 16 --request-length 12000 --num-clients 50 --num-rounds 5 --max-parallel 1 --request-rate 1 --ready-queue-policy random --disable-auto-run --seed 42 --dataset-path /root/ShareGPT_V3_unfiltered_cleaned_split.json --enable-round-barrier

Initially, the system works well when there's no eviction in Mooncake store. When the 20 GiB Memory (set by --global_segment_size) runs out, it reports something like:

I0520 11:27:31.674053  7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 17.53 GB / 20.00 GB (87.7%) | SSD Storage: 0 B / 0 B | Keys: 37818 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=128/128, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=408/0/408, Item=37816/37816), PutEnd:(Req=408/0/408, Item=37816/37816), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=78/0/78, Item=37712/37712), ExistKey:(Req=508/0/508, Item=75216/75216), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=128/128), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=0/20, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0}
I0520 11:27:41.674199  7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.58 GB / 20.00 GB (92.9%) | SSD Storage: 0 B / 0 B | Keys: 40098 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=138/138, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=432/0/432, Item=40096/40096), PutEnd:(Req=432/0/432, Item=40096/40096), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=78/0/78, Item=37712/37712), ExistKey:(Req=540/0/540, Item=80504/80504), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=138/138), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=0/20, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0}
I0520 11:27:42.619714  7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.96399 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05
I0520 11:27:42.634616  7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed.
I0520 11:27:48.084128  7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.956615 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05
I0520 11:27:48.098935  7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed.
I0520 11:27:51.674341  7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.72 GB / 20.00 GB (93.6%) | SSD Storage: 0 B / 0 B | Keys: 45418 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=148/148, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=488/0/488, Item=45416/45416), PutEnd:(Req=488/0/488, Item=45416/45416), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=89/0/89, Item=42849/42849), ExistKey:(Req=608/0/608, Item=90336/90336), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=148/148), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=2/22, keys=5150, size=2.31 GB | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0}
I0520 11:27:52.935034  7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.953358 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05
I0520 11:27:52.950444  7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed.
I0520 11:27:58.329479  7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.954269 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05
I0520 11:27:58.346585  7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed.
I0520 11:28:01.674479  7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.34 GB / 20.00 GB (91.7%) | SSD Storage: 0 B / 0 B | Keys: 49978 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=158/158, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=536/0/536, Item=49976/49976), PutEnd:(Req=536/0/536, Item=49976/49976), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=102/0/102, Item=49218/49218), ExistKey:(Req=670/0/670, Item=100160/100160), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=158/158), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=4/24, keys=10285, size=4.78 GB | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0}
I0520 11:28:02.881091  7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.951984 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05
I0520 11:28:02.897092  7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed.

but some short time later (I think it's the time SGLang trying to get the offloaded object), mooncake client complains the error and crashed:

mooncake_client: /home/runner/work/Mooncake/Mooncake/mooncake-store/src/file_storage.cpp:598: tl::expected<std::shared_ptr<mooncake::FileStorage::AllocatedBatch>, mooncake::ErrorCode> mooncake::FileStorage::AllocateBatch(const std::vector<std::__cxx11::basic_string<char> >&, const std::vector<long int>&): Assertion `sizes[i] <= kMaxSliceSize' failed.

Meanwhile, SGLang reports errors like:

[2026-05-20 11:28:23] INFO:     127.0.0.1:59776 - "POST /generate HTTP/1.1" 200 OK
E0520 11:28:25.865783 10058 real_client.cpp:4336] RPC call failed: End of file
2026-05-20 11:28:25.865693 ERROR    [10057] [coro_rpc_client.hpp:1359] read rpc head failed, error msg:Connection reset by peer. close the socket.value=104
E0520 11:28:25.865811 10057 real_client.cpp:4336] RPC call failed: Connection reset by peer
E0520 11:28:25.865830 10595 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL
E0520 11:28:25.865854 10595 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL
E0520 11:28:25.865856 10595 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL
I0520 11:28:25.865859 10595 real_client.cpp:3556] Time taken for batch_get_into: 2283824us, read store: 2283730us, with memory key count: 0, offload key count: 2
E0520 11:28:25.865865 10848 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL
E0520 11:28:25.865890 10848 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL
E0520 11:28:25.865892 10848 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL
I0520 11:28:25.865895 10848 real_client.cpp:3556] Time taken for batch_get_into: 2283862us, read store: 2283759us, with memory key count: 0, offload key count: 2
2026-05-20 11:28:25.867156 WARNING  [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:25.867136 WARNING  [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:26.985795 WARNING  [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:26.985799 WARNING  [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:27.890918 WARNING  [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:27.890929 WARNING  [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:29.042038 WARNING  [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:29.042057 WARNING  [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused
2026-05-20 11:28:30.133056 WARNING  [10061] [client_pool.hpp:207] reconnect client{0x7f5634059c70},host:{127.0.0.1:50052} out of max limit, stop retry. connect failed
2026-05-20 11:28:30.133075 WARNING  [10061] [client_pool.hpp:445] send request to 127.0.0.1:50052 failed. connection refused.
E0520 11:28:30.133085 10061 real_client.cpp:4331] Dummy Client not available
2026-05-20 11:28:30.133078 WARNING  [10062] [client_pool.hpp:207] reconnect client{0x7fa08c0599d0},host:{127.0.0.1:50052} out of max limit, stop retry. connect failed
2026-05-20 11:28:30.133097 WARNING  [10062] [client_pool.hpp:445] send request to 127.0.0.1:50052 failed. connection refused.
E0520 11:28:30.133106 10062 real_client.cpp:4331] Dummy Client not available
E0520 11:28:30.133113 10848 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL
E0520 11:28:30.133126 10848 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL
E0520 11:28:30.133138 10848 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL
E0520 11:28:30.133138 10595 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL
E0520 11:28:30.133153 10595 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL
E0520 11:28:30.133165 10595 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL
I0520 11:28:30.133186 10848 real_client.cpp:3556] Time taken for batch_get_into: 4266945us, read store: 4266150us, with memory key count: 0, offload key count: 256
I0520 11:28:30.133201 10595 real_client.cpp:3556] Time taken for batch_get_into: 4267002us, read store: 4266200us, with memory key count: 0, offload key count: 256
[2026-05-20 11:28:30 TP0] Prefetch operation dcaf285ba4594e74a998305584a49008 failed to retrieve page 2adfc774d05ad5f7114513c9b1e16ce8a4d6c381f05283a076bc7b64bcce2da5.
[2026-05-20 11:28:30 TP1] Prefetch operation dcaf285ba4594e74a998305584a49008 failed to retrieve page 2adfc774d05ad5f7114513c9b1e16ce8a4d6c381f05283a076bc7b64bcce2da5.

Is it a bug? Or maybe I missed something that need to be pre-configured.

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

Originally created by @TrafalgarZZZ on GitHub (May 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2156 Originally assigned to: @LujhCoconut on GitHub. ### Bug Report ## Bug Description I'm following the [Mooncake SSD Offloading Benchmark](https://kvcache-ai.github.io/Mooncake/performance/ssd-offload-benchmark-results.html) doc to verify Mooncake store's KV cache offloading feature. It consistently fails with the following error: ``` mooncake_client: /home/runner/work/Mooncake/Mooncake/mooncake-store/src/file_storage.cpp:598: tl::expected<std::shared_ptr<mooncake::FileStorage::AllocatedBatch>, mooncake::ErrorCode> mooncake::FileStorage::AllocateBatch(const std::vector<std::__cxx11::basic_string<char> >&, const std::vector<long int>&): Assertion `sizes[i] <= kMaxSliceSize' failed. ``` ## Steps to Reproduce - **sglang**: v0.5.12 - **mooncake-transfer-engine**: 0.3.10.post2 Here's my steps following the "Mooncake SSD Offloading Benchmark" doc: 1. Start mooncake master ``` mooncake_master \ -enable_offload=true \ -http_metadata_server_port=8081 \ -metrics_port=9004 \ -logtostderr ``` 2. Start mooncake client ``` MOONCAKE_OFFLOAD_FILE_STORAGE_PATH="/mnt/nvme/mooncake_store/" \ MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=21474836480 \ MOONCAKE_OFFLOAD_USE_URING=1 \ mooncake_client \ --host=127.0.0.1 \ --global_segment_size=20GB \ --master_server_address=localhost:50051 \ --metadata_server=P2PHANDSHAKE \ --protocol=rdma \ --enable_offload=true \ --port=50052 \ --logtostderr ``` 3. Start SGLang server (with HiCache enabled & set hicache storage backend to mooncake) ``` MOONCAKE_MASTER="127.0.0.1:50051" \ MOONCAKE_GLOBAL_SEGMENT_SIZE=0 \ MOONCAKE_PROTOCOL="rdma" \ python3 -m sglang.launch_server --model-path /mount/model/Qwen3.6-35B-A3B/ --port 30000 --host 0.0.0.0 --enable-metrics --tp=2 --mem-fraction-static 0.30 --page-size 64 --mamba-scheduler-strategy extra_buffer --enable-hierarchical-cache --hicache-write-policy write_through --hicache-ratio 2 --hicache-storage-prefetch-policy wait_complete --hicache-storage-backend mooncake --hicache-mem-layout page_first_direct --attention-backend triton ``` 4. Benchmark the SGLang server with multi-turn benchmark script ``` python3 benchmark/hicache/bench_multiturn.py --host localhost --port 30000 --model-path /mount/model/Qwen3.6-35B-A3B/ --disable-random-sample --output-length 16 --request-length 12000 --num-clients 50 --num-rounds 5 --max-parallel 1 --request-rate 1 --ready-queue-policy random --disable-auto-run --seed 42 --dataset-path /root/ShareGPT_V3_unfiltered_cleaned_split.json --enable-round-barrier ``` Initially, the system works well when there's no eviction in Mooncake store. When the 20 GiB Memory (set by `--global_segment_size`) runs out, it reports something like: ``` I0520 11:27:31.674053 7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 17.53 GB / 20.00 GB (87.7%) | SSD Storage: 0 B / 0 B | Keys: 37818 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=128/128, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=408/0/408, Item=37816/37816), PutEnd:(Req=408/0/408, Item=37816/37816), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=78/0/78, Item=37712/37712), ExistKey:(Req=508/0/508, Item=75216/75216), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=128/128), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/20, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0} I0520 11:27:41.674199 7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.58 GB / 20.00 GB (92.9%) | SSD Storage: 0 B / 0 B | Keys: 40098 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=138/138, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=432/0/432, Item=40096/40096), PutEnd:(Req=432/0/432, Item=40096/40096), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=78/0/78, Item=37712/37712), ExistKey:(Req=540/0/540, Item=80504/80504), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=138/138), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/20, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0} I0520 11:27:42.619714 7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.96399 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05 I0520 11:27:42.634616 7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed. I0520 11:27:48.084128 7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.956615 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05 I0520 11:27:48.098935 7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed. I0520 11:27:51.674341 7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.72 GB / 20.00 GB (93.6%) | SSD Storage: 0 B / 0 B | Keys: 45418 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=148/148, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=488/0/488, Item=45416/45416), PutEnd:(Req=488/0/488, Item=45416/45416), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=89/0/89, Item=42849/42849), ExistKey:(Req=608/0/608, Item=90336/90336), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=148/148), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=2/22, keys=5150, size=2.31 GB | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0} I0520 11:27:52.935034 7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.953358 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05 I0520 11:27:52.950444 7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed. I0520 11:27:58.329479 7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.954269 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05 I0520 11:27:58.346585 7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed. I0520 11:28:01.674479 7017 rpc_service.cpp:245] Master Admin Metrics: role=leader, state=serving, summary=role=leader, state=serving, service_ready=true, master={Mem Storage: 18.34 GB / 20.00 GB (91.7%) | SSD Storage: 0 B / 0 B | Keys: 49978 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total): PutStart=2/22, PutEnd=2/2, PutRevoke=0/0, Get=2/2, Exist=2/2, Del=0/0, DelAll=0/0, Ping=158/158, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0, EvictDiskReplica=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=536/0/536, Item=49976/49976), PutEnd:(Req=536/0/536, Item=49976/49976), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=102/0/102, Item=49218/49218), ExistKey:(Req=670/0/670, Item=100160/100160), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=158/158), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=4/24, keys=10285, size=4.78 GB | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0} I0520 11:28:02.881091 7011 master_service.cpp:2220] [EVICT-TRIGGER] memory_ratio=0.951984 high_watermark=0.95 need_eviction=0 eviction_ratio=0.05 I0520 11:28:02.897092 7011 master_service.cpp:2231] [EVICT-DONE] BatchEvict execution completed. ``` but some short time later (I think it's the time SGLang trying to get the offloaded object), mooncake client complains the error and crashed: ``` mooncake_client: /home/runner/work/Mooncake/Mooncake/mooncake-store/src/file_storage.cpp:598: tl::expected<std::shared_ptr<mooncake::FileStorage::AllocatedBatch>, mooncake::ErrorCode> mooncake::FileStorage::AllocateBatch(const std::vector<std::__cxx11::basic_string<char> >&, const std::vector<long int>&): Assertion `sizes[i] <= kMaxSliceSize' failed. ``` Meanwhile, SGLang reports errors like: ``` [2026-05-20 11:28:23] INFO: 127.0.0.1:59776 - "POST /generate HTTP/1.1" 200 OK E0520 11:28:25.865783 10058 real_client.cpp:4336] RPC call failed: End of file 2026-05-20 11:28:25.865693 ERROR [10057] [coro_rpc_client.hpp:1359] read rpc head failed, error msg:Connection reset by peer. close the socket.value=104 E0520 11:28:25.865811 10057 real_client.cpp:4336] RPC call failed: Connection reset by peer E0520 11:28:25.865830 10595 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL E0520 11:28:25.865854 10595 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL E0520 11:28:25.865856 10595 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL I0520 11:28:25.865859 10595 real_client.cpp:3556] Time taken for batch_get_into: 2283824us, read store: 2283730us, with memory key count: 0, offload key count: 2 E0520 11:28:25.865865 10848 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL E0520 11:28:25.865890 10848 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL E0520 11:28:25.865892 10848 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL I0520 11:28:25.865895 10848 real_client.cpp:3556] Time taken for batch_get_into: 2283862us, read store: 2283759us, with memory key count: 0, offload key count: 2 2026-05-20 11:28:25.867156 WARNING [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:25.867136 WARNING [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:26.985795 WARNING [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:26.985799 WARNING [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:27.890918 WARNING [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:27.890929 WARNING [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:29.042038 WARNING [10061] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:29.042057 WARNING [10062] [coro_rpc_client.hpp:989] client_id 5 failed:Connection refused 2026-05-20 11:28:30.133056 WARNING [10061] [client_pool.hpp:207] reconnect client{0x7f5634059c70},host:{127.0.0.1:50052} out of max limit, stop retry. connect failed 2026-05-20 11:28:30.133075 WARNING [10061] [client_pool.hpp:445] send request to 127.0.0.1:50052 failed. connection refused. E0520 11:28:30.133085 10061 real_client.cpp:4331] Dummy Client not available 2026-05-20 11:28:30.133078 WARNING [10062] [client_pool.hpp:207] reconnect client{0x7fa08c0599d0},host:{127.0.0.1:50052} out of max limit, stop retry. connect failed 2026-05-20 11:28:30.133097 WARNING [10062] [client_pool.hpp:445] send request to 127.0.0.1:50052 failed. connection refused. E0520 11:28:30.133106 10062 real_client.cpp:4331] Dummy Client not available E0520 11:28:30.133113 10848 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL E0520 11:28:30.133126 10848 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL E0520 11:28:30.133138 10848 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL E0520 11:28:30.133138 10595 real_client.cpp:4294] Failed to invoke batch_get_offload_object, client_addr = 127.0.0.1:50052, error is: RPC_FAIL E0520 11:28:30.133153 10595 real_client.cpp:4227] Batch get offload object failed with error: RPC_FAIL E0520 11:28:30.133165 10595 real_client.cpp:3538] Batch get store object failed with error: RPC_FAIL I0520 11:28:30.133186 10848 real_client.cpp:3556] Time taken for batch_get_into: 4266945us, read store: 4266150us, with memory key count: 0, offload key count: 256 I0520 11:28:30.133201 10595 real_client.cpp:3556] Time taken for batch_get_into: 4267002us, read store: 4266200us, with memory key count: 0, offload key count: 256 [2026-05-20 11:28:30 TP0] Prefetch operation dcaf285ba4594e74a998305584a49008 failed to retrieve page 2adfc774d05ad5f7114513c9b1e16ce8a4d6c381f05283a076bc7b64bcce2da5. [2026-05-20 11:28:30 TP1] Prefetch operation dcaf285ba4594e74a998305584a49008 failed to retrieve page 2adfc774d05ad5f7114513c9b1e16ce8a4d6c381f05283a076bc7b64bcce2da5. ``` Is it a bug? Or maybe I missed something that need to be pre-configured. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi 2026-10-29 21:16:56 +00:00

• closed this issue
• added the
  bug
  label

bsakaguchi commented 2026-10-29 21:16:57 +00:00

Author

Owner

Copy link

@LujhCoconut commented on GitHub (May 21, 2026):

I briefly looked into this. This is a size-limit inconsistency bug in Mooncake Store's SSD offloading:

• Write side: master_service.cpp:917 only enforces the kMaxSliceSize (~4MB) limit when using the CACHELIB allocator — other allocators like OffsetAllocator are allowed to put slices larger than 4MB.
• Read side: FileStorage::AllocateBatch (file_storage.cpp:617) enforces assert(sizes[i] <= kMaxSliceSize) unconditionally, regardless of allocator type.

When the put side accepts a large object (e.g., a KV cache page from a 35B model with --page-size=64 can easily exceed 4MB), the get side will crash upon reading it back from SSD.

Fix: Removing this assertion should suffice, as OffsetAllocator can handle larger allocations and BatchLoad reads contiguous data from the bucket file correctly regardless of size. Will dig deeper to confirm and submit a fix.

<!-- gh-comment-id:4504403782 --> @LujhCoconut commented on GitHub (May 21, 2026): I briefly looked into this. This is a size-limit inconsistency bug in Mooncake Store's SSD offloading: - **Write side:** `master_service.cpp:917` only enforces the `kMaxSliceSize` (~4MB) limit when using the `CACHELIB` allocator — other allocators like `OffsetAllocator` are allowed to put slices larger than 4MB. - **Read side:** `FileStorage::AllocateBatch` (`file_storage.cpp:617`) enforces `assert(sizes[i] <= kMaxSliceSize)` unconditionally, regardless of allocator type. When the put side accepts a large object (e.g., a KV cache page from a 35B model with `--page-size=64` can easily exceed 4MB), the get side will crash upon reading it back from SSD. **Fix:** Removing this assertion should suffice, as `OffsetAllocator` can handle larger allocations and `BatchLoad` reads contiguous data from the bucket file correctly regardless of size. Will dig deeper to confirm and submit a fix.

bsakaguchi commented 2026-10-29 21:16:57 +00:00

Author

Owner

Copy link

@TrafalgarZZZ commented on GitHub (May 21, 2026):

@LujhCoconut Glad to hear that! I can try again after #2165 merges.

<!-- gh-comment-id:4505411079 --> @TrafalgarZZZ commented on GitHub (May 21, 2026): @LujhCoconut Glad to hear that! I can try again after #2165 merges.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#594

No description provided.