mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #2187] [Bug]: DRAM使用超过eviction_high_watermark_ratio=0.8后，还没驱逐，导致新请求kv cache卸载到DRAM一直失败 #601
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#601
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xuyucheng220 on GitHub (May 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2187
Bug Report
问题： DRAM使用超过eviction_high_watermark_ratio=0.8后，还没驱逐，导致新请求kv cache卸载到DRAM一直失败
master启动参数及日志：
root 795319 795247 4 11:00 ? 00:11:54 mooncake_master -enable_ha=true -v=2 -cluster_id=mooncake -etcd_endpoints=172.16.0.109:2379;172.16.0.164:2379;172.16.0.231:2379 --rpc-address=172.16.0.231 -rpc_port=52050 -metrics_port=52052 -eviction_high_watermark_ratio=0.8 -eviction_ratio=0.1 -log_dir=/data/mooncake/log -max_log_size=1800 -client_ttl=300 -stderrthreshold=4 -stop_logging_if_full_disk=true -enable_offload=true -offload_on_evict=true -root_fs_dir=/data2/mooncake_ssd_test/
[root@ecs-b1ee-etcd-0001 log]# tail -f mooncake_master.INFO |grep -i Storage
I0522 07:37:01.540355 28222 rpc_service.cpp:265] Master Admin Metrics: role=leader, state=serving, service_ready=true, master={Mem Storage: 248.23 GB / 256.00 GB (97.0%) | NVMe-oF SSD: 0 B / 0 B | SSD Storage: 248.23 GB / infinite | Keys: 18720 (soft-pinned: 0) | Clients: 32 | Requests (Success/Total per sec): PutStart=0.00/0.00, PutEnd=0.00/0.00, PutRevoke=0.00/0.00, Get=0.00/0.00, Exist=0.00/0.00, Del=0.00/0.00, DelAll=0.00/0.00, Ping=32.00/32.00, CopyStart=0.00/0.00, CopyEnd=0.00/0.00, CopyRevoke=0.00/0.00, MoveStart=0.00/0.00, MoveEnd=0.00/0.00, MoveRevoke=0.00/0.00, EvictDiskReplica=0.00/0.00 | Batch Requests (per sec, Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0.00/0.00/30.40, Item=0.00/1215.99), PutEnd:(Req=0.00/0.00/0.00, Item=0.00/0.00), PutRevoke:(Req=0.00/0.00/0.00, Item=0.00/0.00), Get:(Req=0.00/0.00/0.00, Item=0.00/0.00), ExistKey:(Req=32.10/0.00/32.10, Item=1394.68/1394.68), QueryIp:(Req=0.00/0.00/0.00, Item=0.00/0.00), Clear:(Req=0.00/0.00/0.00, Item=0.00/0.00), CreateMoveTask:(Req=0.00/0.00), CreateCopyTask:(Req=0.00/0.00), QueryTask:(Req=0.00/0.00), FetchTasks:(Req=32.00/32.00), MarkTaskToComplete:(Req=0.00/0.00) | Eviction: Success/Attempts=0/649, AllocFail=12160, keys=0, size=0 B | Mem Eviction: Success/Attempts=0/649, keys=0, size=0 B | NoF Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0}, ha={HA Metrics Summary: last_seq=0, applied_seq=0, lag=0, pending=0, mutation_queue=0, batch_commits=0, sync_commits=0, skipped=0, checksum_fail=0, etcd_fail=0, watch_disconn=0, state=0}, leader=172.16.0.231:52050, view_version=1158
更多日志：
I0522 07:33:37.582588 28238 allocator.cpp:248] allocation_failed size=14237696 segment=172.16.0.49:13455 current_size=8329052160
I0522 07:33:37.582589 28238 offset_allocator.cpp:593] OffsetAllocator allocation failed: size=14237696, fake_size=3559424, OffsetAllocatorMetrics{allocated=7.76 GB, allocs=585, capacity=8.00 GB, utilization=97.0%, free_space=2.00 MB, largest_free=2.00 MB}
I0522 07:33:37.582592 28238 allocator.cpp:248] allocation_failed size=14237696 segment=172.16.0.49:13941 current_size=8329052160
I0522 07:33:37.582592 28238 offset_allocator.cpp:593] OffsetAllocator allocation failed: size=14237696, fake_size=3559424, OffsetAllocatorMetrics{allocated=7.76 GB, allocs=585, capacity=8.00 GB, utilization=97.0%, free_space=2.00 MB, largest_free=2.00 MB}
I0522 07:33:37.582594 28238 allocator.cpp:248] allocation_failed size=14237696 segment=172.16.0.49:12788 current_size=8329052160
I0522 07:33:37.582595 28238 offset_allocator.cpp:593] OffsetAllocator allocation failed: size=14237696, fake_size=3559424, OffsetAllocatorMetrics{allocated=7.76 GB, allocs=585, capacity=8.00 GB, utilization=97.0%, free_space=2.00 MB, largest_free=2.00 MB}
I0522 07:33:37.582597 28238 allocator.cpp:248] allocation_failed size=14237696 segment=172.16.0.49:13599 current_size=8329052160
I0522 07:33:37.582602 28238 offset_allocator.cpp:593] OffsetAllocator allocation failed: size=14237696, fake_size=3559424, OffsetAllocatorMetrics{allocated=7.76 GB, allocs=585, capacity=8.00 GB, utilization=97.0%, free_space=2.00 MB, largest_free=2.00 MB}
ascend vllm侧报错：
(Worker_TP12 pid=9141) ERROR 05-22 15:37:49 [mooncake_backend.py:88] Failed to put key ['GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@95afc4a280b04d7b9042bd567f3adbcb3eaefe11f0c6ada0043ae51a20667e48', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@5c88084aa286f8a056cfb44d9ea7010bc327a782cb20b60eaea334501aa7692c', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@833341e09a399fad35c08808b0ab7704027c0acc676d7a51794f34b351892eec', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@45087cb6dcf0615765e33b570df930ef98902e3f8c049d2d58edf1f392be8812', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@9952234a8d9cb8d7acf9afdacb387418ba44f343584aa88554ed6be77c3a379a', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@d4bd0033d0c109a54602a07f8ef733a709b59d38ce7f56493267fa7f96b636b1', 'GLM-5.1-W8A8@pcp0@dcp0@head_or_tp_rank:0@pp_rank:0@group:0@cache_role:kv@cache_family:c1@a56feb1489e6d7c6da39d5c4af9fcad8f7b553272ca0d825bb72d998678e8e4c',
(Worker_TP8 pid=8912) (Worker_TP11 pid=9090) (Worker_TP6 pid=8809) W0522 15:38:22.621187 34272 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
(Worker_TP5 pid=8758) (Worker_TP10 pid=9039) W0522 15:38:22.621425 34326 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
(Worker_TP7 pid=8862) W0522 15:38:22.734968 34323 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
(Worker_TP6 pid=8809) W0522 15:38:22.791013 34307 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0522 15:38:22.791186 34322 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0522 15:38:22.791399 34325 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0522 15:38:22.842311 34276 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
(Worker_TP1 pid=8556) (Worker_TP0 pid=8538) W0522 15:38:22.867929 34314 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0522 15:38:22.904268 34267 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
W0522 15:38:22.957006 34277 client_service.cpp:1933] BatchPut failed for 456 keys due to insufficient space. Consider lowering eviction_high_watermark_ratio or mounting more segments.
Before submitting...
@00fish0 commented on GitHub (May 22, 2026):
master 的日志里 Eviction 的计数显示：
正常情况下，超水位后驱逐会周期性地把一批"租约已过期"的 key 驱逐掉；Attempts=649 说明超过 0.8 水位后驱逐被触发了很多次，但 keys=0 / size=0 说明每次都筛不出可驱逐的对象。属于异常。
不太确定，但一种可能是：Mooncake 驱逐时要求 key 的租约（lease）已过期才会被选中，而 ExistKey/BatchExistKey 会给命中的 key 续一个完整租约（默认 5s）。日志里 ExistKey Item≈1394/s 频率很高，所以有可能是 vllm 侧在用存在性检查做前缀缓存命中判断、反复探测同一批常驻 key，把它们的租约一直续着，导致驱逐永远筛不出候选，总是失败。
@xuyucheng220 commented on GitHub (May 23, 2026):
今天修改配置offload_on_evict=false，从master日志看可以了。因为之前offload_on_evict=true时实际mooncake也是在卸载到dram时就卸载到了ssd（offload_on_evict=true并没有生效）