[GH-ISSUE #5035] [RFC]: Distributed-filesystem offload root with cluster-wide quota and master-driven eviction #1273

New issue

Open

opened 2026-10-29 21:24:42 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:42 +00:00

Owner

Originally created by @kdtmac on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5035

Context

Of the KV caches we evaluated, Mooncake Store is the closest to giving us a single control point: the master owns admission, tenant quota and eviction, and the offload tier extends capacity past DRAM. That centralized shape is exactly what we want — which is why the remaining gap is specific rather than general.

What we rely on today:

master-side eviction watermarks and KV lease TTL;
tenant quota landing in the master (#2112 per-tenant max_bytes; #2153 argues Mooncake should own both resource allocation and object identity, deriving enforceable quota from actual allocatable capacity);
SSD offload via BucketStorageBackend — MOONCAKE_OFFLOAD_FILE_STORAGE_PATH, bounded by MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE_LIST, with one FIFO/LRU order over buckets_, and (per #4230) a cap that deliberately tracks physical usage because Kubernetes emptyDir deployments depend on it.

The gap: our offload root is a shared filesystem, not a local SSD

We want the offload tier on a distributed filesystem (3FS / JuiceFS / NFS / CephFS) so that (a) it survives pod rescheduling onto another node, and (b) capacity is a cluster resource instead of N independent per-node caps.

As #4230 summarizes the current design — one root, one total_size_ counter, one FIFO/LRU order over buckets_ — three things break in that configuration:

Quota becomes per-node. N clients each mounting the same DFS enforce N independent caps over one shared namespace: the aggregate is N × cap, and the multi-root work in #4230 multiplies it further. There is no cluster-level budget and no namespace/tenant attribution for the offload tier.
Eviction is per-client over a local view. Two clients each evict from their own buckets_ ordering. A bucket written by client A can be evicted by client B while A still considers it resident; neither accounts for the other's bytes. On shared storage "evict the oldest bucket I know about" is not a global policy — it's two independent policies over one dataset.
Physical-usage accounting doesn't hold. #3426 already showed fs::space() ignoring cgroup/emptyDir quotas. On a DFS, fs::space() reports the whole filesystem — shared with unrelated workloads, and usually wrong about the share we own (reservations, replication, project quotas). So the one number the offload tier uses to decide "full" isn't meaningful.

There is a real risk here beyond waste: two clients racing on the same bucket, one deleting while the other reads, is the same class of bug as #4686 (L4 offload matched by key alone) and #4725 (stale replica attached to a republished key) — those are local-SSD races; a shared root makes them cluster-wide and much easier to hit.

Proposal

Make offload-tier capacity policy master-owned, mirroring what already exists for DRAM:

Register the offload root as a cluster-scoped resource instead of a per-client local path: clients report usage and liveness, the master holds the aggregate budget (per namespace/tenant), so quota is enforced once and attributed once.
Centralize eviction for that tier: the master selects candidates (global LRU/TTL, reusing the existing watermark and lease-TTL semantics), clients execute deletes idempotently. Keep the current per-client FIFO/LRU as the local-SSD default so emptyDir deployments are unaffected.
Make capacity an explicit number (a master-side offload_capacity_bytes or a tenant quota row) rather than a statfs guess, so a DFS with replication or project quotas doesn't misreport available space.
Expose it: per-namespace used/limit for the offload tier in the master's metrics, next to the DRAM numbers, so "offload is full" is observable before it becomes an error.

Questions

Is a shared / distributed-filesystem offload root in scope, or is the offload tier intentionally node-local? If it's intentionally local, what is the recommended way to get cluster-shared, quota-bounded capacity — put 3FS behind the DRAM tier through a different API, or run a store per node?
If DFS offload is in scope: should quota and eviction for it live in the master (consistent with #2153's "Mooncake should own both layers"), or be delegated to the filesystem?
Does #4230's multi-root model have room for "one shared root, many clients", or is it strictly "many local roots, one client"?
If it is planned, is there a milestone or tracking issue we can follow?

We'd be glad to contribute a cluster-scoped offload-capacity registration plus master-driven eviction, with node-local behaviour preserved as the default, if maintainers think the shape is reviewable.

References

#2112 — per-tenant storage quota in master
#2153 — dynamic tenant quota and scoped-key isolation
#4230 — multi-disk SSD offload for the bucket storage backend (current single-root / single-counter / single-FIFO design)
#3426 — disk eviction relies on fs::space(), ignores cgroup/emptyDir quotas
#4686, #4725 — key-only matching / stale-replica races in the offload path

Originally created by @kdtmac on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5035 ### Context Of the KV caches we evaluated, Mooncake Store is the closest to giving us a single control point: the **master** owns admission, tenant quota and eviction, and the offload tier extends capacity past DRAM. That centralized shape is exactly what we want — which is why the remaining gap is specific rather than general. What we rely on today: - master-side eviction watermarks and KV lease TTL; - tenant quota landing in the master (#2112 per-tenant `max_bytes`; #2153 argues Mooncake should own both resource allocation and object identity, deriving enforceable quota from actual allocatable capacity); - SSD offload via `BucketStorageBackend` — `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`, bounded by `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE_LIST`, with one FIFO/LRU order over `buckets_`, and (per #4230) a cap that deliberately tracks *physical* usage because Kubernetes `emptyDir` deployments depend on it. ### The gap: our offload root is a shared filesystem, not a local SSD We want the offload tier on a distributed filesystem (3FS / JuiceFS / NFS / CephFS) so that (a) it survives pod rescheduling onto another node, and (b) capacity is a cluster resource instead of N independent per-node caps. As #4230 summarizes the current design — one root, one `total_size_` counter, one FIFO/LRU order over `buckets_` — three things break in that configuration: 1. **Quota becomes per-node.** N clients each mounting the same DFS enforce N independent caps over one shared namespace: the aggregate is N × cap, and the multi-root work in #4230 multiplies it further. There is no cluster-level budget and no namespace/tenant attribution for the offload tier. 2. **Eviction is per-client over a local view.** Two clients each evict from their own `buckets_` ordering. A bucket written by client A can be evicted by client B while A still considers it resident; neither accounts for the other's bytes. On shared storage "evict the oldest bucket I know about" is not a global policy — it's two independent policies over one dataset. 3. **Physical-usage accounting doesn't hold.** #3426 already showed `fs::space()` ignoring cgroup/`emptyDir` quotas. On a DFS, `fs::space()` reports the *whole filesystem* — shared with unrelated workloads, and usually wrong about the share we own (reservations, replication, project quotas). So the one number the offload tier uses to decide "full" isn't meaningful. There is a real risk here beyond waste: two clients racing on the same bucket, one deleting while the other reads, is the same class of bug as #4686 (L4 offload matched by key alone) and #4725 (stale replica attached to a republished key) — those are local-SSD races; a shared root makes them cluster-wide and much easier to hit. ### Proposal Make offload-tier capacity policy **master-owned**, mirroring what already exists for DRAM: 1. **Register the offload root as a cluster-scoped resource** instead of a per-client local path: clients report usage and liveness, the master holds the aggregate budget (per namespace/tenant), so quota is enforced once and attributed once. 2. **Centralize eviction for that tier**: the master selects candidates (global LRU/TTL, reusing the existing watermark and lease-TTL semantics), clients execute deletes idempotently. Keep the current per-client FIFO/LRU as the local-SSD default so `emptyDir` deployments are unaffected. 3. **Make capacity an explicit number** (a master-side `offload_capacity_bytes` or a tenant quota row) rather than a `statfs` guess, so a DFS with replication or project quotas doesn't misreport available space. 4. **Expose it**: per-namespace used/limit for the offload tier in the master's metrics, next to the DRAM numbers, so "offload is full" is observable before it becomes an error. ### Questions 1. Is a shared / distributed-filesystem offload root in scope, or is the offload tier intentionally node-local? If it's intentionally local, what is the recommended way to get cluster-shared, quota-bounded capacity — put 3FS behind the DRAM tier through a different API, or run a store per node? 2. If DFS offload is in scope: should quota and eviction for it live in the master (consistent with #2153's "Mooncake should own both layers"), or be delegated to the filesystem? 3. Does #4230's multi-root model have room for "one shared root, many clients", or is it strictly "many local roots, one client"? 4. If it is planned, is there a milestone or tracking issue we can follow? We'd be glad to contribute a cluster-scoped offload-capacity registration plus master-driven eviction, with node-local behaviour preserved as the default, if maintainers think the shape is reviewable. ### References - #2112 — per-tenant storage quota in master - #2153 — dynamic tenant quota and scoped-key isolation - #4230 — multi-disk SSD offload for the bucket storage backend (current single-root / single-counter / single-FIFO design) - #3426 — disk eviction relies on `fs::space()`, ignores cgroup/`emptyDir` quotas - #4686, #4725 — key-only matching / stale-replica races in the offload path

bsakaguchi commented

2026-10-29 21:24:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 28, 2026):

Thanks for opening this issue, @kdtmac!

Field	Value
Issue	#5035
GitHub user ID	`22338707`
Reporter	@kdtmac

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 28, 2026): Thanks for opening this issue, @kdtmac! | Field | Value | |-------|-------| | **Issue** | #5035 | | **GitHub user ID** | `22338707` | | **Reporter** | @kdtmac | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:57 +00:00

[PR #1273] [MERGED] [CI] Add retry mechanism to handle GitHub API rate limit in test-sglang-integration job #2123