kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3812] [RFC]: dfs replica supports dynamic capacity scaling #975

New issue

Open

opened 2026-10-29 21:21:30 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:21:30 +00:00

Owner

Originally created by @sun-baoshan-sunshine on GitHub (Sep 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3812

Changes proposed

背景

很高兴看到mooncake 目前完善了dfs写入支持的相关工作#2683 ，在测试使用过程中，我发现目前机制下，还不能支持dfs后端的动态扩容。更详细的描述是，dfs 后端的总容量大小由单个文件大小以及总的文件数量决定，这些作为配置在master启动时设定，而一旦服务启动后，没有任何方式能够再进行更改。我认为这是一种目前机制的缺陷，在推理场景下，初期的评估不足或者dfs 后端容量暂时的紧缺，可能导致出现后期 dfs 容量设置的太小的问题，需要调整。但是在当前机制下，只能通过master 重启的方式重新约定dfs 后端总容量，而master 一旦重启，会导致dfs 中已有的 kvcache 数据元信息丢失，进而变为孤儿数据在dfs中，这本身对dfs 是无害的，但是在推理角度来看，历史缓存的kvcache全部都需要重新计算。
在这个issue 中，我想就动态扩容的问题提出一些探讨，增加dfs 实现动态扩容的能力。

初步设计思路

关于动态扩容的实现方式，我觉得比较合理的是master 开放api 服务，由外部手动调用发起。这个接口将集成在master_admin_service.h中，等待外部发起调用。扩容操作应当由操作人评估当前环境以及后端dfs 资源后决定是否发起。
扩容操作应当只增加文件数量，而不是增加文件大小。基于现有的实现机制，增加独立的文件个数是比较明智的选择，增加单个文件大小实现起来会非常复杂。
master 启动时需要扫描一次dfs 中存留的文件个数，这是因为假设master 发起过扩容，期间重启后配置恢复了原样，会导致先前的扩容失效。因此，扫描dfs 文件个数，取max(文件数，配置参数)，能够保证先前的扩容有效。这样一来，dfs 文件的命名以及过滤，需要严谨的考虑。

实现步骤

目前初步评估，这个pr 的代码改动不会很大。具体的，可以分为两个步骤完成：

实现dfs 后端 shard 数量的可增长
增加 master service 动态扩容接口，同时master 和 client 实现动态扩容逻辑

考虑项

扩容和驱逐的关系，扩容后需要同步更新驱逐中的逻辑
client 如何处理扩容产生的新文件
......

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @sun-baoshan-sunshine on GitHub (Sep 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3812 ### Changes proposed ### 背景很高兴看到mooncake 目前完善了dfs写入支持的相关工作[#2683](https://github.com/kvcache-ai/Mooncake/pull/2683) ，在测试使用过程中，我发现目前机制下，还不能支持dfs后端的动态扩容。更详细的描述是，dfs 后端的总容量大小由单个文件大小以及总的文件数量决定，这些作为配置在master启动时设定，而一旦服务启动后，没有任何方式能够再进行更改。我认为这是一种目前机制的缺陷，在推理场景下，初期的评估不足或者dfs 后端容量暂时的紧缺，可能导致出现后期 dfs 容量设置的太小的问题，需要调整。但是在当前机制下，只能通过master 重启的方式重新约定dfs 后端总容量，而master 一旦重启，会导致dfs 中已有的 kvcache 数据元信息丢失，进而变为孤儿数据在dfs中，这本身对dfs 是无害的，但是在推理角度来看，历史缓存的kvcache全部都需要重新计算。在这个issue 中，我想就动态扩容的问题提出一些探讨，增加dfs 实现动态扩容的能力。 ### 初步设计思路 - 关于动态扩容的实现方式，我觉得比较合理的是master 开放api 服务，由外部手动调用发起。这个接口将集成在master_admin_service.h中，等待外部发起调用。扩容操作应当由操作人评估当前环境以及后端dfs 资源后决定是否发起。 - 扩容操作应当只增加文件数量，而不是增加文件大小。基于现有的实现机制，增加独立的文件个数是比较明智的选择，增加单个文件大小实现起来会非常复杂。 - master 启动时需要扫描一次dfs 中存留的文件个数，这是因为假设master 发起过扩容，期间重启后配置恢复了原样，会导致先前的扩容失效。因此，扫描dfs 文件个数，取max(文件数，配置参数)，能够保证先前的扩容有效。这样一来，dfs 文件的命名以及过滤，需要严谨的考虑。 ### 实现步骤目前初步评估，这个pr 的代码改动不会很大。具体的，可以分为两个步骤完成： - [ ] 实现dfs 后端 shard 数量的可增长 - [ ] 增加 master service 动态扩容接口，同时master 和 client 实现动态扩容逻辑 ### 考虑项 - 扩容和驱逐的关系，扩容后需要同步更新驱逐中的逻辑 - client 如何处理扩容产生的新文件 ...... ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:21:30 +00:00

bsakaguchi commented

2026-10-29 21:21:31 +00:00

Author

Owner

@sun-baoshan-sunshine commented on GitHub (Sep 1, 2026):

@fcczzz Please take a look at whether this issue is feasible，thanks

@sun-baoshan-sunshine commented on GitHub (Sep 1, 2026): @fcczzz Please take a look at whether this issue is feasible，thanks

bsakaguchi commented

2026-10-29 21:21:31 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 1, 2026):

Thanks for opening this issue, @sun-baoshan-sunshine!

Field	Value
Issue	#3812
GitHub user ID	`78401405`
Reporter	@sun-baoshan-sunshine

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 1, 2026): Thanks for opening this issue, @sun-baoshan-sunshine! | Field | Value | |-------|-------| | **Issue** | #3812 | | **GitHub user ID** | `78401405` | | **Reporter** | @sun-baoshan-sunshine | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:21:31 +00:00

Author

Owner

@fcczzz commented on GitHub (Sep 1, 2026):

Thanks for proposing this. The proposal looks useful, but a few points should be clarified before implementation:

Existing clients open a fixed number of shard files and will reject descriptors for newly added shards. The RFC should define client-side lazy opening or a versioned update/acknowledgement mechanism.
Shard filenames must not depend on the total shard count. Currently, expanding from 100 to 101 shards changes names such as dfs_shard_00.data to dfs_shard_000.data.
Expansion must be atomic with concurrent allocation, free, and eviction. New shards should only become allocatable after their files and allocator states are fully ready.
The current hash(key) % shard_count placement tries only one shard. Adding empty shards may therefore not help keys that still map to full shards. A fallback placement strategy and updated eviction semantics are needed.

Overall, the direction is feasible, but these lifecycle and compatibility details should be included in the RFC.

@fcczzz commented on GitHub (Sep 1, 2026): Thanks for proposing this. The proposal looks useful, but a few points should be clarified before implementation: 1. Existing clients open a fixed number of shard files and will reject descriptors for newly added shards. The RFC should define client-side lazy opening or a versioned update/acknowledgement mechanism. 2. Shard filenames must not depend on the total shard count. Currently, expanding from 100 to 101 shards changes names such as `dfs_shard_00.data` to `dfs_shard_000.data`. 3. Expansion must be atomic with concurrent allocation, free, and eviction. New shards should only become allocatable after their files and allocator states are fully ready. 4. The current `hash(key) % shard_count` placement tries only one shard. Adding empty shards may therefore not help keys that still map to full shards. A fallback placement strategy and updated eviction semantics are needed. Overall, the direction is feasible, but these lifecycle and compatibility details should be included in the RFC.

bsakaguchi referenced this issue

2026-10-29 21:23:17 +00:00

[GH-ISSUE #4491] [Bug]: Timed-out put is revoked while its queued slices are still pending; they later overwrite the next object at that address #1142

bsakaguchi referenced this issue

2026-10-29 21:31:32 +00:00

[PR #993] [MERGED] [Store]: Cleanup processing objects if transferring timedout (#975) #1938