kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2898] [RFC]: Parallelize long-running Mooncake Store unit tests #755

New issue

Closed

opened 2026-10-29 21:19:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:08 +00:00

Owner

Originally created by @Aionw on GitHub (Jul 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2898

Status

Draft / implementation in review.

This RFC replaces the oversized prototype PR #2896 with six smaller draft PRs that have an explicit review and merge topology.

Motivation

Several Mooncake Store unit-test executables contain many semantically unrelated cases and run for a long time as one scheduling unit. Splitting those cases by behavior gives CTest more useful parallel work and makes failures easier to locate, but creating one executable per source file increases compile/link overhead.

The proposal therefore separates two concerns:

Keep source files organized by semantic behavior.
Use a small CMake helper to combine compatible files into a bounded number of Unity-built test targets.

No hard-coded shard index or test-name routing table is introduced.

Invariants

Preserve all 296 source-defined GTest cases.
Changes to test-only fixture behavior are acceptable when needed for isolation.
Production behavior must not change.
Each domain split must remain independently reviewable.
The final target-balancing change must be isolated in its own commit/PR layer.

PR topology

                         +--> #2900 DummyClient + offload --------+
                         +--> #2901 SSD + snapshot SSD -----------+
RFC #2898 / main --------+--> #2902 promotion --------------------+--> #2905 Unity/target balance
                         +--> #2903 MasterService core -----------+
                         +--> #2904 snapshot MasterService -------+

The five domain PRs are all based directly on the same main commit and can be reviewed in parallel. #2905 contains those five commits plus one final CMake-only commit; review that final commit after the domain PRs, then let its diff shrink as #2900–#2904 merge.

Tracking

Order	PR	Scope	Source tests	Review dependency
1	#2900	DummyClient + offload	17	RFC only
2	#2901	SSD + snapshot SSD	26	RFC only
3	#2902	Promotion	40	RFC only
4	#2903	MasterService core	144	RFC only
5	#2904	Snapshot MasterService	69	RFC only
6	#2905	Unity build and target balancing	296 total	#2900–#2904

CMake design

add_store_test() remains the single registration primitive. The final PR adds add_store_unity_test(), which delegates to it and enables CMake UNITY_BUILD for compatible semantic source groups.

The result is:

30 independently registered targets in the five domain PRs, useful for isolated review and validation.
21 balanced targets in the final integrated layout.
Semantic directories and filenames remain intact.
No numeric shard constants or hard-coded test-case mapping in CMake.

Validation

Five independent branches:

All targeted builds passed.
30/30 CTest targets passed.
Source-defined test count: 17 + 26 + 40 + 144 + 69 = 296.
Four-way group wall times: 32.69s, 43.32s, 36.63s, 85.97s, and 89.85s.

Final integration branch:

21/21 targets built on current main.
21/21 CTest targets passed with ctest -j4 in 204.53s.
Isolated prototype measurement: 148.05s target build and 203.81s test time.
A contended cold build measured 490.52s after four concurrent worktree builds; it is retained as a host-load datapoint and is not used as a regression comparison.

Acceptance criteria

296 source-defined tests preserved.
Semantic source layout replaces the long monoliths.
Final test wall time is at most 210s at four-way concurrency.
Isolated changed-target build is at most 160s.
Combined isolated build + test time is at most 360s.
Domain PRs can be reviewed in parallel.
Target balancing is isolated for focused review.

Rollout

Review #2900–#2904 in parallel.
Merge accepted domain PRs in any order, resolving only localized CMake registration overlap if needed.
Rebase #2905 as those PRs land.
Review the final CMake-only target-balancing commit.
Mark the drafts ready only after human review of every changed line and successful CI.

Superseded prototype

#2896 demonstrated the full approach but is too large to review effectively. It is superseded by this RFC and #2900–#2905.

Originally created by @Aionw on GitHub (Jul 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2898 ## Status Draft / implementation in review. This RFC replaces the oversized prototype PR #2896 with six smaller draft PRs that have an explicit review and merge topology. ## Motivation Several Mooncake Store unit-test executables contain many semantically unrelated cases and run for a long time as one scheduling unit. Splitting those cases by behavior gives CTest more useful parallel work and makes failures easier to locate, but creating one executable per source file increases compile/link overhead. The proposal therefore separates two concerns: 1. Keep source files organized by semantic behavior. 2. Use a small CMake helper to combine compatible files into a bounded number of Unity-built test targets. No hard-coded shard index or test-name routing table is introduced. ## Invariants - Preserve all 296 source-defined GTest cases. - Changes to test-only fixture behavior are acceptable when needed for isolation. - Production behavior must not change. - Each domain split must remain independently reviewable. - The final target-balancing change must be isolated in its own commit/PR layer. ## PR topology ```text +--> #2900 DummyClient + offload --------+ +--> #2901 SSD + snapshot SSD -----------+ RFC #2898 / main --------+--> #2902 promotion --------------------+--> #2905 Unity/target balance +--> #2903 MasterService core -----------+ +--> #2904 snapshot MasterService -------+ ``` The five domain PRs are all based directly on the same `main` commit and can be reviewed in parallel. #2905 contains those five commits plus one final CMake-only commit; review that final commit after the domain PRs, then let its diff shrink as #2900–#2904 merge. ## Tracking | Order | PR | Scope | Source tests | Review dependency | | --- | --- | --- | ---: | --- | | 1 | #2900 | DummyClient + offload | 17 | RFC only | | 2 | #2901 | SSD + snapshot SSD | 26 | RFC only | | 3 | #2902 | Promotion | 40 | RFC only | | 4 | #2903 | MasterService core | 144 | RFC only | | 5 | #2904 | Snapshot MasterService | 69 | RFC only | | 6 | #2905 | Unity build and target balancing | 296 total | #2900–#2904 | ## CMake design `add_store_test()` remains the single registration primitive. The final PR adds `add_store_unity_test()`, which delegates to it and enables CMake `UNITY_BUILD` for compatible semantic source groups. The result is: - 30 independently registered targets in the five domain PRs, useful for isolated review and validation. - 21 balanced targets in the final integrated layout. - Semantic directories and filenames remain intact. - No numeric shard constants or hard-coded test-case mapping in CMake. ## Validation Five independent branches: - All targeted builds passed. - 30/30 CTest targets passed. - Source-defined test count: 17 + 26 + 40 + 144 + 69 = 296. - Four-way group wall times: 32.69s, 43.32s, 36.63s, 85.97s, and 89.85s. Final integration branch: - 21/21 targets built on current `main`. - 21/21 CTest targets passed with `ctest -j4` in 204.53s. - Isolated prototype measurement: 148.05s target build and 203.81s test time. - A contended cold build measured 490.52s after four concurrent worktree builds; it is retained as a host-load datapoint and is not used as a regression comparison. ## Acceptance criteria - [x] 296 source-defined tests preserved. - [x] Semantic source layout replaces the long monoliths. - [x] Final test wall time is at most 210s at four-way concurrency. - [x] Isolated changed-target build is at most 160s. - [x] Combined isolated build + test time is at most 360s. - [x] Domain PRs can be reviewed in parallel. - [x] Target balancing is isolated for focused review. ## Rollout 1. Review #2900–#2904 in parallel. 2. Merge accepted domain PRs in any order, resolving only localized CMake registration overlap if needed. 3. Rebase #2905 as those PRs land. 4. Review the final CMake-only target-balancing commit. 5. Mark the drafts ready only after human review of every changed line and successful CI. ## Superseded prototype #2896 demonstrated the full approach but is too large to review effectively. It is superseded by this RFC and #2900–#2905.

bsakaguchi

2026-10-29 21:19:08 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 14, 2026):

Thanks for opening this issue, @Aionw!

Field	Value
Issue	#2898
GitHub user ID	`41376987`
Reporter	@Aionw

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 14, 2026): Thanks for opening this issue, @Aionw! | Field | Value | |-------|-------| | **Issue** | #2898 | | **GitHub user ID** | `41376987` | | **Reporter** | @Aionw | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 13, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 13, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 21, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 21, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.