kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3556] [Track]: Mooncake Build, CI, and Release Roadmap #911

New issue

Open

opened 2026-10-29 21:20:49 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:49 +00:00

Owner

Originally created by @Aionw on GitHub (Aug 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3556

This RFC tracks Mooncake's build, CI, and release evolution. Contributions and feedback are welcome.

Focus

The work is divided into four directions:

C++ Build — one explicit and modular native build graph.
Python Build — one Python source tree and one standard package backend.
CI — clear PR, nightly, and release CI responsibilities, implemented with small reusable workflows.
Release — one reviewable process from release branch to published artifacts.

The overall dependency is:

C++ Build + Python Build  -->  Reusable CI  -->  Release

Build interfaces come first. CI should consume them instead of implementing another build path, and Release should publish the artifacts already validated by CI.

1. C++ Build

Goal: use CMake as the single native build graph for standalone components, language bindings, tests, and Python wheels.

Direction

One build graph. CMake owns feature flags, targets, tests, and install rules for standalone components, language bindings, and Python wheels.
Clear component boundaries. Transfer Engine remains independently buildable; Store exposes one native SDK boundary; and PG/EP native cores do not depend on the PyTorch C++ ABI.
One dependency policy. Prefer maintained, high-performance libraries with few transitive dependencies. Acquire, pin, and link them target-by-target through CMake when practical; keep only toolchains, drivers, and vendor runtimes external with explicit supported versions.

Tracking

2. Python Build

Goal: make one tracked Python package and one PEP 517 backend the source of every editable install and wheel variant.

Direction

One source tree. Code shipped under import mooncake has one authoritative location; builds neither copy tracked sources nor modify package metadata.
One build interface. PEP 517 is the public entry point for editable installs and every wheel profile, and its backend consumes CMake install outputs.
One install contract. Package dependencies and public imports are explicit, and every wheel is validated from a clean environment outside the source checkout.

Tracking

#3534
#3425

3. CI

Goal: provide three distinct gates over one shared build and artifact contract: PR CI decides whether a change may merge, Nightly CI reports the health of main, and Release CI qualifies one exact candidate for publication.

Current baseline

ci.yml is the main PR workflow for main and release/** and exposes one aggregate CI Gate. It already reuses the standard wheel builder, but still mixes native builds, wheel installation, TENT, EFA, ROCm, and SGLang integration in one large caller. Privileged hardware E2E is a separate label-authorized workflow.
nightly.yml owns the wider wheel matrix, MUSA and container builds, Ascend hardware, full CTest and language-binding tests, coverage, TestPyPI snapshots, and failure notification. These responsibilities are currently coupled into one aggregate gate.
Release automation is split by package variant. Standard and EFA wheels reuse builders, while ROCm, Ascend, and MUSA keep custom build-and-publish paths. A final tag still starts new builds, and container publication is a separate manual workflow.

Target ownership

Tier	Decision	Owns	Does not own
PR CI	May this change merge?	Fast deterministic checks and representative affected builds/tests	The full platform matrix or publication
Nightly CI	Is `main` healthy across supported configurations?	Full matrices, long tests, hardware tests, snapshots, and diagnosis	Merge approval or production credentials
Release CI	Are these exact candidate artifacts publishable?	Full core artifacts, validation evidence, and candidate manifest	Rebuilding after approval

PR CI

Required merge signal. Every PR update produces one stable gate from deterministic checks plus the affected portable build and test profiles.
Artifact-first validation. Source changes build the reference profile; packaging and shared-build changes build affected core wheels, then validate their installation, public interfaces, and portable integration paths.
Trusted hardware path. Privileged E2E runs only after a maintainer authorizes an exact PR SHA, and consumes artifacts produced for that SHA.

Nightly CI

Broad health coverage. Run the full supported build matrix, portable and language-binding tests, long integrations, coverage, container builds, and available hardware suites through the same profiles used elsewhere.
Independent diagnosis. Build, test, hardware, coverage, and snapshot results remain independent, retain focused artifacts, and classify failures without one broken job hiding the rest.
Non-production snapshots. TestPyPI publication follows artifact validation, uses no production credentials, and reports separately from the health of main.

Release CI

Build once. Produce the complete core matrix from one immutable candidate commit using the shared build profiles.
Qualify exact artifacts. Validate package identity, compatibility, public interfaces, and native linkage, then emit one immutable manifest containing the source, required profiles, artifact names, and digests.
Promote without rebuilding. One aggregate gate qualifies the manifest; the release publisher verifies it, while extended variants may backfill independently from the same tag.

The proposed core set is standard CUDA, CUDA 13, and non-CUDA wheels for x86_64 and arm64. EFA, ROCm, Ascend, and MUSA are extended variants until a later RFC promotes one into the core release gate.

CI implementation direction

Profiles define builds. Each profile owns its runner, container, dependencies, CMake flags, package identity, and supported matrix once.
Workflows only orchestrate. Top-level workflows select profiles and gates; reusable jobs build, validate, run hardware tests, and promote artifacts. Vendor paths keep only vendor-specific setup.
Trust follows artifacts. Artifacts carry source SHA, profile, version, and digest; builders have no publication credentials; and trusted workflows never execute unapproved PR code.

Tracking

4. Release

Goal: make a release a reviewable promotion of one validated commit and its existing artifacts, not a fresh build triggered at the end.

Proposed flow

Create release/** from the selected main commit; release fixes continue through normal PR CI.
Set the final version and run Release CI once. It builds, validates, and records the candidate artifacts in an immutable manifest.
Tag the exact candidate commit after validation. The publisher verifies the tag and manifest, then promotes the existing artifacts without rebuilding.
Extended variants may retry or backfill from the same tag. Versioned container images consume published wheels and pass entrypoint smoke tests before mutable tags are promoted.
A failed published release uses a new version; published files and tags are never overwritten.

Tracking

#3130

Decisions requested

Should standard CUDA, CUDA 13, and non-CUDA x86_64/arm64 wheels form the core release gate?
Which maintainer role may authorize privileged PR hardware CI, and what cleanup guarantee must each self-hosted runner provide?

Originally created by @Aionw on GitHub (Aug 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3556 This RFC tracks Mooncake's build, CI, and release evolution. Contributions and feedback are welcome. ## Focus The work is divided into four directions: 1. **C++ Build** — one explicit and modular native build graph. 2. **Python Build** — one Python source tree and one standard package backend. 3. **CI** — clear PR, nightly, and release CI responsibilities, implemented with small reusable workflows. 4. **Release** — one reviewable process from release branch to published artifacts. The overall dependency is: ```text C++ Build + Python Build --> Reusable CI --> Release ``` Build interfaces come first. CI should consume them instead of implementing another build path, and Release should publish the artifacts already validated by CI. ## 1. C++ Build **Goal:** use CMake as the single native build graph for standalone components, language bindings, tests, and Python wheels. ### Direction - **One build graph.** CMake owns feature flags, targets, tests, and install rules for standalone components, language bindings, and Python wheels. - **Clear component boundaries.** Transfer Engine remains independently buildable; Store exposes one native SDK boundary; and PG/EP native cores do not depend on the PyTorch C++ ABI. - **One dependency policy.** Prefer maintained, high-performance libraries with few transitive dependencies. Acquire, pin, and link them target-by-target through CMake when practical; keep only toolchains, drivers, and vendor runtimes external with explicit supported versions. ### Tracking - [ ] #3800 - [ ] #3181 - [ ] #3553 - [ ] #3550 - [x] #3537 - [x] #3276 - [x] #3206 - [x] #2883 ## 2. Python Build **Goal:** make one tracked Python package and one PEP 517 backend the source of every editable install and wheel variant. ### Direction - **One source tree.** Code shipped under `import mooncake` has one authoritative location; builds neither copy tracked sources nor modify package metadata. - **One build interface.** PEP 517 is the public entry point for editable installs and every wheel profile, and its backend consumes CMake install outputs. - **One install contract.** Package dependencies and public imports are explicit, and every wheel is validated from a clean environment outside the source checkout. ### Tracking - [ ] #3534 - [ ] #3425 ## 3. CI **Goal:** provide three distinct gates over one shared build and artifact contract: PR CI decides whether a change may merge, Nightly CI reports the health of `main`, and Release CI qualifies one exact candidate for publication. ### Current baseline - `ci.yml` is the main PR workflow for `main` and `release/**` and exposes one aggregate `CI Gate`. It already reuses the standard wheel builder, but still mixes native builds, wheel installation, TENT, EFA, ROCm, and SGLang integration in one large caller. Privileged hardware E2E is a separate label-authorized workflow. - `nightly.yml` owns the wider wheel matrix, MUSA and container builds, Ascend hardware, full CTest and language-binding tests, coverage, TestPyPI snapshots, and failure notification. These responsibilities are currently coupled into one aggregate gate. - Release automation is split by package variant. Standard and EFA wheels reuse builders, while ROCm, Ascend, and MUSA keep custom build-and-publish paths. A final tag still starts new builds, and container publication is a separate manual workflow. ### Target ownership | Tier | Decision | Owns | Does not own | | --- | --- | --- | --- | | PR CI | May this change merge? | Fast deterministic checks and representative affected builds/tests | The full platform matrix or publication | | Nightly CI | Is `main` healthy across supported configurations? | Full matrices, long tests, hardware tests, snapshots, and diagnosis | Merge approval or production credentials | | Release CI | Are these exact candidate artifacts publishable? | Full core artifacts, validation evidence, and candidate manifest | Rebuilding after approval | ### PR CI - **Required merge signal.** Every PR update produces one stable gate from deterministic checks plus the affected portable build and test profiles. - **Artifact-first validation.** Source changes build the reference profile; packaging and shared-build changes build affected core wheels, then validate their installation, public interfaces, and portable integration paths. - **Trusted hardware path.** Privileged E2E runs only after a maintainer authorizes an exact PR SHA, and consumes artifacts produced for that SHA. ### Nightly CI - **Broad health coverage.** Run the full supported build matrix, portable and language-binding tests, long integrations, coverage, container builds, and available hardware suites through the same profiles used elsewhere. - **Independent diagnosis.** Build, test, hardware, coverage, and snapshot results remain independent, retain focused artifacts, and classify failures without one broken job hiding the rest. - **Non-production snapshots.** TestPyPI publication follows artifact validation, uses no production credentials, and reports separately from the health of `main`. ### Release CI - **Build once.** Produce the complete core matrix from one immutable candidate commit using the shared build profiles. - **Qualify exact artifacts.** Validate package identity, compatibility, public interfaces, and native linkage, then emit one immutable manifest containing the source, required profiles, artifact names, and digests. - **Promote without rebuilding.** One aggregate gate qualifies the manifest; the release publisher verifies it, while extended variants may backfill independently from the same tag. The proposed core set is standard CUDA, CUDA 13, and non-CUDA wheels for x86_64 and arm64. EFA, ROCm, Ascend, and MUSA are extended variants until a later RFC promotes one into the core release gate. ### CI implementation direction - **Profiles define builds.** Each profile owns its runner, container, dependencies, CMake flags, package identity, and supported matrix once. - **Workflows only orchestrate.** Top-level workflows select profiles and gates; reusable jobs build, validate, run hardware tests, and promote artifacts. Vendor paths keep only vendor-specific setup. - **Trust follows artifacts.** Artifacts carry source SHA, profile, version, and digest; builders have no publication credentials; and trusted workflows never execute unapproved PR code. ### Tracking - [x] #3445 - [x] #3338 - [x] #2898 - [x] #3485 - [x] #3148 - [x] #3137 - [x] #3283 - [x] #3049 - [x] #2950 - [x] #3184 ## 4. Release **Goal:** make a release a reviewable promotion of one validated commit and its existing artifacts, not a fresh build triggered at the end. ### Proposed flow 1. Create `release/**` from the selected `main` commit; release fixes continue through normal PR CI. 2. Set the final version and run Release CI once. It builds, validates, and records the candidate artifacts in an immutable manifest. 3. Tag the exact candidate commit after validation. The publisher verifies the tag and manifest, then promotes the existing artifacts without rebuilding. 4. Extended variants may retry or backfill from the same tag. Versioned container images consume published wheels and pass entrypoint smoke tests before mutable tags are promoted. 5. A failed published release uses a new version; published files and tags are never overwritten. ### Tracking - [ ] #3130 ## Decisions requested 1. Should standard CUDA, CUDA 13, and non-CUDA x86_64/arm64 wheels form the core release gate? 2. Which maintainer role may authorize privileged PR hardware CI, and what cleanup guarantee must each self-hosted runner provide?

bsakaguchi added the

RFC

Roadmap

labels

2026-10-29 21:20:49 +00:00

bsakaguchi commented

2026-10-29 21:20:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 20, 2026):

Thanks for opening this issue, @Aionw!

Field	Value
Issue	#3556
GitHub user ID	`41376987`
Reporter	@Aionw

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 20, 2026): Thanks for opening this issue, @Aionw! | Field | Value | |-------|-------| | **Issue** | #3556 | | **GitHub user ID** | `41376987` | | **Reporter** | @Aionw | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:12 +00:00

[PR #911] [MERGED] Mooncake-Backend Chunked Transfer #1898