kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4751] [RFC]: Restore reliable nightly CI and TestPyPI publishing #1210

New issue

Open

opened 2026-10-29 21:24:00 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:24:00 +00:00

Owner

Originally created by @stmatengss on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4751

Changes proposed

Motivation and scope

Nightly should provide a timely, interpretable health signal for main, and validated nightly wheels should be publishable without repeatedly exhausting TestPyPI storage. Recent runs have instead mixed source regressions, unavailable hardware, hosted-runner failures, and publishing failures into an incomplete aggregate result.

This RFC proposes a staged recovery of the existing nightly workflow. It is a focused follow-up to the broader build/CI/release roadmap in #3556. The first implementation is #4750; storage administration and longer-term publication policy require separate decisions.

Problems observed

Area	Evidence	Conclusion and remaining uncertainty
ROCm hardware scheduling	Nightly requires a runner with `self-hosted`, `Linux`, `X64`, `amd`, `rocm`, `gfx950`, and `mooncake-pd`. During investigation, the repository runner inventory contained only an ARM64 Ascend runner. The hardware job in this nightly run remained unassigned; the run was later cancelled by a newer request in the `nightly-main` concurrency group.	Unconditional scheduling can prevent nightly from completing when no matching MI350X runner is available. A skipped hardware test must be reported as missing coverage.
Native test compilation	`scenario.cpp` still called `BatchProbeKey` with two arguments after the API required a `GrantLeasePolicy` argument.	Already fixed on `main` by #4729. The recovery PR does not duplicate that fix. Subsequent publication skips caused by failed prerequisites are not upload failures.
Docker runner loss	Several Docker jobs reported that the hosted runner lost communication with GitHub. Complete logs were unavailable for those failed jobs.	Resource pressure is a plausible cause, but the available evidence does not establish OOM or disk exhaustion. Bound build concurrency and collect diagnostics before attributing the failure.
TestPyPI publication	An upload failed with HTTP 400 after transferring the first approximately 95 MB wheel. All 26 files had passed `twine check`. The server response body was absent from the log.	Public storage totals strongly suggest a capacity problem. Some other wheels also exceed the default per-file limit. The actual project limits and the original HTTP 400 reason still require confirmation.

TestPyPI storage evidence

Public JSON metadata was inspected during this investigation on October 19, 2026 (Asia/Shanghai). The sums below count the sizes of all listed files; private project settings were not accessible.

Project	Listed files	Listed file bytes	Approximate size
`mooncake-transfer-engine`	131	10,667,841,775	9.94 GiB
`mooncake-transfer-engine-cuda13`	120	10,206,105,359	9.51 GiB
`mooncake-transfer-engine-non-cuda`	—	4,499,499,434	4.19 GiB

Warehouse defaults are 100 MiB per file and 10 GiB per project; project-specific limits may differ. Under the default project limit, the main package has only about 66 MiB available, less than the approximately 95 MB upload that failed. The collected wheel set also contains files around 119–121 MB, above the default per-file limit. These are two distinct limits and both need checking. See storage-limit documentation and Warehouse defaults.

The latest visible main-package development release had only three files, so a successful upload of one file is not sufficient evidence that a nightly release is complete.

Proposed changes

1. Restore bounded, explicit nightly behavior

Implemented in #4750:

Select ROCm hardware coverage once per run. Enable it with the NIGHTLY_ROCM_INTEGRATION=true repository variable or the run_rocm_integration manual input. Otherwise emit a warning and job summary explaining the missing coverage. ROCm wheel builds and smoke tests remain required.
Accept only the hardware job's intentional skipped result when it was disabled. When enabled, failure, cancellation, or skipping must fail the nightly gate. Preserve strict checks for all other required jobs and the existing explicit skip_publish mode.
Run Docker builds through the configured Buildx builder, limit BuildKit concurrency to one operation, and default BUILD_JOBS/MAX_JOBS to two. Build the complete Dockerfile with a cache-only output, avoiding a second image-store export. Add memory/disk diagnostics and a 180-minute job timeout.
Check that TestPyPI credentials are present before attempting uploads. Use noninteractive Twine with verbose responses so future failures include the actionable server error.
Add regression tests that execute the workflow shell blocks and verify skip decisions, required-job failures, credential handling, and Docker exit-status propagation.

The tradeoff is explicit: a run may complete without MI350X hardware coverage while the runner is unavailable. It must not advertise that coverage as passing. Lower Docker concurrency may increase build time; the timeout and hosted measurements will show whether the default is practical.

2. Recover TestPyPI publishing with project-owner actions

Sign in to the TestPyPI account that owns these projects and open project management. Check each project's storage usage and configured limits.
Review old development releases and delete an agreed subset to reclaim space. A proposed starting policy is to retain the latest seven development releases for each package, while preserving stable and release-candidate versions. Deletion is irreversible; yanking does not free storage. Keep any versions needed by existing tests before deleting them.
Check the per-file limit for both CUDA variants. If it is still 100 MiB, request an increase, for example to 200 MB, through the support process linked in the storage-limit documentation. Clearly identify the TestPyPI index, project URLs, and measured wheel sizes. Confirm enough project capacity for the complete next wheel set as well.
Check TESTPYPI_API_TOKEN in the GitHub nightly environment. It must be a TestPyPI token with upload access to every project in the wheel set. The first publication of the ROCm project also requires permission to create it if it does not yet exist. The observed HTTP 400 alone is not a reason to rotate a valid token.
After the repository fix is merged and storage/limits are ready, start a new nightly run from the updated branch with publishing enabled. Rerunning an old failure uses its old commit. Confirm that every expected package, architecture, and Python-version wheel is present, then verify the existing install/import gate.

No package deletion, quota change, or secret update is included in #4750 or has been performed by this investigation.

3. Prevent recurrence

These are follow-up proposals, not behavior already implemented by #4750:

Assign an owner and documented availability expectations to each hardware profile. Decide when MI350X coverage should become mandatory again and enable the repository variable once the matching runner is ready.
Adopt a retention policy with named ownership for TestPyPI development releases. A larger quota alone postpones the same failure. If daily artifacts need long retention, evaluate a package host with a managed lifecycle instead of indefinite TestPyPI accumulation.
Follow #3556 by reporting source health, hardware coverage, and snapshot publication separately. Keep each required profile's result strict, and make an incomplete publication visible without obscuring build/test diagnostics. The current recovery PR preserves the existing aggregate policy.

Validation and acceptance criteria

Current evidence for #4750 at b1da7c6cfd669fd17055d764f7824562e032224d:

33 CI-script unit tests and 6 existing TestPyPI release-gate tests passed locally.
Changed-file pre-commit, actionlint 1.7.12, and syntax checks for all 36 nightly shell steps passed.
The real hosted Docker build completed successfully in about 32 minutes.
The complete nightly run is still pending at the time of writing. It was dispatched with skip_publish=true; this is not evidence of successful TestPyPI publication or MI350X hardware testing.

Recovery is complete when:

A full run reaches an aggregate result without waiting indefinitely for disabled hardware, while every enabled required job remains enforced.
Hosted Docker builds complete within the configured bounds, with diagnostics available if failures recur.
A publishing-enabled run uploads a complete wheel set and passes install/import validation after owner-side TestPyPI recovery.
Missing hardware coverage and publication status are visible, and the retention/runner owners are recorded.

#3556 owns the broader build, CI, and release architecture; this RFC addresses the current nightly failure chain and its operational recovery.
#4450 already proposes twine --verbose. #4750 includes that overlapping diagnostic change within the broader recovery.
#2951 addresses ARM64 package access and Docker disk use in an older workflow layout. #4750 retains current disk cleanup and adds concurrency bounds and diagnostics.
#4729 already fixes the native test compilation regression; #4743 records related nightly failures.

Maintainer decisions: accept the temporary opt-in MI350X policy, choose a TestPyPI retention window and owner, and decide whether longer-lived nightly packages should move to a lifecycle-managed host.

Before submitting a new issue...

Searched relevant issues and read the contribution/workflow documentation; related proposals and overlapping changes are linked above.

Originally created by @stmatengss on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4751 ### Changes proposed ## Motivation and scope Nightly should provide a timely, interpretable health signal for `main`, and validated nightly wheels should be publishable without repeatedly exhausting TestPyPI storage. Recent runs have instead mixed source regressions, unavailable hardware, hosted-runner failures, and publishing failures into an incomplete aggregate result. This RFC proposes a staged recovery of the existing nightly workflow. It is a focused follow-up to the broader build/CI/release roadmap in #3556. The first implementation is #4750; storage administration and longer-term publication policy require separate decisions. ## Problems observed | Area | Evidence | Conclusion and remaining uncertainty | | --- | --- | --- | | ROCm hardware scheduling | Nightly requires a runner with `self-hosted`, `Linux`, `X64`, `amd`, `rocm`, `gfx950`, and `mooncake-pd`. During investigation, the repository runner inventory contained only an ARM64 Ascend runner. The hardware job in [this nightly run](https://github.com/kvcache-ai/Mooncake/actions/runs/39048302137) remained unassigned; the run was later cancelled by a newer request in the `nightly-main` concurrency group. | Unconditional scheduling can prevent nightly from completing when no matching MI350X runner is available. A skipped hardware test must be reported as missing coverage. | | Native test compilation | `scenario.cpp` still called `BatchProbeKey` with two arguments after the API required a `GrantLeasePolicy` argument. | Already fixed on `main` by #4729. The recovery PR does not duplicate that fix. Subsequent publication skips caused by failed prerequisites are not upload failures. | | Docker runner loss | Several Docker jobs reported that the hosted runner lost communication with GitHub. Complete logs were unavailable for those failed jobs. | Resource pressure is a plausible cause, but the available evidence does not establish OOM or disk exhaustion. Bound build concurrency and collect diagnostics before attributing the failure. | | TestPyPI publication | [An upload failed with HTTP 400](https://github.com/kvcache-ai/Mooncake/actions/runs/38596132610/job/116368595261) after transferring the first approximately 95 MB wheel. All 26 files had passed `twine check`. The server response body was absent from the log. | Public storage totals strongly suggest a capacity problem. Some other wheels also exceed the default per-file limit. The actual project limits and the original HTTP 400 reason still require confirmation. | ### TestPyPI storage evidence Public JSON metadata was inspected during this investigation on October 19, 2026 (Asia/Shanghai). The sums below count the sizes of all listed files; private project settings were not accessible. | Project | Listed files | Listed file bytes | Approximate size | | --- | ---: | ---: | ---: | | [`mooncake-transfer-engine`](https://test.pypi.org/pypi/mooncake-transfer-engine/json) | 131 | 10,667,841,775 | 9.94 GiB | | [`mooncake-transfer-engine-cuda13`](https://test.pypi.org/pypi/mooncake-transfer-engine-cuda13/json) | 120 | 10,206,105,359 | 9.51 GiB | | [`mooncake-transfer-engine-non-cuda`](https://test.pypi.org/pypi/mooncake-transfer-engine-non-cuda/json) | — | 4,499,499,434 | 4.19 GiB | Warehouse defaults are 100 MiB per file and 10 GiB per project; project-specific limits may differ. Under the default project limit, the main package has only about 66 MiB available, less than the approximately 95 MB upload that failed. The collected wheel set also contains files around 119–121 MB, above the default per-file limit. These are two distinct limits and both need checking. See [storage-limit documentation](https://docs.pypi.org/project-management/storage-limits/) and [Warehouse defaults](https://github.com/pypi/warehouse/blob/main/warehouse/constants.py). The latest visible main-package development release had only three files, so a successful upload of one file is not sufficient evidence that a nightly release is complete. ## Proposed changes ### 1. Restore bounded, explicit nightly behavior Implemented in #4750: - Select ROCm hardware coverage once per run. Enable it with the `NIGHTLY_ROCM_INTEGRATION=true` repository variable or the `run_rocm_integration` manual input. Otherwise emit a warning and job summary explaining the missing coverage. ROCm wheel builds and smoke tests remain required. - Accept only the hardware job's intentional `skipped` result when it was disabled. When enabled, failure, cancellation, or skipping must fail the nightly gate. Preserve strict checks for all other required jobs and the existing explicit `skip_publish` mode. - Run Docker builds through the configured Buildx builder, limit BuildKit concurrency to one operation, and default `BUILD_JOBS`/`MAX_JOBS` to two. Build the complete Dockerfile with a cache-only output, avoiding a second image-store export. Add memory/disk diagnostics and a 180-minute job timeout. - Check that TestPyPI credentials are present before attempting uploads. Use noninteractive Twine with verbose responses so future failures include the actionable server error. - Add regression tests that execute the workflow shell blocks and verify skip decisions, required-job failures, credential handling, and Docker exit-status propagation. The tradeoff is explicit: a run may complete without MI350X hardware coverage while the runner is unavailable. It must not advertise that coverage as passing. Lower Docker concurrency may increase build time; the timeout and hosted measurements will show whether the default is practical. ### 2. Recover TestPyPI publishing with project-owner actions 1. Sign in to the **TestPyPI** account that owns these projects and open [project management](https://test.pypi.org/manage/projects/). Check each project's storage usage and configured limits. 2. Review old development releases and delete an agreed subset to reclaim space. A proposed starting policy is to retain the latest seven development releases for each package, while preserving stable and release-candidate versions. Deletion is irreversible; yanking does not free storage. Keep any versions needed by existing tests before deleting them. 3. Check the per-file limit for both CUDA variants. If it is still 100 MiB, request an increase, for example to 200 MB, through the support process linked in the storage-limit documentation. Clearly identify the TestPyPI index, project URLs, and measured wheel sizes. Confirm enough project capacity for the complete next wheel set as well. 4. Check `TESTPYPI_API_TOKEN` in the GitHub `nightly` environment. It must be a TestPyPI token with upload access to every project in the wheel set. The first publication of the ROCm project also requires permission to create it if it does not yet exist. The observed HTTP 400 alone is not a reason to rotate a valid token. 5. After the repository fix is merged and storage/limits are ready, start a **new** nightly run from the updated branch with publishing enabled. Rerunning an old failure uses its old commit. Confirm that every expected package, architecture, and Python-version wheel is present, then verify the existing install/import gate. No package deletion, quota change, or secret update is included in #4750 or has been performed by this investigation. ### 3. Prevent recurrence These are follow-up proposals, not behavior already implemented by #4750: - Assign an owner and documented availability expectations to each hardware profile. Decide when MI350X coverage should become mandatory again and enable the repository variable once the matching runner is ready. - Adopt a retention policy with named ownership for TestPyPI development releases. A larger quota alone postpones the same failure. If daily artifacts need long retention, evaluate a package host with a managed lifecycle instead of indefinite TestPyPI accumulation. - Follow #3556 by reporting source health, hardware coverage, and snapshot publication separately. Keep each required profile's result strict, and make an incomplete publication visible without obscuring build/test diagnostics. The current recovery PR preserves the existing aggregate policy. ## Validation and acceptance criteria Current evidence for #4750 at `b1da7c6cfd669fd17055d764f7824562e032224d`: - 33 CI-script unit tests and 6 existing TestPyPI release-gate tests passed locally. - Changed-file pre-commit, actionlint 1.7.12, and syntax checks for all 36 nightly shell steps passed. - The real [hosted Docker build](https://github.com/kvcache-ai/Mooncake/actions/runs/39147854418/job/117735252975) completed successfully in about 32 minutes. - The [complete nightly run](https://github.com/kvcache-ai/Mooncake/actions/runs/39147854418) is still pending at the time of writing. It was dispatched with `skip_publish=true`; this is not evidence of successful TestPyPI publication or MI350X hardware testing. Recovery is complete when: - A full run reaches an aggregate result without waiting indefinitely for disabled hardware, while every enabled required job remains enforced. - Hosted Docker builds complete within the configured bounds, with diagnostics available if failures recur. - A publishing-enabled run uploads a complete wheel set and passes install/import validation after owner-side TestPyPI recovery. - Missing hardware coverage and publication status are visible, and the retention/runner owners are recorded. ## Related work and decisions requested - #3556 owns the broader build, CI, and release architecture; this RFC addresses the current nightly failure chain and its operational recovery. - #4450 already proposes `twine --verbose`. #4750 includes that overlapping diagnostic change within the broader recovery. - #2951 addresses ARM64 package access and Docker disk use in an older workflow layout. #4750 retains current disk cleanup and adds concurrency bounds and diagnostics. - #4729 already fixes the native test compilation regression; #4743 records related nightly failures. Maintainer decisions: accept the temporary opt-in MI350X policy, choose a TestPyPI retention window and owner, and decide whether longer-lived nightly packages should move to a lifecycle-managed host. ### Before submitting a new issue... - [x] Searched relevant issues and read the contribution/workflow documentation; related proposals and overlapping changes are linked above.

bsakaguchi added the

RFC

label

2026-10-29 21:24:00 +00:00

bsakaguchi commented

2026-10-29 21:24:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 18, 2026):

Thanks for opening this issue, @stmatengss!

Field	Value
Issue	#4751
GitHub user ID	`11641725`
Reporter	@stmatengss

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 18, 2026): Thanks for opening this issue, @stmatengss! | Field | Value | |-------|-------| | **Issue** | #4751 | | **GitHub user ID** | `11641725` | | **Reporter** | @stmatengss | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:01 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 18, 2026):

One data point for the problem table, from triaging the linked nightly run on #4749: the nightly-coverage / nightly-test legs failed on a real compile break, but the mechanism was a landing-order race, not a signal-quality problem. #4523 (BatchProbe DSL verb, landed 10-15) added a call in the then-current two-argument form and #4314 (BatchProbeKey gains a leading GrantLeasePolicy, landed 10-16 about 25 hours later) changed the signature; each was green against the main it was tested on, and main was broken for roughly 34 hours until #4729 fixed the call site. So for the "source regressions" row, at least this instance, the gap was that neither PR's CI compiled the other's final tree. A merge queue (or a required rebase-on-merge for files touching the same API surface) is the class of fix that closes it; leg separation alone would still have shipped the break, just with a cleaner report.

Separately, the Build Docker Image leg in that run failed with zero failed steps, which matches the hosted-runner/provisioning category rather than anything in the Dockerfile.

@he-yufeng commented on GitHub (Oct 18, 2026): One data point for the problem table, from triaging the linked nightly run on #4749: the `nightly-coverage` / `nightly-test` legs failed on a real compile break, but the mechanism was a landing-order race, not a signal-quality problem. #4523 (BatchProbe DSL verb, landed 10-15) added a call in the then-current two-argument form and #4314 (BatchProbeKey gains a leading `GrantLeasePolicy`, landed 10-16 about 25 hours later) changed the signature; each was green against the main it was tested on, and main was broken for roughly 34 hours until #4729 fixed the call site. So for the "source regressions" row, at least this instance, the gap was that neither PR's CI compiled the other's final tree. A merge queue (or a required rebase-on-merge for files touching the same API surface) is the class of fix that closes it; leg separation alone would still have shipped the break, just with a cleaner report. Separately, the `Build Docker Image` leg in that run failed with zero failed steps, which matches the hosted-runner/provisioning category rather than anything in the Dockerfile.

bsakaguchi commented

2026-10-29 21:24:02 +00:00

Author

Owner

@whn09 commented on GitHub (Oct 21, 2026):

Data point on the TestPyPI 400: with #4750's verbose upload, the 2026-10-20 nightly (run) now shows the reason. The first upload is rejected with:

400 Project size too large. Limit for project 'mooncake-transfer-engine' total size is 10 GB.

Current TestPyPI usage for mooncake-transfer-engine: 24 releases, 10.67 GB, mostly 0.3.14.dev202609xx at ~0.7 GB each.
One nightly now uploads 28 files, ~2.2 GB. Even an empty project only fits about 4 nightlies under 10 GB, so a one-off cleanup will fill up again within a week. It needs either automatic retention (keep the last N dev releases) or a limit increase.

@whn09 commented on GitHub (Oct 21, 2026): Data point on the TestPyPI 400: with #4750's verbose upload, the 2026-10-20 nightly ([run](https://github.com/kvcache-ai/Mooncake/actions/runs/39455533682)) now shows the reason. The first upload is rejected with: ``` 400 Project size too large. Limit for project 'mooncake-transfer-engine' total size is 10 GB. ``` - Current TestPyPI usage for `mooncake-transfer-engine`: 24 releases, 10.67 GB, mostly `0.3.14.dev202609xx` at ~0.7 GB each. - One nightly now uploads 28 files, ~2.2 GB. Even an empty project only fits about 4 nightlies under 10 GB, so a one-off cleanup will fill up again within a week. It needs either automatic retention (keep the last N dev releases) or a limit increase.

bsakaguchi referenced this issue

2026-10-29 21:32:34 +00:00

[PR #1210] [CLOSED] [Store] feat Uncouple client segment meta from allocator #2074