[GH-ISSUE #4609] Nightly build/test failure - 2026-10-12 #1176

New issue

Open

opened 2026-10-29 21:23:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:37 +00:00

Owner

Originally created by @github-actions[bot] on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4609

Nightly Failure Report

Run: https://github.com/kvcache-ai/Mooncake/actions/runs/38154523646
Branch: refs/heads/main
Timestamp: 2026-10-12T16:47:12.294Z

Please investigate the failed jobs in the workflow run linked above.

Originally created by @github-actions[bot] on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4609 ## Nightly Failure Report **Run**: https://github.com/kvcache-ai/Mooncake/actions/runs/38154523646 **Branch**: refs/heads/main **Timestamp**: 2026-10-12T16:47:12.294Z Please investigate the failed jobs in the workflow run linked above.

bsakaguchi added the

nightly-failure

label

2026-10-29 21:23:37 +00:00

bsakaguchi commented

2026-10-29 21:23:37 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 12, 2026):

Triage for this run (head 559f83cb). Four red legs, one stacked root cause, one fixed bug, one flake.

publish-testpypi has been dead for a month, and it is two size limits stacked on TestPyPI, not flakiness:

Project total size. mooncake-transfer-engine sits at 9.94 GiB against the default 10 GiB project limit on TestPyPI, so since about 09-12 every upload fails with a plain 400, including the 88 MiB aarch64 wheels. The index confirms it: nothing newer than dev20260911, and that one is already partial (3 of 8 files).
Per-file size. The x86_64 CUDA wheels crossed 100 MiB right at the same time (97.4 MiB on 09-10, 111.5 MiB tonight). Those files were dying first and will keep 400ing even after the project has headroom again.

--skip-existing does not help with either, it only skips "already exists" responses. The bare 400 with no per-file log is also why this stayed invisible for a month.

tone-sglang-integration is downstream damage. That leg downloads the exact nightly cuda13 wheel from TestPyPI before submitting the T-one job; nothing has published, so it fails at job creation. Fixing publish should bring it back.

The other two legs are unrelated. build-musa hit the missing using-declaration for mc_fence_noflush, which #4579 fixed a day after this nightly's head. Build Docker Image died on a runner shutdown signal (exit 143), an infra flake.

For a real fix, in order of what unblocks fastest: prune the old dev2026xxxx nightlies from TestPyPI (entirely in our own hands, buys immediate headroom), then ask warehouse support for a project size bump plus a per-file bump for the two CUDA wheel projects, since the x86_64 wheels will keep growing. Longer term the nightly probably wants a retention step so the 10 GiB does not fill up again, and the publish step should report per file instead of aborting on the first 400. Happy to take the workflow half once we pick a direction.

@he-yufeng commented on GitHub (Oct 12, 2026): Triage for this run (head 559f83cb). Four red legs, one stacked root cause, one fixed bug, one flake. `publish-testpypi` has been dead for a month, and it is two size limits stacked on TestPyPI, not flakiness: 1. Project total size. `mooncake-transfer-engine` sits at 9.94 GiB against the default 10 GiB project limit on TestPyPI, so since about 09-12 every upload fails with a plain 400, including the 88 MiB aarch64 wheels. The index confirms it: nothing newer than `dev20260911`, and that one is already partial (3 of 8 files). 2. Per-file size. The x86_64 CUDA wheels crossed 100 MiB right at the same time (97.4 MiB on 09-10, 111.5 MiB tonight). Those files were dying first and will keep 400ing even after the project has headroom again. `--skip-existing` does not help with either, it only skips "already exists" responses. The bare 400 with no per-file log is also why this stayed invisible for a month. `tone-sglang-integration` is downstream damage. That leg downloads the exact nightly cuda13 wheel from TestPyPI before submitting the T-one job; nothing has published, so it fails at job creation. Fixing publish should bring it back. The other two legs are unrelated. `build-musa` hit the missing using-declaration for `mc_fence_noflush`, which #4579 fixed a day after this nightly's head. `Build Docker Image` died on a runner shutdown signal (exit 143), an infra flake. For a real fix, in order of what unblocks fastest: prune the old `dev2026xxxx` nightlies from TestPyPI (entirely in our own hands, buys immediate headroom), then ask warehouse support for a project size bump plus a per-file bump for the two CUDA wheel projects, since the x86_64 wheels will keep growing. Longer term the nightly probably wants a retention step so the 10 GiB does not fill up again, and the publish step should report per file instead of aborting on the first 400. Happy to take the workflow half once we pick a direction.

bsakaguchi referenced this issue

2026-10-29 21:32:24 +00:00

[PR #1176] [MERGED] [MUSA] Enable USE_MNNVL #2051