[GH-ISSUE #4485] Nightly build/test failure - 2026-10-06 #1138

New issue

Open

opened 2026-10-29 21:23:15 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:15 +00:00

Owner

Originally created by @github-actions[bot] on GitHub (Oct 6, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4485

Nightly Failure Report

Run: https://github.com/kvcache-ai/Mooncake/actions/runs/37339233838
Branch: refs/heads/main
Timestamp: 2026-10-06T16:14:30.888Z

Please investigate the failed jobs in the workflow run linked above.

Originally created by @github-actions[bot] on GitHub (Oct 6, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4485 ## Nightly Failure Report **Run**: https://github.com/kvcache-ai/Mooncake/actions/runs/37339233838 **Branch**: refs/heads/main **Timestamp**: 2026-10-06T16:14:30.888Z Please investigate the failed jobs in the workflow run linked above.

bsakaguchi added the

nightly-failure

label

2026-10-29 21:23:15 +00:00

bsakaguchi commented

2026-10-29 21:23:16 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 7, 2026):

Triage: the only test-level failure in this run was master_service_evict_scenario_test, case OpLogDurabilityGatesTenantQuotaReclamation (PutStart(before-durable) succeeded where the scenario expected TENANT_QUOTA_EXCEEDED, plus an ETCD_KEY_NOT_EXIST where the teardown expected OK, with the OpLog writer skipped for lack of an HA connstring). The same job in the 10-06 nightly (run 37494105226) is green, so this reads as a transient environment race rather than a main regression. Watching the next runs; if this case fails again I'll dig into the durability-gate path locally.

@he-yufeng commented on GitHub (Oct 7, 2026): Triage: the only test-level failure in this run was `master_service_evict_scenario_test`, case `OpLogDurabilityGatesTenantQuotaReclamation` (`PutStart(before-durable)` succeeded where the scenario expected TENANT_QUOTA_EXCEEDED, plus an `ETCD_KEY_NOT_EXIST` where the teardown expected OK, with the OpLog writer skipped for lack of an HA connstring). The same job in the 10-06 nightly (run 37494105226) is green, so this reads as a transient environment race rather than a main regression. Watching the next runs; if this case fails again I'll dig into the durability-gate path locally.

bsakaguchi commented

2026-10-29 21:23:16 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 7, 2026):

Nightly follow-up (run 37494105226, scheduled 2026-10-06T16:14Z): no recurrence of the durability-gate case. All completed build and test jobs are green, including nightly-test, nightly-coverage, ascend, musa, rocm builds and the full wheel matrix. The two failures are infrastructure, not code:

publish-testpypi: 400 Bad Request from test.pypi.org on upload.
tone-sglang-integration: external runner (job_id 265388) never reported back; the polling loop hit its 720-attempt timeout.

So no local dig this round. Will keep watching the scheduled runs.

@he-yufeng commented on GitHub (Oct 7, 2026): Nightly follow-up (run 37494105226, scheduled 2026-10-06T16:14Z): no recurrence of the durability-gate case. All completed build and test jobs are green, including nightly-test, nightly-coverage, ascend, musa, rocm builds and the full wheel matrix. The two failures are infrastructure, not code: - `publish-testpypi`: 400 Bad Request from test.pypi.org on upload. - `tone-sglang-integration`: external runner (job_id 265388) never reported back; the polling loop hit its 720-attempt timeout. So no local dig this round. Will keep watching the scheduled runs.