[GH-ISSUE #1858] [Tracking] CI Test Failures and Fixes #531

New issue

Closed

opened 2026-10-29 21:15:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:15:58 +00:00

Owner

Originally created by @00fish0 on GitHub (Apr 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1858

This issue tracks CI test failures, flaky tests, and infrastructure issues across Mooncake's CI pipeline.

Scope: We focus on failures in the main branch and PR CI that are NOT caused by the PR itself.

Quick Links

Ongoing Issues

Date	Test	Job	Error	Category	Status	Assignee	Link
2026-04-09	`HealthCheckTest.ReturnsTwoWhenMasterDown`	build (3.10)	Timeout ~18.8s, RPC_FAIL "connection refused" on unmount after master killed	flaky	Enabled	-	Failure
2026-04-09	`HealthCheckTest.HttpReturns503WhenMasterDown`	build (3.12)	Timeout ~18.8s, same pattern as above	flaky	Enabled	-	Failure
2026-04-03	`TestMooncakeStress.test_stress_consistency_fixed`	test-wheel-ubuntu (ubuntu-24.04, 3.12)	Data mismatch on key `stress_fixed_t5_14` (800 items, 8 threads)	flaky	Enabled	-	Failure

Recently Fixed

Date	Test	Fix	Related
-	-	-	-

Infrastructure

Issue	Status	Assignee	Related
-	-	-	-

How to Use This Issue

Reporting a new failure

Comment with:

Date of failure
Test name (e.g., health_check_test.HealthCheckTest.ReturnsTwoWhenMasterDown)
Job name (e.g., build (3.10), test-wheel-ubuntu (ubuntu-24.04, 3.12))
Error summary (one line)
Category: flaky / bug / environment / timeout / infrastructure
Link to the failed CI run

When a failure is fixed

Move the entry from "Ongoing Issues" to "Recently Fixed" with a link to the fix PR.

Originally created by @00fish0 on GitHub (Apr 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1858 This issue tracks CI test failures, flaky tests, and infrastructure issues across Mooncake's CI pipeline. **Scope:** We focus on failures in the `main` branch and PR CI that are NOT caused by the PR itself. ## Quick Links - [CI Workflow: ci.yml](https://github.com/kvcache-ai/Mooncake/blob/main/.github/workflows/ci.yml) - [Integration Test (T-one)](https://github.com/kvcache-ai/Mooncake/blob/main/.github/workflows/integration-test.yml) - [CI CUDA 13](https://github.com/kvcache-ai/Mooncake/blob/main/.github/workflows/ci_cu13.yml) - [CI Ascend](https://github.com/kvcache-ai/Mooncake/blob/main/.github/workflows/ci_ascend.yml) ## Ongoing Issues | Date | Test | Job | Error | Category | Status | Assignee | Link | |------|------|-----|-------|----------|--------|----------|------| | 2026-04-09 | `HealthCheckTest.ReturnsTwoWhenMasterDown` | build (3.10) | Timeout ~18.8s, RPC_FAIL "connection refused" on unmount after master killed | flaky | Enabled | - | [Failure](https://github.com/kvcache-ai/Mooncake/actions/runs/24170824608/job/70542148383?pr=1846) | | 2026-04-09 | `HealthCheckTest.HttpReturns503WhenMasterDown` | build (3.12) | Timeout ~18.8s, same pattern as above | flaky | Enabled | - | [Failure](https://github.com/kvcache-ai/Mooncake/actions/runs/24207701065/job/70667963417?pr=1846) | | 2026-04-03 | `TestMooncakeStress.test_stress_consistency_fixed` | test-wheel-ubuntu (ubuntu-24.04, 3.12) | Data mismatch on key `stress_fixed_t5_14` (800 items, 8 threads) | flaky | Enabled | - | [Failure](https://github.com/kvcache-ai/Mooncake/actions/runs/23883620269/job/69643888109?pr=1800) | ## Recently Fixed  | Date | Test | Fix | Related | |------|------|-----|---------| | - | - | - | - | ## Infrastructure  | Issue | Status | Assignee | Related | |-------|--------|----------|---------| | - | - | - | - | ## How to Use This Issue ### Reporting a new failure Comment with: 1. **Date** of failure 2. **Test name** (e.g., `health_check_test.HealthCheckTest.ReturnsTwoWhenMasterDown`) 3. **Job name** (e.g., `build (3.10)`, `test-wheel-ubuntu (ubuntu-24.04, 3.12)`) 4. **Error summary** (one line) 5. **Category**: `flaky` / `bug` / `environment` / `timeout` / `infrastructure` 6. **Link** to the failed CI run ### When a failure is fixed Move the entry from "Ongoing Issues" to "Recently Fixed" with a link to the fix PR.

bsakaguchi closed this issue

2026-10-29 21:15:58 +00:00

bsakaguchi commented

2026-10-29 21:15:59 +00:00

Author

Owner

@ykwd commented on GitHub (Apr 16, 2026):

The content is put into this Pinned issue: https://github.com/kvcache-ai/Mooncake/issues/1276

@ykwd commented on GitHub (Apr 16, 2026): The content is put into this Pinned issue: https://github.com/kvcache-ai/Mooncake/issues/1276

bsakaguchi referenced this issue

2026-10-29 21:29:01 +00:00

[PR #531] [MERGED] [Build] Optimize store build control for wheel and local build #1629