kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3459] [Bug][TENT] RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit races with ProgressWorker #887

New issue

Closed

opened 2026-10-29 21:20:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:33 +00:00

Owner

Originally created by @staryxchen on GitHub (Aug 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3459

Bug Report

Environment

Repo: kvcache-ai/Mooncake
Observed on: PR #3380 CI, job tent-ci (cuda-off)
(run 31924225931)
Binary: tent_runtime_queue_dispatch_test
Test: RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit

This is an intermittent unit-test race, not a production RDMA failure. It does not require a GPU or a real NIC.

Description

DispatchesOnlyOneWindowOnSubmit asserts that a runtime-queue submit with max_dispatch_owners=1 dispatches only the first owner before the test thread polls:

ASSERT_TRUE(engine.submitTransfer(batch, {req0, req1}).ok());
EXPECT_EQ(fake_rdma->submit_calls.load(), 1);  // line 310

That assertion is racing the background ProgressWorker.

When enable_runtime_queue is true, TransferEngineImpl also forces enable_progress_worker_=true. submitTransfer then:

refillDispatchWindow() — dispatches owner 0
notifyRuntimeQueueReady() — wakes ProgressWorker
returns to the test

The test's FakeTransport writes COMPLETED into the fake sub-batch at submit time (no poll_status_factory). If the worker runs progressRuntimeQueue() before the EXPECT_EQ, it observes owner 0 as terminal, releases the window, and immediately dispatches owner 1. The test then sees submit_calls == 2.

The sibling test KeepsDispatchWindowUntilOwnerIsTerminal is the intended coverage for “do not refill until the owner is terminal”: it keeps the first owner PENDING until the test flips a flag, and it passed in the same failing job.

Why this does not show up on every PR

Two independent filters:

Most PRs never run this test. tent-ci is path-filtered to mooncake-transfer-engine/**, mooncake-common/**, top-level CMake/dependencies.sh/ci.yml. Store-only PRs skip the job. Even when tent-ci runs, only cuda-off / cuda-off-metrics-on / ub-mock execute TENT tests; cuda-on is build-only.

When the test does run, the assertion is timing-dependent. Same branch, same test binary:

PR #3380 run	`tent-ci (cuda-off)`
2026-08-14 15:28 (31814699595)	PASS
2026-08-15 04:13 (31863877471)	PASS
2026-08-15 06:11 (31868414521)	PASS
2026-08-16 03:25 (31924225931)	FAIL `submit_calls` is 2, expected 1

In the failing run, tent-ci (cuda-off-metrics-on) and Debug tent-ci (ub-mock) still passed 36/36. A metrics or Debug build changes scheduling enough for the assertion to win.

This is a flake in the test's synchronization, not a deterministic dispatch-window regression.

Evidence from the failing job

runtime_queue_dispatch_test.cpp:312: Failure
Expected equality of these values:
  fake_rdma->submit_calls.load()
    Which is: 2
  1
[  FAILED  ] RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit
35 - tent_runtime_queue_dispatch_test (Failed)

KeepsDispatchWindowUntilOwnerIsTerminal passed in that same job.

Steps to reproduce

Build TENT tests with USE_CUDA=OFF (Release / RelWithDebInfo, same as tent-ci (cuda-off)).
Run:

./mooncake-transfer-engine/tent/tests/tent_runtime_queue_dispatch_test   --gtest_filter=RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit

Repeat under load. The assertion at line 310 intermittently fails with submit_calls == 2. A single passing run does not mean the race is gone.

Expected behavior

The test should check that submit admission only fills one dispatch slot while the first owner is still in-flight. That requires the first fake task to stay non-terminal until the test explicitly completes it (same pattern as KeepsDispatchWindowUntilOwnerIsTerminal).

It should not assert “the progress worker has not run yet”.

Suggested fix

In DispatchesOnlyOneWindowOnSubmit, give FakeTransport a PENDING poll factory until after the submit_calls == 1 check, then complete owner 0 and expect the second submit. Do not change runtime-queue dispatch logic for this failure.

Alternatively, disable the progress worker for this one test if the goal is strictly to unit-test refillDispatchWindow() on the submit path. Keeping the owner PENDING is the smaller, more realistic change.

Before submitting

Searched for existing issues covering this test / runtime-queue window refill race
Confirmed this is independent of the EFA/ROCm auditwheel/patchelf CI failures on the same PR

Originally created by @staryxchen on GitHub (Aug 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3459 ## Bug Report ### Environment - Repo: `kvcache-ai/Mooncake` - Observed on: PR #3380 CI, job `tent-ci (cuda-off)` ([run 31924225931](https://github.com/kvcache-ai/Mooncake/actions/runs/31924225931/job/95109201751)) - Binary: `tent_runtime_queue_dispatch_test` - Test: `RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit` This is an intermittent unit-test race, not a production RDMA failure. It does not require a GPU or a real NIC. ### Description `DispatchesOnlyOneWindowOnSubmit` asserts that a runtime-queue submit with `max_dispatch_owners=1` dispatches only the first owner **before the test thread polls**: ```cpp ASSERT_TRUE(engine.submitTransfer(batch, {req0, req1}).ok()); EXPECT_EQ(fake_rdma->submit_calls.load(), 1); // line 310 ``` That assertion is racing the background `ProgressWorker`. When `enable_runtime_queue` is true, `TransferEngineImpl` also forces `enable_progress_worker_=true`. `submitTransfer` then: 1. `refillDispatchWindow()` — dispatches owner 0 2. `notifyRuntimeQueueReady()` — wakes `ProgressWorker` 3. returns to the test The test's `FakeTransport` writes `COMPLETED` into the fake sub-batch at submit time (no `poll_status_factory`). If the worker runs `progressRuntimeQueue()` before the `EXPECT_EQ`, it observes owner 0 as terminal, releases the window, and immediately dispatches owner 1. The test then sees `submit_calls == 2`. The sibling test `KeepsDispatchWindowUntilOwnerIsTerminal` is the intended coverage for “do not refill until the owner is terminal”: it keeps the first owner `PENDING` until the test flips a flag, and it passed in the same failing job. ### Why this does not show up on every PR Two independent filters: 1. **Most PRs never run this test.** `tent-ci` is path-filtered to `mooncake-transfer-engine/**`, `mooncake-common/**`, top-level CMake/`dependencies.sh`/`ci.yml`. Store-only PRs skip the job. Even when `tent-ci` runs, only `cuda-off` / `cuda-off-metrics-on` / `ub-mock` execute TENT tests; `cuda-on` is build-only. 2. **When the test does run, the assertion is timing-dependent.** Same branch, same test binary: | PR #3380 run | `tent-ci (cuda-off)` | |---|---| | 2026-08-14 15:28 ([31814699595](https://github.com/kvcache-ai/Mooncake/actions/runs/31814699595)) | PASS | | 2026-08-15 04:13 ([31863877471](https://github.com/kvcache-ai/Mooncake/actions/runs/31863877471)) | PASS | | 2026-08-15 06:11 ([31868414521](https://github.com/kvcache-ai/Mooncake/actions/runs/31868414521)) | PASS | | 2026-08-16 03:25 ([31924225931](https://github.com/kvcache-ai/Mooncake/actions/runs/31924225931)) | **FAIL** `submit_calls` is 2, expected 1 | In the failing run, `tent-ci (cuda-off-metrics-on)` and Debug `tent-ci (ub-mock)` still passed 36/36. A metrics or Debug build changes scheduling enough for the assertion to win. This is a flake in the test's synchronization, not a deterministic dispatch-window regression. ### Evidence from the failing job ``` runtime_queue_dispatch_test.cpp:312: Failure Expected equality of these values: fake_rdma->submit_calls.load() Which is: 2 1 [ FAILED ] RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit 35 - tent_runtime_queue_dispatch_test (Failed) ``` `KeepsDispatchWindowUntilOwnerIsTerminal` passed in that same job. ### Steps to reproduce 1. Build TENT tests with `USE_CUDA=OFF` (Release / RelWithDebInfo, same as `tent-ci (cuda-off)`). 2. Run: ```bash ./mooncake-transfer-engine/tent/tests/tent_runtime_queue_dispatch_test --gtest_filter=RuntimeQueueDispatch.DispatchesOnlyOneWindowOnSubmit ``` 3. Repeat under load. The assertion at line 310 intermittently fails with `submit_calls == 2`. A single passing run does not mean the race is gone. ### Expected behavior The test should check that **submit admission** only fills one dispatch slot while the first owner is still in-flight. That requires the first fake task to stay non-terminal until the test explicitly completes it (same pattern as `KeepsDispatchWindowUntilOwnerIsTerminal`). It should **not** assert “the progress worker has not run yet”. ### Suggested fix In `DispatchesOnlyOneWindowOnSubmit`, give `FakeTransport` a `PENDING` poll factory until after the `submit_calls == 1` check, then complete owner 0 and expect the second submit. Do not change runtime-queue dispatch logic for this failure. Alternatively, disable the progress worker for this one test if the goal is strictly to unit-test `refillDispatchWindow()` on the submit path. Keeping the owner `PENDING` is the smaller, more realistic change. ### Before submitting - [x] Searched for existing issues covering this test / runtime-queue window refill race - [x] Confirmed this is independent of the EFA/ROCm `auditwheel`/`patchelf` CI failures on the same PR

bsakaguchi

2026-10-29 21:20:33 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:20:34 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 16, 2026):

Thanks for opening this issue, @staryxchen!

Field	Value
Issue	#3459
GitHub user ID	`151037142`
Reporter	@staryxchen

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 16, 2026): Thanks for opening this issue, @staryxchen! | Field | Value | |-------|-------| | **Issue** | #3459 | | **GitHub user ID** | `151037142` | | **Reporter** | @staryxchen | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.