[GH-ISSUE #3839] Nightly build/test failure - 2026-09-02 #982

New issue

Closed

opened 2026-10-29 21:21:38 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:21:38 +00:00

Owner

Originally created by @github-actions[bot] on GitHub (Sep 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3839

Nightly Failure Report

Run: https://github.com/kvcache-ai/Mooncake/actions/runs/33653582223
Branch: refs/heads/main
Timestamp: 2026-09-02T17:12:56.703Z

Please investigate the failed jobs in the workflow run linked above.

Originally created by @github-actions[bot] on GitHub (Sep 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3839 ## Nightly Failure Report **Run**: https://github.com/kvcache-ai/Mooncake/actions/runs/33653582223 **Branch**: refs/heads/main **Timestamp**: 2026-09-02T17:12:56.703Z Please investigate the failed jobs in the workflow run linked above.

bsakaguchi

2026-10-29 21:21:38 +00:00

closed this issue
added the
nightly-failure
label

bsakaguchi commented

2026-10-29 21:21:39 +00:00

Author

Owner

@rishabhsinha17 commented on GitHub (Sep 3, 2026):

Triaged the three failing legs of this run:

nightly-coverage / rdma_endpoint_state_test: not flaky, a deterministic LeakSanitizer failure at exit. All 8 tests pass, then LSan reports Direct leak of 1 byte(s) from RdmaEndPoint::construct via the test helper installZeroQpEndpoint (RdmaEndpointLifecycleGateTest.BidirectionalActiveHandshakeDoesNotHoldLifecycleGateDuringRpc). Root cause in rdma_endpoint.cpp: construct(cq, /*num_qp_list=*/0) allocates wr_depth_list_ = new std::atomic<int>[0]() (a real allocation), but ~RdmaEndPoint only calls deconstructLocked() when !qp_list_.empty(), so a constructed zero-QP endpoint destroyed without an explicit deconstruct() never frees wr_depth_list_. Same signature in the 2026-09-01 nightly (#3824), so it has been red since the zero-QP test landed, not since #3814. Taking the one-line destructor fix; PR shortly, verified under an ASan build.

build-wheels (cuda13, x86_64) / Build project: same torch 2.14.0 breakage tracked in #3838; #3842 landed on main today, so tonight's run should clear it.

tone-sglang-integration (on #3824): external tone infra query step, no repo change implicated.

@rishabhsinha17 commented on GitHub (Sep 3, 2026): Triaged the three failing legs of this run: **nightly-coverage / rdma_endpoint_state_test**: not flaky, a deterministic LeakSanitizer failure at exit. All 8 tests pass, then LSan reports `Direct leak of 1 byte(s)` from `RdmaEndPoint::construct` via the test helper `installZeroQpEndpoint` (`RdmaEndpointLifecycleGateTest.BidirectionalActiveHandshakeDoesNotHoldLifecycleGateDuringRpc`). Root cause in `rdma_endpoint.cpp`: `construct(cq, /*num_qp_list=*/0)` allocates `wr_depth_list_ = new std::atomic<int>[0]()` (a real allocation), but `~RdmaEndPoint` only calls `deconstructLocked()` when `!qp_list_.empty()`, so a constructed zero-QP endpoint destroyed without an explicit `deconstruct()` never frees `wr_depth_list_`. Same signature in the 2026-09-01 nightly (#3824), so it has been red since the zero-QP test landed, not since #3814. Taking the one-line destructor fix; PR shortly, verified under an ASan build. **build-wheels (cuda13, x86_64) / Build project**: same torch 2.14.0 breakage tracked in #3838; #3842 landed on main today, so tonight's run should clear it. **tone-sglang-integration** (on #3824): external tone infra query step, no repo change implicated.