kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2590] [Bug]: build-musa CI frequently fails during Build project #682

New issue

Closed

opened 2026-10-29 21:18:05 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:18:05 +00:00

Owner

Originally created by @Icedcoco on GitHub (Jun 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2590

Bug Report

Summary

The current GitHub CI frequently fails in the build-musa job under the Build & Test (Linux) workflow. This blocks the final CI Gate for many PRs even when the regular Linux build jobs pass.

Environment

Workflow: Build & Test (Linux)
Job: build-musa
Container: mthreads/musa:rc4.3.0-devel-ubuntu22.04-amd64
CMake options include:
- -DUSE_MUSA=ON
- -DUSE_MNNVL=ON
- -DUSE_ETCD=ON
- -DSTORE_USE_ETCD=ON
- -DUSE_CXL=ON
- -DUSE_TCP=ON

Observed behavior

The failure happens in the Build project step. The error signature is consistent across multiple CI runs and points to missing CUDA-to-MUSA API mappings in the NVLink transport build path.

Representative errors:

mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:69:54: error: 'cudaEventDisableTiming' was not declared in this scope; did you mean 'musaEventDisableTiming'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:69:20: error: 'cudaEventCreateWithFlags' was not declared in this scope; did you mean 'musaEventCreateWithFlags'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:75:30: error: 'cudaEventDestroy' was not declared in this scope; did you mean 'musaEventDestroy'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:383:54: error: 'cudaStreamPerThread' was not declared in this scope; did you mean 'musaStreamPerThread'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:383:9: error: 'cudaEventRecord' was not declared in this scope; did you mean 'musaEventRecord'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:391:16: error: 'cudaStreamWaitEvent' was not declared in this scope; did you mean 'musaStreamWaitEvent'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:464:28: error: 'cudaStreamQuery' was not declared in this scope; did you mean 'musaStreamQuery'?
mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:471:36: error: 'cudaErrorNotReady' was not declared in this scope; did you mean 'musaErrorNotReady'?

Example failed runs:

Expected behavior

build-musa should compile successfully, or at least fail only when a PR changes the MUSA / MNNVL / NVLink build path.

Impact

This makes CI noisy and blocks unrelated PRs. Since the failures are concentrated in the MUSA build job while other Linux build jobs often pass, it is hard for contributors to distinguish real PR regressions from the existing MUSA CI failure.

Possible direction

The failing symbols are CUDA runtime APIs or constants used by nvlink_transport.cpp. The MUSA compatibility layer likely needs mappings for the missing stream/event APIs and constants, or the MUSA build path should avoid compiling CUDA-only NVLink code that depends on them.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Icedcoco on GitHub (Jun 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2590 ### Bug Report ## Summary The current GitHub CI frequently fails in the `build-musa` job under the `Build & Test (Linux)` workflow. This blocks the final `CI Gate` for many PRs even when the regular Linux build jobs pass. ## Environment - Workflow: `Build & Test (Linux)` - Job: `build-musa` - Container: `mthreads/musa:rc4.3.0-devel-ubuntu22.04-amd64` - CMake options include: - `-DUSE_MUSA=ON` - `-DUSE_MNNVL=ON` - `-DUSE_ETCD=ON` - `-DSTORE_USE_ETCD=ON` - `-DUSE_CXL=ON` - `-DUSE_TCP=ON` ## Observed behavior The failure happens in the `Build project` step. The error signature is consistent across multiple CI runs and points to missing CUDA-to-MUSA API mappings in the NVLink transport build path. Representative errors: ```text mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:69:54: error: 'cudaEventDisableTiming' was not declared in this scope; did you mean 'musaEventDisableTiming'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:69:20: error: 'cudaEventCreateWithFlags' was not declared in this scope; did you mean 'musaEventCreateWithFlags'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:75:30: error: 'cudaEventDestroy' was not declared in this scope; did you mean 'musaEventDestroy'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:383:54: error: 'cudaStreamPerThread' was not declared in this scope; did you mean 'musaStreamPerThread'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:383:9: error: 'cudaEventRecord' was not declared in this scope; did you mean 'musaEventRecord'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:391:16: error: 'cudaStreamWaitEvent' was not declared in this scope; did you mean 'musaStreamWaitEvent'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:464:28: error: 'cudaStreamQuery' was not declared in this scope; did you mean 'musaStreamQuery'? mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp:471:36: error: 'cudaErrorNotReady' was not declared in this scope; did you mean 'musaErrorNotReady'? ``` Example failed runs: - https://github.com/kvcache-ai/Mooncake/actions/runs/28068238819 - https://github.com/kvcache-ai/Mooncake/actions/runs/28048536877 - https://github.com/kvcache-ai/Mooncake/actions/runs/28036619557 ## Expected behavior `build-musa` should compile successfully, or at least fail only when a PR changes the MUSA / MNNVL / NVLink build path. ## Impact This makes CI noisy and blocks unrelated PRs. Since the failures are concentrated in the MUSA build job while other Linux build jobs often pass, it is hard for contributors to distinguish real PR regressions from the existing MUSA CI failure. ## Possible direction The failing symbols are CUDA runtime APIs or constants used by `nvlink_transport.cpp`. The MUSA compatibility layer likely needs mappings for the missing stream/event APIs and constants, or the MUSA build path should avoid compiling CUDA-only NVLink code that depends on them. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:05 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:06 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 24, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#2590
GitHub user ID	`102317026`
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 24, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #2590 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:06 +00:00

Author

Owner

@ykwd commented on GitHub (Jun 24, 2026):

@yeahdongcn Could you take a look when you have time? Thanks

@ykwd commented on GitHub (Jun 24, 2026): @yeahdongcn Could you take a look when you have time? Thanks

bsakaguchi commented

2026-10-29 21:18:07 +00:00

Author

Owner

@yeahdongcn commented on GitHub (Jun 24, 2026):

@yeahdongcn Could you take a look when you have time? Thanks

I guess this should be fixed by https://github.com/kvcache-ai/Mooncake/pull/2576

@yeahdongcn commented on GitHub (Jun 24, 2026): > [@yeahdongcn](https://github.com/yeahdongcn) Could you take a look when you have time? Thanks I guess this should be fixed by https://github.com/kvcache-ai/Mooncake/pull/2576

bsakaguchi commented

2026-10-29 21:18:07 +00:00

Author

Owner

@RuixiangMa commented on GitHub (Jun 24, 2026):

@ykwd @yeahdongcn It should have already been fixed in https://github.com/kvcache-ai/Mooncake/pull/2575 ， @Icedcoco you can pull the latest commit and have a try.

@RuixiangMa commented on GitHub (Jun 24, 2026): @ykwd @yeahdongcn It should have already been fixed in https://github.com/kvcache-ai/Mooncake/pull/2575 ， @Icedcoco you can pull the latest commit and have a try.

bsakaguchi commented

2026-10-29 21:18:08 +00:00

Author

Owner

@yeahdongcn commented on GitHub (Jun 24, 2026):

@ykwd @yeahdongcn It should have already been fixed in #2575 ， @Icedcoco you can pull the latest commit and have a try.

Thanks for updating the mapping! I just rebased #2576 (discarded my mapping changes).

@yeahdongcn commented on GitHub (Jun 24, 2026): > [@ykwd](https://github.com/ykwd) [@yeahdongcn](https://github.com/yeahdongcn) It should have already been fixed in [#2575](https://github.com/kvcache-ai/Mooncake/pull/2575) ， [@Icedcoco](https://github.com/Icedcoco) you can pull the latest commit and have a try. Thanks for updating the mapping! I just rebased #2576 (discarded my mapping changes).