[GH-ISSUE #4439] [Bug]: test-wheel CI legs red on all PRs: unpinned pip install torch pulls 2.14.1, wheel only ships PG builds up to 2.14.0 #1117

New issue

Closed

opened 2026-10-29 21:23:03 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:03 +00:00

Owner

Originally created by @he-yufeng on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4439

The wheel test legs of CI are red on every PR right now, including PRs that touch nothing near the process-group code. Two current examples: #3889 and #4433, both failing test-wheel-ubuntu with the same six errors:

ERROR: test_pg_collectives (unittest.loader._FailedTest.test_pg_collectives)
ImportError: Failed to import test module: test_pg_collectives
ModuleNotFoundError: No module named 'mooncake.pg_2_14_1'
ImportError: Mooncake PG was not built against torch==2.14.1.

The chain end to end:

_build-wheel.yaml builds the PG extension for EP_TORCH_VERSIONS = 2.11.0;2.12.0;2.12.1;2.13.0;2.14.0, so the wheel ships mooncake.pg_2_11_0 up to mooncake.pg_2_14_0. The build log confirms exactly those five variants.
scripts/test_installation.sh:42 then runs an unpinned pip install torch numpy in the test venv. PyPI's latest torch is now 2.14.1, so that is what the test env gets.
mooncake/pg.py derives the module name from the installed torch (pg_ + dotted version with underscores) and looks for mooncake.pg_2_14_1, which was never built. All six test_pg_* modules fail at import, the py3.12 leg goes red, and the py3.10 leg gets canceled by fail-fast.

So every torch patch release on PyPI breaks the wheel test until EP_TORCH_VERSIONS catches up, and every open PR shows a red CI Gate in the meantime.

Possible fixes, in increasing scope:

Pin the test env to a torch the wheel actually ships. The wheel is installed a few lines earlier in the same script, so it can enumerate the available mooncake.pg_* modules and install the newest matching torch. Self-healing, no per-release bump needed.
Add 2.14.1 to EP_TORCH_VERSIONS in _build-wheel.yaml. Works until the next torch release.
Longer term, pg.py could fall back to the newest available build within the same minor line when the exact patch version is missing. That would also help end users who install the wheel next to a newer torch, but it is an ABI-compatibility judgement call.

I can send the first option as a small PR if the direction sounds right.

Searched for existing reports of this (pg_2_14_1, wheel CI red), found none.

Originally created by @he-yufeng on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4439 The wheel test legs of CI are red on every PR right now, including PRs that touch nothing near the process-group code. Two current examples: #3889 and #4433, both failing `test-wheel-ubuntu` with the same six errors: ERROR: test_pg_collectives (unittest.loader._FailedTest.test_pg_collectives) ImportError: Failed to import test module: test_pg_collectives ModuleNotFoundError: No module named 'mooncake.pg_2_14_1' ImportError: Mooncake PG was not built against torch==2.14.1. The chain end to end: 1. `_build-wheel.yaml` builds the PG extension for `EP_TORCH_VERSIONS = 2.11.0;2.12.0;2.12.1;2.13.0;2.14.0`, so the wheel ships `mooncake.pg_2_11_0` up to `mooncake.pg_2_14_0`. The build log confirms exactly those five variants. 2. `scripts/test_installation.sh:42` then runs an unpinned `pip install torch numpy` in the test venv. PyPI's latest torch is now 2.14.1, so that is what the test env gets. 3. `mooncake/pg.py` derives the module name from the installed torch (`pg_` + dotted version with underscores) and looks for `mooncake.pg_2_14_1`, which was never built. All six `test_pg_*` modules fail at import, the py3.12 leg goes red, and the py3.10 leg gets canceled by fail-fast. So every torch patch release on PyPI breaks the wheel test until `EP_TORCH_VERSIONS` catches up, and every open PR shows a red CI Gate in the meantime. Possible fixes, in increasing scope: - Pin the test env to a torch the wheel actually ships. The wheel is installed a few lines earlier in the same script, so it can enumerate the available `mooncake.pg_*` modules and install the newest matching torch. Self-healing, no per-release bump needed. - Add 2.14.1 to `EP_TORCH_VERSIONS` in `_build-wheel.yaml`. Works until the next torch release. - Longer term, `pg.py` could fall back to the newest available build within the same minor line when the exact patch version is missing. That would also help end users who install the wheel next to a newer torch, but it is an ABI-compatibility judgement call. I can send the first option as a small PR if the direction sounds right. Searched for existing reports of this (pg_2_14_1, wheel CI red), found none.

bsakaguchi

2026-10-29 21:23:03 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

Thanks for opening this issue, @he-yufeng!

Field	Value
Issue	#4439
GitHub user ID	`40085740`
Reporter	@he-yufeng

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 1, 2026): Thanks for opening this issue, @he-yufeng! | Field | Value | |-------|-------| | **Issue** | #4439 | | **GitHub user ID** | `40085740` | | **Reporter** | @he-yufeng | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:05 +00:00

Author

Owner

@UNIDY2002 commented on GitHub (Oct 6, 2026):

We'd better get #3673 merged ASAP...

@UNIDY2002 commented on GitHub (Oct 6, 2026): We'd better get #3673 merged ASAP...