[GH-ISSUE #4584] [Bug]: Release workflows treat v0.3.14rc1-style tags as stable and publish them to PyPI #1169

New issue

Open

opened 2026-10-29 21:23:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:33 +00:00

Owner

Originally created by @Aionw on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4584

Bug Report

Release workflows tell stable tags from prerelease tags only by whether the tag name contains -. A tag written in canonical PEP 440 form, such as v0.3.14rc1, is therefore treated as a stable release and published to production PyPI.

Current behavior

release.yaml, release-cuda13, release-non-cuda, the three EFA workflows, release-npu, release-musa and release-rocm all trigger on v*. They treat a tag as stable when it contains no - (!contains(github.ref_name, '-'), or the *-* check in the MUSA/ROCm dispatch validation).
pre-release.yaml only matches v*-rc*, v*-alpha*, v*-beta* and v*-pre*.

0.3.14rc1 is the canonical PEP 440 spelling; 0.3.14-rc1 is an accepted alternative that normalizes to it. Someone used to Python packaging could reasonably push v0.3.14rc1. That tag:

does not match pre-release.yaml, so no prerelease build, GitHub Pre-release or TestPyPI validation runs;
passes every stable gate, so all release workflows build and publish to production PyPI.

The stable path also ignores the version in the tag. _build-wheel.yaml exports VERSION=${GITHUB_REF_NAME#v}, but only patches pyproject.toml when version-override is set, and scripts/build_wheel.sh never reads VERSION. The custom NPU/MUSA/ROCm paths likewise build from pyproject.toml as-is. So the published version is whatever mooncake-wheel/pyproject.toml contains:

if it has already been bumped for the upcoming release (e.g. 0.3.14), the rc tag publishes the final 0.3.14 to PyPI ahead of schedule, and that version can never be re-uploaded;
otherwise the upload collides with an existing version and fails partway, after the GitHub Release assets have already been attached.

The same coarse rule also makes tags like v0.3.13-1 (PEP 440 0.3.13.post1) skip every release workflow without any error.

Expected behavior

Every release workflow classifies a tag the same way, and each tag lands on exactly one path:

stable tag: all stable workflows run, and the wheel version equals the tag version;
prerelease tag (in either spelling): only the prerelease path runs, with nothing published to production PyPI;
anything else: fails explicitly instead of silently skipping or publishing.

Possible fix

Classify the tag once with packaging.Version (e.g. extend scripts/ci/testpypi_wheel_gate.py normalize-version into a shared classify step), and have every release workflow gate on its output instead of contains(..., '-'). On the stable path, check that the tag version matches pyproject.toml before publishing. That check is one of the invariants proposed in #3534 ("Git tags, pyproject.toml, and wheel metadata must agree") and could land ahead of the full migration.

A cheaper interim option: document that release tags must use the -rcN / -alphaN / -betaN form, and have the stable workflows fail on tags whose PEP 440 parse is a prerelease.

#4574 changes release-npu.yaml to classify tags with packaging.Version. If it lands as is, NPU would diverge from the other workflows on these tags (review: https://github.com/kvcache-ai/Mooncake/pull/4574#pullrequestreview-5487709039).
#3534, #3556: release-architecture RFC and roadmap.

Before submitting...

Ensure you searched for relevant issues and read the documentation

Originally created by @Aionw on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4584 ### Bug Report Release workflows tell stable tags from prerelease tags only by whether the tag name contains `-`. A tag written in canonical PEP 440 form, such as `v0.3.14rc1`, is therefore treated as a stable release and published to production PyPI. ### Current behavior - `release.yaml`, `release-cuda13`, `release-non-cuda`, the three EFA workflows, `release-npu`, `release-musa` and `release-rocm` all trigger on `v*`. They treat a tag as stable when it contains no `-` (`!contains(github.ref_name, '-')`, or the `*-*` check in the MUSA/ROCm dispatch validation). - `pre-release.yaml` only matches `v*-rc*`, `v*-alpha*`, `v*-beta*` and `v*-pre*`. `0.3.14rc1` is the canonical PEP 440 spelling; `0.3.14-rc1` is an accepted alternative that normalizes to it. Someone used to Python packaging could reasonably push `v0.3.14rc1`. That tag: 1. does not match `pre-release.yaml`, so no prerelease build, GitHub Pre-release or TestPyPI validation runs; 2. passes every stable gate, so all release workflows build and publish to production PyPI. The stable path also ignores the version in the tag. `_build-wheel.yaml` exports `VERSION=${GITHUB_REF_NAME#v}`, but only patches `pyproject.toml` when `version-override` is set, and `scripts/build_wheel.sh` never reads `VERSION`. The custom NPU/MUSA/ROCm paths likewise build from `pyproject.toml` as-is. So the published version is whatever `mooncake-wheel/pyproject.toml` contains: - if it has already been bumped for the upcoming release (e.g. `0.3.14`), the rc tag publishes the **final** `0.3.14` to PyPI ahead of schedule, and that version can never be re-uploaded; - otherwise the upload collides with an existing version and fails partway, after the GitHub Release assets have already been attached. The same coarse rule also makes tags like `v0.3.13-1` (PEP 440 `0.3.13.post1`) skip every release workflow without any error. ### Expected behavior Every release workflow classifies a tag the same way, and each tag lands on exactly one path: - stable tag: all stable workflows run, and the wheel version equals the tag version; - prerelease tag (in either spelling): only the prerelease path runs, with nothing published to production PyPI; - anything else: fails explicitly instead of silently skipping or publishing. ### Possible fix Classify the tag once with `packaging.Version` (e.g. extend `scripts/ci/testpypi_wheel_gate.py normalize-version` into a shared classify step), and have every release workflow gate on its output instead of `contains(..., '-')`. On the stable path, check that the tag version matches `pyproject.toml` before publishing. That check is one of the invariants proposed in #3534 ("Git tags, `pyproject.toml`, and wheel metadata must agree") and could land ahead of the full migration. A cheaper interim option: document that release tags must use the `-rcN` / `-alphaN` / `-betaN` form, and have the stable workflows fail on tags whose PEP 440 parse is a prerelease. ### Related - #4574 changes `release-npu.yaml` to classify tags with `packaging.Version`. If it lands as is, NPU would diverge from the other workflows on these tags (review: https://github.com/kvcache-ai/Mooncake/pull/4574#pullrequestreview-5487709039). - #3534, #3556: release-architecture RFC and roadmap. ### Before submitting... - [x] Ensure you searched for relevant issues and read the documentation

bsakaguchi added the

bug

label

2026-10-29 21:23:33 +00:00

bsakaguchi commented

2026-10-29 21:23:34 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 12, 2026):

Thanks for opening this issue, @Aionw!

Field	Value
Issue	#4584
GitHub user ID	`41376987`
Reporter	@Aionw

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 12, 2026): Thanks for opening this issue, @Aionw! | Field | Value | |-------|-------| | **Issue** | #4584 | | **GitHub user ID** | `41376987` | | **Reporter** | @Aionw | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.