[PR #244] Fix source-wheel CMake cache invalidation for CUDA and Python toolchains #253

New issue

Open

opened 2026-10-26 22:22:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:22:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/244
Author: @xmarre
Created: 10/12/2026
Status: 🔄 Open

Base: main ← Head: fix/kitchen-cmake-build-cache-python-toolchain-20261012

📝 Commits (1)

0e2d78c Fix source-wheel CMake toolchain cache including stale CUDART provenance

📊 Changes

2 files changed (+252 additions, -0 deletions)

View changed files

📝 setup.py (+103 -0)
➕ tests/test_cuda_cmake_cache.py (+149 -0)

📄 Description

Summary

Fix source-wheel builds that unintentionally reuse CMake compiler, FindPython or CUDA runtime-library discovery from a previous environment. This is build-system only: setup.py and tests/test_cuda_cmake_cache.py; it changes no CUDA kernels, attention numerics or model API. Separated from numerical PR #242.

Observed failures

In a managed Python 3.12 / CUDA 13.0 WSL source build, stale CMAKE_CUDA_COMPILER from CUDA 11.5 caused CMake to reset mid-configuration, losing COMFY_CUDA_ARCHS. Separately, CMake's internal _Python_EXECUTABLE/_Python_INCLUDE_DIR pointed at system Python 3.10 despite a correct command-line Python_EXECUTABLE. Later CUDA interoperability investigation #245 also showed a build with CUDA 13 NVCC linked to libcudart.so.11.0: checking compiler identity alone is insufficient when CUDA_cudart_LIBRARY has been cached independently.

Fix

Before CUDA configure, compare cached public/internal CUDA compiler, Python interpreter/header artifacts, explicit CUDAToolkit_ROOT, and cached shared/static CUDA runtime libraries against the selected toolkit root.
Reset only generated build/temp.../cuda CMake state when a mismatch is found. Preserve the HIP build folder and incremental CUDA builds when identities match. Handle stale CMakeFiles left without a cache.
Pass Python_EXECUTABLE, Python_INCLUDE_DIR and Python_ROOT_DIR explicitly; fail early and clearly if the selected interpreter has no Python.h.
Handle Windows paths on different drive letters without raising commonpath errors.

Verification

Fork focused CI run 38213411045: 9/9 cache regression tests passed, Ruff check + formatting passed on Python 3.12. New tests cover cached external CUDA 11 runtimes despite matching NVCC, valid in-toolkit CUDART paths, and stale toolkit root, in addition to six original cases.
Earlier combined source-wheel CI 38201552241 passed and the owner successfully compiled/ran the resulting installed extension on RTX PRO 6000 Blackwell with CUDA 13. Updated full wheel CI is running; it is not yet evidence of a passed Windows/Linux native build at this head.
Caller ComfyUI-Patcher issue #68 separately owns checkout-revision/build-artifact lifecycle. This PR only fixes the Python/CUDA toolkit selection of an individual source-tree build.

Review notes

Review how FindCUDAToolkit caches library paths across environments, whether cache invalidation is sufficiently narrow, and CMake/Python discovery on Windows. A bad external CUDA runtime found again after a cache reset is rejected independently by #246 in its Linux shared-runtime mode. No promise that the cache reset alone repairs PyTorch/CUDA runtime interoperability.

Patch structure: one commit on upstream main be9985bf2977133fb5a48aac4a9df8b6760194c8.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/244 **Author:** [@xmarre](https://github.com/xmarre) **Created:** 10/12/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/kitchen-cmake-build-cache-python-toolchain-20261012` --- ### 📝 Commits (1) - [`0e2d78c`](https://github.com/Comfy-Org/comfy-kitchen/commit/0e2d78cd4769b301c72b252aee8ff20380128a81) Fix source-wheel CMake toolchain cache including stale CUDART provenance ### 📊 Changes **2 files changed** (+252 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `setup.py` (+103 -0) ➕ `tests/test_cuda_cmake_cache.py` (+149 -0) </details> ### 📄 Description ## Summary Fix source-wheel builds that unintentionally reuse CMake compiler, FindPython or CUDA runtime-library discovery from a previous environment. This is **build-system only**: `setup.py` and `tests/test_cuda_cmake_cache.py`; it changes no CUDA kernels, attention numerics or model API. Separated from [numerical PR #242](https://github.com/Comfy-Org/comfy-kitchen/pull/242). ### Observed failures In a managed Python 3.12 / CUDA 13.0 WSL source build, stale `CMAKE_CUDA_COMPILER` from CUDA 11.5 caused CMake to reset mid-configuration, losing `COMFY_CUDA_ARCHS`. Separately, CMake's internal `_Python_EXECUTABLE`/`_Python_INCLUDE_DIR` pointed at system Python 3.10 despite a correct command-line `Python_EXECUTABLE`. Later CUDA interoperability investigation [#245](https://github.com/Comfy-Org/comfy-kitchen/issues/245) also showed a build with CUDA 13 NVCC linked to **`libcudart.so.11.0`**: checking compiler identity alone is insufficient when `CUDA_cudart_LIBRARY` has been cached independently. ### Fix - Before CUDA configure, compare cached public/internal CUDA compiler, Python interpreter/header artifacts, explicit `CUDAToolkit_ROOT`, and cached shared/static CUDA runtime libraries against the selected toolkit root. - Reset **only** generated `build/temp.../cuda` CMake state when a mismatch is found. Preserve the HIP build folder and incremental CUDA builds when identities match. Handle stale `CMakeFiles` left without a cache. - Pass `Python_EXECUTABLE`, `Python_INCLUDE_DIR` and `Python_ROOT_DIR` explicitly; fail early and clearly if the selected interpreter has no `Python.h`. - Handle Windows paths on different drive letters without raising `commonpath` errors. ### Verification - [Fork focused CI run 38213411045](https://github.com/xmarre/comfy-kitchen/actions/runs/38213411045): **9/9 cache regression tests passed**, Ruff check + formatting passed on Python 3.12. New tests cover cached external CUDA 11 runtimes despite matching NVCC, valid in-toolkit CUDART paths, and stale toolkit root, in addition to six original cases. - Earlier combined source-wheel CI [38201552241](https://github.com/xmarre/comfy-kitchen/actions/runs/38201552241) passed and the owner successfully compiled/ran the resulting installed extension on RTX PRO 6000 Blackwell with CUDA 13. Updated [full wheel CI](https://github.com/xmarre/comfy-kitchen/actions/runs/38213411074) is running; it is **not yet evidence of a passed Windows/Linux native build at this head**. - Caller [ComfyUI-Patcher issue #68](https://github.com/xmarre/ComfyUI-Patcher/issues/68) separately owns checkout-revision/build-artifact lifecycle. This PR only fixes the Python/CUDA toolkit selection of an individual source-tree build. ### Review notes Review how FindCUDAToolkit caches library paths across environments, whether cache invalidation is sufficiently narrow, and CMake/Python discovery on Windows. A bad external CUDA runtime found **again after a cache reset** is rejected independently by [#246](https://github.com/Comfy-Org/comfy-kitchen/pull/246) in its Linux shared-runtime mode. No promise that the cache reset alone repairs PyTorch/CUDA runtime interoperability. **Patch structure:** one commit on upstream `main` `be9985bf2977133fb5a48aac4a9df8b6760194c8`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:22:07 +00:00

bsakaguchi referenced this issue

2026-10-26 22:22:11 +00:00

[PR #253] Route INT8 linear_input_act through the int8_linear custom op #258