[PR #246] Linux CUDA runtime interoperability: matched shared CUDART and relocatable wheels #252

New issue

Open

opened 2026-10-26 22:22:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:22:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/246
Author: @xmarre
Created: 10/12/2026
Status: 🔄 Open

Base: main ← Head: experiment/issue245-shared-cudart-linux-20261012

📝 Commits (1)

6803e41 Fix Linux CUDA runtime selection and ensure wheel RPATH is relocatable

📊 Changes

4 files changed (+205 additions, -11 deletions)

View changed files

📝 .github/workflows/build-wheels.yml (+10 -0)
📝 comfy_kitchen/backends/cuda/CMakeLists.txt (+52 -11)
➕ tests/check_cuda_wheel_elf.py (+51 -0)
➕ tests/manual_cuda_runtime_interop.py (+92 -0)

📄 Description

Summary

Fix #245 on Linux: a Kitchen native CUDA call (even non-Sol quantize_int8_rowwise and Sol with token_aug=0) synchronized successfully, but a subsequent PyTorch fill_ launch raised cudaErrorInvalidResourceHandle on the owner's WSL Blackwell / PyTorch 2.10.0+cu130 environment. The failure reproduced across import orders, streams, allocator modes and workspace lifetimes.

This PR does not modify any CUDA kernel or Sol algorithm. It changes Linux runtime selection/linking and tests. Windows retains existing static-CUDART behavior. Companion #244 separately fixes stale FindCUDAToolkit/Python CMake caches; #242 separately fixes actual Sol over-budget token nondeterminism.

Root-cause isolation

A first experimental shared-runtime build compiled with CUDA 13 NVCC but linked against Ubuntu libcudart.so.11.0; both non-Sol and Sol interop tests still failed. This invalidated that A/B. The mismatch was confirmed via ELF DT_NEEDED and /proc/self/maps.
A corrected build linked the selected toolkit's libcudart.so.13 instead, and excluded the host CUDA 11 runtime. The previously deterministic failures disappeared: 3/3 consecutive native quantization + PyTorch operations and 3/3 consecutive native Sol + PyTorch operations passed. Owner subsequently passed 7/7 CUDA numerical tests and a complete H3 run 01907 with a real 12-group overflow and bitwise A/A′ low-source replay. Interop evidence. This demonstrates a working matched shared-CUDA13 configuration, not that static CUDART is categorically broken.

Changes

On Linux, locate libcudart.so.<selected toolkit major> inside the explicitly selected CUDAToolkit_ROOT; fail CMake configure rather than accidentally using an unrelated host CUDA runtime. Link the selected shared soname and set CUDA runtime mode Shared. Preserve static CUDART on Windows.
Do not embed the builder's absolute CUDA path in a wheel. Since setup.py emits the CMake build output directly to the wheel, use BUILD_WITH_INSTALL_RPATH TRUE, INSTALL_RPATH "$ORIGIN" and INSTALL_RPATH_USE_LINK_PATH FALSE. CUDA runtime remains an external dependency as required by existing auditwheel --exclude libcudart.so.13 wheel policy; the consumer must provide a compatible CUDA13 runtime (normally via its PyTorch environment).
Repeat-call diagnostic tests/manual_cuda_runtime_interop.py checks ELF DT_NEEDED, rejects wrong CUDA major and absolute ELF search paths, then exercises non-Sol or plain Sol across multiple native/PyTorch calls.
tests/check_cuda_wheel_elf.py checks the built wheel artifact for CUDA13 DT_NEEDED and rejects absolute RPATH/RUNPATH. Apply it to both x86_64 and ARM64 Linux wheel jobs in .github/workflows/build-wheels.yml.

Validation and limitations

Previous correct-CUDA13 build cd624cfa passed owner RTX PRO 6000 Blackwell native interop 3/3 + 3/3, and full run 01907. That old build still contained an absolute toolkit RPATH.
New commit 6803e41b removes that RPATH and adds wheel-level checks. Fork wheel CI 38213413185 is running; x86_64/ARM64 artifact checks and updated installed-wheel GPU interop on this new head are pending. Do not present it as fully wheel or runtime qualified until both succeed.
This PR does not resolve all PyTorch runtime combinations: Linux shared-CUDART builds depend on an available compatible runtime. Existing wheel repair excludes GPU libraries, so installation/loader behavior with other PyTorch CUDA versions requires maintainer review. Check platform matrix, runtime sourcing/loading, and whether dynamic linkage should be opt-in versus Linux default.

Patch structure: one commit on upstream main be9985bf2977133fb5a48aac4a9df8b6760194c8. Changes to numerical PR #242 or user model settings are intentionally excluded. No merge/release performed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/246 **Author:** [@xmarre](https://github.com/xmarre) **Created:** 10/12/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `experiment/issue245-shared-cudart-linux-20261012` --- ### 📝 Commits (1) - [`6803e41`](https://github.com/Comfy-Org/comfy-kitchen/commit/6803e41bc08b34e3552b974568e644c9e03e5829) Fix Linux CUDA runtime selection and ensure wheel RPATH is relocatable ### 📊 Changes **4 files changed** (+205 additions, -11 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/build-wheels.yml` (+10 -0) 📝 `comfy_kitchen/backends/cuda/CMakeLists.txt` (+52 -11) ➕ `tests/check_cuda_wheel_elf.py` (+51 -0) ➕ `tests/manual_cuda_runtime_interop.py` (+92 -0) </details> ### 📄 Description ## Summary Fix [#245](https://github.com/Comfy-Org/comfy-kitchen/issues/245) on Linux: a Kitchen native CUDA call (even **non-Sol** `quantize_int8_rowwise` and Sol with `token_aug=0`) synchronized successfully, but a subsequent PyTorch `fill_` launch raised `cudaErrorInvalidResourceHandle` on the owner's WSL Blackwell / PyTorch `2.10.0+cu130` environment. The failure reproduced across import orders, streams, allocator modes and workspace lifetimes. **This PR does not modify any CUDA kernel or Sol algorithm.** It changes Linux runtime selection/linking and tests. Windows retains existing static-CUDART behavior. Companion [#244](https://github.com/Comfy-Org/comfy-kitchen/pull/244) separately fixes stale FindCUDAToolkit/Python CMake caches; [#242](https://github.com/Comfy-Org/comfy-kitchen/pull/242) separately fixes actual Sol over-budget token nondeterminism. ### Root-cause isolation - A first experimental shared-runtime build compiled with CUDA 13 NVCC but linked against **Ubuntu `libcudart.so.11.0`**; both non-Sol and Sol interop tests still failed. This invalidated that A/B. The mismatch was confirmed via ELF `DT_NEEDED` and `/proc/self/maps`. - A corrected build linked the selected toolkit's **`libcudart.so.13`** instead, and excluded the host CUDA 11 runtime. The previously deterministic failures disappeared: **3/3** consecutive native quantization + PyTorch operations and **3/3** consecutive native Sol + PyTorch operations passed. Owner subsequently passed 7/7 CUDA numerical tests and a complete H3 run 01907 with a real 12-group overflow and bitwise A/A′ low-source replay. [Interop evidence](https://github.com/Comfy-Org/comfy-kitchen/issues/245#issuecomment-6118364237). This demonstrates a working *matched shared-CUDA13 configuration*, not that static CUDART is categorically broken. ### Changes - On Linux, locate `libcudart.so.<selected toolkit major>` **inside the explicitly selected `CUDAToolkit_ROOT`**; fail CMake configure rather than accidentally using an unrelated host CUDA runtime. Link the selected shared soname and set CUDA runtime mode `Shared`. Preserve static CUDART on Windows. - **Do not embed the builder's absolute CUDA path** in a wheel. Since `setup.py` emits the CMake build output directly to the wheel, use `BUILD_WITH_INSTALL_RPATH TRUE`, `INSTALL_RPATH "$ORIGIN"` and `INSTALL_RPATH_USE_LINK_PATH FALSE`. CUDA runtime remains an **external** dependency as required by existing `auditwheel --exclude libcudart.so.13` wheel policy; the consumer must provide a compatible CUDA13 runtime (normally via its PyTorch environment). - Repeat-call diagnostic `tests/manual_cuda_runtime_interop.py` checks ELF `DT_NEEDED`, rejects wrong CUDA major and absolute ELF search paths, then exercises non-Sol or plain Sol across multiple native/PyTorch calls. - `tests/check_cuda_wheel_elf.py` checks the **built wheel artifact** for CUDA13 `DT_NEEDED` and rejects absolute RPATH/RUNPATH. Apply it to both x86_64 and ARM64 Linux wheel jobs in `.github/workflows/build-wheels.yml`. ### Validation and limitations - Previous correct-CUDA13 build `cd624cfa` passed owner RTX PRO 6000 Blackwell native interop **3/3 + 3/3**, and full run 01907. **That old build still contained an absolute toolkit RPATH.** - New commit `6803e41b` removes that RPATH and adds wheel-level checks. [Fork wheel CI 38213413185](https://github.com/xmarre/comfy-kitchen/actions/runs/38213413185) is running; x86_64/ARM64 artifact checks and updated installed-wheel GPU interop on this new head are **pending**. Do not present it as fully wheel or runtime qualified until both succeed. - This PR does not resolve all PyTorch runtime combinations: Linux shared-CUDART builds depend on an available compatible runtime. Existing wheel repair excludes GPU libraries, so installation/loader behavior with other PyTorch CUDA versions requires maintainer review. Check platform matrix, runtime sourcing/loading, and whether dynamic linkage should be opt-in versus Linux default. **Patch structure:** one commit on upstream `main` `be9985bf2977133fb5a48aac4a9df8b6760194c8`. Changes to numerical PR #242 or user model settings are intentionally excluded. No merge/release performed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:22:07 +00:00

bsakaguchi referenced this issue

2026-10-26 22:22:12 +00:00

[PR #252] [MERGED] HIP port of #251: SiTU input activation in the INT8 ConvRot quantizer #260