Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #245] Sol-Attn repeated call invalid resource handle on Blackwell with CUDA 13 source wheel #67

New issue

Open

opened 2026-10-26 22:19:55 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-26 22:19:55 +00:00

Owner

Originally created by @xmarre on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/245

Reproduction status (2026-10-12)

An owner building comfy-kitchen 0.2.37 from source in a managed WSL Conda comfy312 environment (PyTorch 2.10.0+cu130, NVIDIA RTX PRO 6000 Blackwell, CUDA 13.0.88 NVCC) observes torch.AcceleratorError: CUDA error: invalid resource handle on subsequent Sol-Attn calls.

Critical scope: This was first observed while testing draft Sol token-overflow PR #242, stacked with independent build-only PR #244. It also reproduces with token_aug=0, which bypasses every new token/retry kernel in #242. There is no clean wheel baseline A/B yet; do not conclude this is an upstream baseline defect or introduced by #242. #242's token_aug=0 changes are limited to an extra zero-byte workspace slot.

Controlled isolation

The installed compiled _C.abi3.so imports, advertises CUDA availability and exposes the added tokOverflow workspace field.
Four isolated CUDA checks pass when used once: PyTorch matmul; PyTorch matmul after importing Kitchen; non-Sol Kitchen cuda.quantize_int8_rowwise; one plain ck.sol_attn(q,k,v,tau=4.,token_aug=0) + torch.cuda.synchronize().
Six fresh-process A/B trials use fixed t=4096,h=2,d=128, BF16 QKV, identical seed, CUDA_LAUNCH_BLOCKING=1, synchronization before and after each call: all 6 first Sol calls synchronize successfully; all 6 second calls fail on _block_lengths -> torch.full(...device='cuda'), before a second native launch.
3 trials import Kitchen before PyTorch CUDA initialization and 3 initialize CUDA via torch.cuda.init() before importing Kitchen. All exhibit identical first-success / second-failure, so import order does not explain it.
Additional plain/augmented/saturated-token initial tests failed with invalid resource handle on sync; the earlier multi-case pytest run yielded 4 failures, all error handling rather than numerical assertions.
/proc/self/maps reports both nvidia/cu13/lib/libcudart.so.13 and nvidia/cuda_runtime/lib/libcudart.so.12; coexistence is an observation, not proven causal. Native extension links CUDA::cudart_static and passes the PyTorch current CUDA stream handle.

Next diagnostic

Keep input and CUDA environment fixed, and compare in fresh processes:

PyTorch allocation-only + repeated non-Sol Kitchen CUDA control.
High-level ck.sol_attn (creates a temporary internal workspace per call).
Direct cuda._C.sol_attn with externally allocated persistent workspace/output and explicit synchronization + tiny allocator probes after each call.
Direct binding with ephemeral workspace per call, to isolate lifetime/recycling.
If needed, explicit nondefault stream and caching allocator A/B, then stage-local C++ CUDA checks.

Do not change H3 defaults, gate Sol, alter numerical overflow code, or merge the draft on the basis of this failure. A separate clean upstream wheel A/B remains necessary to establish provenance.

Originally created by @xmarre on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/245 ## Reproduction status (2026-10-12) An owner building comfy-kitchen `0.2.37` from source in a managed WSL Conda `comfy312` environment (PyTorch `2.10.0+cu130`, NVIDIA RTX PRO 6000 Blackwell, CUDA 13.0.88 NVCC) observes `torch.AcceleratorError: CUDA error: invalid resource handle` on subsequent Sol-Attn calls. **Critical scope:** This was first observed while testing [draft Sol token-overflow PR #242](https://github.com/Comfy-Org/comfy-kitchen/pull/242), stacked with independent build-only [PR #244](https://github.com/Comfy-Org/comfy-kitchen/pull/244). **It also reproduces with `token_aug=0`**, which bypasses every new token/retry kernel in #242. There is **no clean wheel baseline A/B yet**; do not conclude this is an upstream baseline defect or introduced by #242. #242's `token_aug=0` changes are limited to an extra zero-byte workspace slot. ### Controlled isolation - The installed compiled `_C.abi3.so` imports, advertises CUDA availability and exposes the added `tokOverflow` workspace field. - Four isolated CUDA checks pass when used once: PyTorch matmul; PyTorch matmul after importing Kitchen; non-Sol Kitchen `cuda.quantize_int8_rowwise`; one plain `ck.sol_attn(q,k,v,tau=4.,token_aug=0)` + `torch.cuda.synchronize()`. - Six fresh-process A/B trials use fixed `t=4096,h=2,d=128`, BF16 QKV, identical seed, `CUDA_LAUNCH_BLOCKING=1`, synchronization before and after each call: **all 6 first Sol calls synchronize successfully; all 6 second calls fail** on `_block_lengths -> torch.full(...device='cuda')`, before a second native launch. - 3 trials import Kitchen before PyTorch CUDA initialization and 3 initialize CUDA via `torch.cuda.init()` before importing Kitchen. **All exhibit identical first-success / second-failure**, so import order does not explain it. - Additional plain/augmented/saturated-token initial tests failed with invalid resource handle on sync; the earlier multi-case pytest run yielded 4 failures, all error handling rather than numerical assertions. - `/proc/self/maps` reports both `nvidia/cu13/lib/libcudart.so.13` and `nvidia/cuda_runtime/lib/libcudart.so.12`; coexistence is an observation, not proven causal. Native extension links `CUDA::cudart_static` and passes the PyTorch current CUDA stream handle. ### Next diagnostic Keep input and CUDA environment fixed, and compare in fresh processes: 1. PyTorch allocation-only + repeated non-Sol Kitchen CUDA control. 2. High-level `ck.sol_attn` (creates a temporary internal `workspace` per call). 3. Direct `cuda._C.sol_attn` with externally allocated **persistent** workspace/output and explicit synchronization + tiny allocator probes after each call. 4. Direct binding with ephemeral workspace per call, to isolate lifetime/recycling. 5. If needed, explicit nondefault stream and caching allocator A/B, then stage-local C++ CUDA checks. Do not change H3 defaults, gate Sol, alter numerical overflow code, or merge the draft on the basis of this failure. A separate clean upstream wheel A/B remains necessary to establish provenance.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Additional isolation: non-Sol Kitchen CUDA also poisons a subsequent PyTorch operation (2026-10-12)

Owner supplied kitchen-workspace-lifetime.log, five fresh processes with CUDA_LAUNCH_BLOCKING=1, same BF16 QKV, sync after first native operation, and torch.full((64,),1.,device='cuda') plus sync as the PyTorch probe:

Case	initial call + synchronization	subsequent probe
PyTorch tensor `copy_` (twice)	PASS	PASS
unrelated Kitchen `cuda.quantize_int8_rowwise`	PASS	invalid resource handle
public `ck.sol_attn(...token_aug=0)`	PASS	invalid resource handle
direct `cuda._C.sol_attn` retaining workspaces/outputs	PASS	invalid resource handle
direct `cuda._C.sol_attn` dropping workspace	PASS	invalid resource handle

All failures occur at same script line in the probe stage after successful synchronization of the first native call. This is not specific to Sol, #242's overflow retry, import order, or short-lived workspace. The error could originate in the PyTorch probe's allocation or its subsequent synchronization (current traceback line alone must be cross-checked with exact script).

Likely shared layer to investigate: extension links CUDA::cudart_static; the owner previously reported libcudart.so.13 and libcudart.so.12 mapped alongside PyTorch 2.10.0+cu130. There is an analogous NVIDIA CUDA-Q issue #4236 for distinct runtimes interacting with PyTorch and causing invalid resource handles, but this is not proof of the same mechanism here.

Next discriminating test: in fresh processes compare default stream vs explicit nondefault PyTorch stream, PyTorch caching allocator enabled vs disabled, and place log prints between torch.empty, fill_, and torch.cuda.synchronize to locate the precise failing operation. Log torch.cuda.current_stream().cuda_stream, cuda runtime maps, and PyTorch allocator settings. If confirmed runtime-related, test a separate build-linkage PR (CUDA::cudart dynamic instead of static) with Windows/Linux CI and native Blackwell evaluation; do not roll it into numerical PR #242 without proving it.

@xmarre commented on GitHub (Oct 12, 2026): ## Additional isolation: non-Sol Kitchen CUDA also poisons a subsequent PyTorch operation (2026-10-12) Owner supplied `kitchen-workspace-lifetime.log`, five fresh processes with `CUDA_LAUNCH_BLOCKING=1`, same BF16 QKV, sync after first native operation, and `torch.full((64,),1.,device='cuda')` plus sync as the PyTorch probe: | Case | initial call + synchronization | subsequent probe | |---|---|---| | PyTorch tensor `copy_` (twice) | PASS | PASS | | unrelated Kitchen `cuda.quantize_int8_rowwise` | PASS | **invalid resource handle** | | public `ck.sol_attn(...token_aug=0)` | PASS | **invalid resource handle** | | direct `cuda._C.sol_attn` retaining workspaces/outputs | PASS | **invalid resource handle** | | direct `cuda._C.sol_attn` dropping workspace | PASS | **invalid resource handle** | All failures occur at same script line in the probe stage *after successful synchronization of the first native call*. This is **not specific to Sol, #242's overflow retry, import order, or short-lived workspace**. The error could originate in the PyTorch probe's allocation or its subsequent synchronization (current traceback line alone must be cross-checked with exact script). **Likely shared layer to investigate:** extension links `CUDA::cudart_static`; the owner previously reported `libcudart.so.13` **and** `libcudart.so.12` mapped alongside PyTorch `2.10.0+cu130`. There is an analogous NVIDIA [CUDA-Q issue #4236](https://github.com/NVIDIA/cuda-quantum/issues/4236) for distinct runtimes interacting with PyTorch and causing invalid resource handles, but this is not proof of the same mechanism here. Next discriminating test: in fresh processes compare default stream vs explicit nondefault PyTorch stream, PyTorch caching allocator enabled vs disabled, and place log prints between `torch.empty`, `fill_`, and `torch.cuda.synchronize` to locate the precise failing operation. Log `torch.cuda.current_stream().cuda_stream`, cuda runtime maps, and PyTorch allocator settings. If confirmed runtime-related, test a separate **build-linkage** PR (`CUDA::cudart` dynamic instead of static) with Windows/Linux CI and native Blackwell evaluation; do not roll it into numerical PR #242 without proving it.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Four-case stream/allocator isolation: all fail specifically on a subsequent PyTorch kernel (2026-10-12)

Owner's kitchen-runtime-isolation.log repeats a non-Sol Kitchen operation with CUDA_LAUNCH_BLOCKING=1. Independent processes:

Stream	`PYTORCH_NO_CUDA_MEMORY_CACHING`	`cuda.quantize_int8_rowwise` + `torch.cuda.synchronize()`	`torch.empty(64,device='cuda')`	`probe.fill_(1.)`
Default (handle 0x0)	0	PASS	PASS	cudaErrorInvalidResourceHandle
Non-default (nonzero handle)	0	PASS	PASS	same
Default	1	PASS	PASS	same
Non-default	1	PASS	PASS	same

The second native call is never reached. Caching, nondefault streams, transient Sol-Attn workspace, import order and Sol overflow logic are now excluded as necessary conditions. In particular, this is not a failure of memory allocation; failure occurs when PyTorch tries to launch fill_ after the successful native kernel and synchronization.

Kitchen links CUDA::cudart_static (in comfy_kitchen/backends/cuda/CMakeLists.txt), while loaded process contains libcudart.so.13 and libcudart.so.12. NVIDIA's best practices state multiple static runtimes may coexist, so linkage incompatibility remains a hypothesis, not proof. A distinct experiment comparing shared CUDART linking can help, but packaging/ABI implications must be checked before upstream adoption. Do not roll this into numerical #242.

Next qualification: gather readelf -d/loader mapping and driver context / stream state at the failing boundary; experimentally build the same extension with CUDART shared linkage in separate draft PR, then run the exact quantization->PyTorch fill reproduction against both builds. Keep #242 and #244 unchanged.

@xmarre commented on GitHub (Oct 12, 2026): ## Four-case stream/allocator isolation: all fail specifically on a subsequent PyTorch kernel (2026-10-12) Owner's `kitchen-runtime-isolation.log` repeats a non-Sol Kitchen operation with `CUDA_LAUNCH_BLOCKING=1`. Independent processes: | Stream | `PYTORCH_NO_CUDA_MEMORY_CACHING` | `cuda.quantize_int8_rowwise` + `torch.cuda.synchronize()` | `torch.empty(64,device='cuda')` | `probe.fill_(1.)` | |---|---|---|---|---| | Default (handle 0x0) | 0 | PASS | PASS | **cudaErrorInvalidResourceHandle** | | Non-default (nonzero handle) | 0 | PASS | PASS | **same** | | Default | 1 | PASS | PASS | **same** | | Non-default | 1 | PASS | PASS | **same** | The second native call is never reached. Caching, nondefault streams, transient Sol-Attn workspace, import order and Sol overflow logic are now excluded as *necessary* conditions. In particular, this is not a failure of memory *allocation*; failure occurs when PyTorch tries to **launch** `fill_` after the successful native kernel and synchronization. Kitchen links `CUDA::cudart_static` (in `comfy_kitchen/backends/cuda/CMakeLists.txt`), while loaded process contains `libcudart.so.13` and `libcudart.so.12`. NVIDIA's best practices state multiple static runtimes may coexist, so linkage incompatibility remains a **hypothesis**, not proof. A distinct *experiment* comparing shared CUDART linking can help, but packaging/ABI implications must be checked before upstream adoption. Do not roll this into numerical #242. Next qualification: gather `readelf -d`/loader mapping and driver context / stream state at the failing boundary; experimentally build the same extension with CUDART shared linkage in separate draft PR, then run the exact quantization->PyTorch fill reproduction against both builds. Keep #242 and #244 unchanged.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Linkage-only A/B now available: draft #246

Based on the four-case allocator/stream isolation above, I published a separate, single-commit diagnostic PR that switches Linux only from CUDA::cudart_static to CUDA::cudart and sets both target runtime properties to Shared, leaving Windows static and the entire numerical CUDA source untouched. The native reproducer tests/manual_cuda_runtime_interop.py checks repeated non-Sol quantization and plain Sol calls followed by PyTorch fill_, and prints readelf's extension libcudart dependency.

Fork wheel CI: https://github.com/xmarre/comfy-kitchen/actions/runs/38207087682 (initially queued).
Owner can temporarily stack #242 + #244 + #246 via Patcher and rebuild transactionally.
Confirm DT_NEEDED: libcudart.so.13 and 3 iterations of --operation quantize followed by --operation sol using the existing script.
If the shared-runtime build cannot import or the exact failure persists, document and remove only #246, restoring the prior two-PR stack; do not infer linkage was causal.
NVIDIA documentation explicitly permits coexistence of multiple static CUDA runtimes; this is an experimental A/B, not a proposed default or upstream-ready fix. Dynamic-runtime dependency bundling/loader compatibility also requires validation before merge.

This leaves numerical draft #242 and build-system draft #244 unchanged.

@xmarre commented on GitHub (Oct 12, 2026): ### Linkage-only A/B now available: [draft #246](https://github.com/Comfy-Org/comfy-kitchen/pull/246) Based on the four-case allocator/stream isolation above, I published **a separate, single-commit diagnostic PR** that switches **Linux only** from `CUDA::cudart_static` to `CUDA::cudart` and sets both target runtime properties to `Shared`, leaving Windows static and the entire numerical CUDA source untouched. The native reproducer `tests/manual_cuda_runtime_interop.py` checks repeated **non-Sol** quantization and plain Sol calls followed by PyTorch `fill_`, and prints `readelf`'s extension `libcudart` dependency. - Fork wheel CI: https://github.com/xmarre/comfy-kitchen/actions/runs/38207087682 (initially queued). - Owner can temporarily stack **#242 + #244 + #246** via Patcher and rebuild transactionally. - Confirm `DT_NEEDED: libcudart.so.13` and 3 iterations of `--operation quantize` followed by `--operation sol` using the existing script. - If the shared-runtime build cannot import or the exact failure persists, document and remove **only #246**, restoring the prior two-PR stack; **do not infer linkage was causal**. - NVIDIA documentation explicitly permits coexistence of multiple static CUDA runtimes; this is an experimental A/B, **not** a proposed default or upstream-ready fix. Dynamic-runtime dependency bundling/loader compatibility also requires validation before merge. This leaves numerical draft #242 and build-system draft #244 unchanged.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Root A/B invalidated by ELF link evidence — CUDA 11 selected despite CUDA 13 NVCC (2026-10-12)

Owner installed experimental draft PR #246 alongside numerical #242 and build #244, then ran its manual quantize/Sol probes. Both reported:

Torch 2.10.0+cu130 / RTX PRO 6000 Blackwell
ELF DT_NEEDED: libcudart.so.11.0
Mapped: /usr/lib/x86_64-linux-gnu/libcudart.so.11.5.117
Also loaded: .../nvidia/cu13/lib/libcudart.so.13, .../nvidia/cuda_runtime/lib/libcudart.so.12
Native sync: PASS
PyTorch allocate: PASS
PyTorch fill: cudaErrorInvalidResourceHandle

This is a concrete compiler/runtime selection mismatch, not a successful matched-CUDA13 shared-runtime experiment. The failure does not falsify matched shared linkage, and a shared-runtime fix must never be qualified with the CUDA 11 dependency present. CMake's CUDA::cudart target resolved to the wrong library, potentially via its cached library search; exact selection mechanism still needs proof. Linkage code never altered the numerical CUDA kernels.

Draft #246 is updated to one-commit cd624cfafe28f6ae526f670967dc3117154a85b0. On Linux it explicitly links libcudart.so.<CUDAToolkit_VERSION_MAJOR> only from the selected CUDAToolkit_ROOT library paths, otherwise configuration fails. It logs CUDA::cudart imported location and selected pinned path, and tests now reject a DT_NEEDED CUDART major mismatch before running the GPU repro. Windows static linkage and all numerical sources remain unchanged. CI run 38207788536 is building the revised code.

Owner next step after CI: refresh only temporary #246 overlay in Patcher, rebuild, and first inspect the new DT_NEEDED. Only if libcudart.so.13 and no old cudart are needed by the extension should the quantize/sol three-iteration A/B be interpreted. No H3 full run/merge/release while the common CUDA interop problem persists.

@xmarre commented on GitHub (Oct 12, 2026): ## Root A/B invalidated by ELF link evidence — CUDA 11 selected despite CUDA 13 NVCC (2026-10-12) Owner installed experimental [draft PR #246](https://github.com/Comfy-Org/comfy-kitchen/pull/246) alongside numerical #242 and build #244, then ran its manual quantize/Sol probes. Both reported: ``` Torch 2.10.0+cu130 / RTX PRO 6000 Blackwell ELF DT_NEEDED: libcudart.so.11.0 Mapped: /usr/lib/x86_64-linux-gnu/libcudart.so.11.5.117 Also loaded: .../nvidia/cu13/lib/libcudart.so.13, .../nvidia/cuda_runtime/lib/libcudart.so.12 Native sync: PASS PyTorch allocate: PASS PyTorch fill: cudaErrorInvalidResourceHandle ``` This is a **concrete compiler/runtime selection mismatch**, not a successful matched-CUDA13 shared-runtime experiment. The failure does **not** falsify matched shared linkage, and a shared-runtime fix must never be qualified with the CUDA 11 dependency present. CMake's `CUDA::cudart` target resolved to the wrong library, potentially via its cached library search; exact selection mechanism still needs proof. Linkage code never altered the numerical CUDA kernels. Draft #246 is updated to **one-commit `cd624cfafe28f6ae526f670967dc3117154a85b0`**. On Linux it explicitly links `libcudart.so.<CUDAToolkit_VERSION_MAJOR>` only from the **selected `CUDAToolkit_ROOT`** library paths, otherwise configuration fails. It logs `CUDA::cudart` imported location and selected pinned path, and tests now reject a `DT_NEEDED` CUDART major mismatch *before* running the GPU repro. Windows static linkage and all numerical sources remain unchanged. [CI run 38207788536](https://github.com/xmarre/comfy-kitchen/actions/runs/38207788536) is building the revised code. **Owner next step after CI:** refresh only temporary #246 overlay in Patcher, rebuild, and first inspect the new `DT_NEEDED`. Only if `libcudart.so.13` and no old cudart are needed by the extension should the quantize/sol three-iteration A/B be interpreted. No H3 full run/merge/release while the common CUDA interop problem persists.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Matched CUDA 13 runtime: repeat-call repro now passes (Blackwell, 2026-10-12)

The owner refreshed diagnostic PR #246 to head cd624cfafe28f6ae526f670967dc3117154a85b0 and rebuilt the same Kitchen overlay stack (#242 numerical, #244 build, #246 shared-runtime A/B). Confirmed installed _C.abi3.so ELF DT_NEEDED: libcudart.so.13 on PyTorch 2.10.0+cu130, RTX PRO 6000 Blackwell WSL. Loaded shared CUDA runtimes include versions 13 and 12, but not the host's CUDA 11.5.

With CUDA_LAUNCH_BLOCKING=1 and an explicit native-sync → PyTorch allocation → fill_ → sync sequence, both non-Sol Kitchen quantize_int8_rowwise 3/3 PASS and plain Sol-Attn token_aug=0 3/3 PASS. Under the old static-runtime build, both failed after first call at the PyTorch fill_ probe. The first shared-runtime test (prior head #246) had actually linked libcudart.so.11.0 and failed: it was an invalid matched-CUDA13 A/B.

Inference: strong causal support for native/PyTorch CUDA runtime-linkage compatibility as the class of problem; does not prove static linkage is intrinsically wrong, because the static binary's internal CUDART version has not been established. In particular, coexistence of libcudart.so.12 with 13 is still observed and harmless in this six-call test.

Disposition: low-level interop blocker cleared for this three-iteration WSL test environment only. Keep #246 DRAFT pending packaging audit: absolute INSTALL_RPATH to local Conda CUDA install is unsuitable for general-purpose redistribution; cross-machine wheel audit, Windows behavior, selected CUDA toolkit detection and loader interoperability must be qualified separately before proposing upstream merge. Now proceed with original #242 CUDA overflow numerical suite. Do not merge/release any PR yet.

@xmarre commented on GitHub (Oct 12, 2026): ## Matched CUDA 13 runtime: **repeat-call repro now passes** (Blackwell, 2026-10-12) The owner refreshed [diagnostic PR #246](https://github.com/Comfy-Org/comfy-kitchen/pull/246) to head `cd624cfafe28f6ae526f670967dc3117154a85b0` and rebuilt the same Kitchen overlay stack (#242 numerical, #244 build, #246 shared-runtime A/B). Confirmed installed `_C.abi3.so` ELF `DT_NEEDED: libcudart.so.13` on PyTorch `2.10.0+cu130`, RTX PRO 6000 Blackwell WSL. Loaded shared CUDA runtimes include versions **13 and 12**, but **not** the host's CUDA 11.5. With `CUDA_LAUNCH_BLOCKING=1` and an explicit native-sync → PyTorch allocation → `fill_` → sync sequence, **both** non-Sol Kitchen `quantize_int8_rowwise` **3/3 PASS** and plain Sol-Attn `token_aug=0` **3/3 PASS**. Under the old static-runtime build, both failed after first call at the PyTorch `fill_` probe. The first shared-runtime test (prior head #246) had actually linked `libcudart.so.11.0` and failed: it was an invalid matched-CUDA13 A/B. **Inference:** strong causal support for native/PyTorch CUDA runtime-linkage *compatibility* as the class of problem; does **not** prove static linkage is intrinsically wrong, because the static binary's internal CUDART version has not been established. In particular, coexistence of libcudart.so.12 with 13 is still observed and harmless in this six-call test. **Disposition:** low-level interop blocker cleared **for this three-iteration WSL test environment only**. Keep #246 DRAFT pending packaging audit: absolute `INSTALL_RPATH` to local Conda CUDA install is unsuitable for general-purpose redistribution; cross-machine wheel audit, Windows behavior, selected CUDA toolkit detection and loader interoperability must be qualified separately before proposing upstream merge. Now proceed with original #242 CUDA overflow numerical suite. **Do not merge/release any PR yet.**

bsakaguchi commented

2026-10-26 22:19:57 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Status: the corrected matched-CUDA13 experimental linkage (#246 head cd624cfa) passed 3/3 repeated non-Sol and 3/3 repeated plain Sol interop smoke checks on the owner's Blackwell, and now 7/7 actual GPU numerical tests in the stack (#242+#244+#246). This confirms the previously reproducible PyTorch invalid resource-handle blocker does not appear in these tests with matched shared CUDART 13. Keep #245 open for runtime selection/linkage provenance and portable wheel fix, #246 draft; do not infer dynamic CUDART is universally necessary or that static CUDART is intrinsically unsafe.

@xmarre commented on GitHub (Oct 12, 2026): Status: the corrected matched-CUDA13 experimental linkage (#246 head cd624cfa) passed 3/3 repeated non-Sol and 3/3 repeated plain Sol interop smoke checks on the owner's Blackwell, and now 7/7 actual GPU numerical tests in the stack (#242+#244+#246). This confirms the previously reproducible PyTorch invalid resource-handle blocker does **not** appear in these tests with matched shared CUDART 13. Keep #245 open for runtime selection/linkage provenance and portable wheel fix, #246 draft; do not infer dynamic CUDART is universally necessary or that static CUDART is intrinsically unsafe.

bsakaguchi commented

2026-10-26 22:19:57 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Review-ready PR cluster / issue #245 status (2026-10-12)

The owner had cudaErrorInvalidResourceHandle after both Sol and unrelated quantization kernels under mixed runtime linkage, but achieved 3/3 repeated non-Sol + 3/3 repeated plain Sol interop PASS, 7/7 numerical CUDA tests, and a full deterministic H3 01907 continuation with matched shared CUDA13. The Linux shared-runtime source fix is now review-ready in PR #246 at 6803e41b; it explicitly selects the matching toolkit CUDART and replaces the previous absolute CUDA toolkit RPATH with $ORIGIN, and CI now checks actual ELF dependencies in both Linux wheel architectures. Separate #244 (0e2d78cd) now invalidates stale CUDA_cudart_LIBRARY/toolkit root cache entries; Python-focused 9/9 PASS. Both PRs remain unmerged. The new relocatable #246 revision needs CI/wheel-ELF verification and post-rebuild owner GPU interop check before release. Do not close this issue based only on PR review readiness.

@xmarre commented on GitHub (Oct 12, 2026): ### Review-ready PR cluster / issue #245 status (2026-10-12) The owner had `cudaErrorInvalidResourceHandle` after both Sol and unrelated quantization kernels under mixed runtime linkage, but achieved **3/3 repeated non-Sol + 3/3 repeated plain Sol interop PASS**, **7/7 numerical CUDA tests**, and a full deterministic H3 01907 continuation with matched shared CUDA13. The Linux shared-runtime source fix is now review-ready in [PR #246](https://github.com/Comfy-Org/comfy-kitchen/pull/246) at `6803e41b`; it explicitly selects the matching toolkit CUDART and replaces the previous absolute CUDA toolkit RPATH with `$ORIGIN`, and CI now checks actual ELF dependencies in both Linux wheel architectures. Separate [#244](https://github.com/Comfy-Org/comfy-kitchen/pull/244) (`0e2d78cd`) now invalidates stale `CUDA_cudart_LIBRARY`/toolkit root cache entries; Python-focused 9/9 PASS. Both PRs remain unmerged. **The new relocatable #246 revision needs CI/wheel-ELF verification and post-rebuild owner GPU interop check before release**. Do not close this issue based only on PR review readiness.

bsakaguchi referenced this issue

2026-10-26 22:20:33 +00:00

[PR #67] [MERGED] Fix GitHub Actions CLA workflow for commit authors #119