[GH-ISSUE #241] CUDA sol_attn token_aug overflow: atomic reservations select nondeterministic retained IDs despite histogram parity #69

New issue

Open

opened 2026-10-26 22:19:55 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:19:55 +00:00

Owner

Originally created by @xmarre on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/241

Summary

CUDA Sol-Attn token augmentation has a reproducible real-model over-budget reservation case on the installed comfy-kitchen==0.2.37 H3 chunked-producer path. Two frozen A/A′ replays have identical reservation-count and score-histogram SHA256 digests at one attention block but different retained token-ID digests. The CUDA implementation sorts the IDs it retained, but its capped list is populated using an atomicAdd reservation across independent split CTAs. Sorting cannot make a scheduling-dependent subset deterministic once more than n_tok slots were requested.

This issue distinguishes confirmed over-budget counts and retained-list changes from the still-unproven attribution of the full downstream MiniMax-H3 visual artifact to this one mechanism.

Real GPU evidence (2026-10-12)

NVIDIA RTX PRO 6000 Blackwell (SM120), PyTorch 2.10.0+cu130, comfy-kitchen==0.2.37, H3 50-block transformer.
Native H3 source-sampling A/B/A′ frozen diagnostic: logical prediction 5, block 45, sol_attn_chunked, t=36445, h=56, token_aug=256, groups=15960.
The observer reads tokCnt, tokHist and the initialized portion only of tokIdx from the installed Core workspace after a verified successful Python wrapper return; it does not replay or monkeypatch Core.
Both A and A′: one verified Python wrapper entry and return each; complete=true; 12 groups over budget, 282 exactly at budget, max reservations = 2218, max top-bin population = 2283, max histogram-window population = 29908.
A and A′ have identical full tokCnt hash f166919c... and identical full tokHist hash 4066ac26..., yet the canonical retained-ID hashes differ: A 658bd3fd... vs A′ 9205548f....
B (different controlled source prefix): 11 overflow groups, max reservations 2110; different histograms as expected.
A/A′ first actual H3 prediction is bitwise identical; later selected native video prediction first differs at logical step 5 (RMS 0.0185534, max 0.90234375), and final source clean suffix also differs (RMS 0.110349). The protected 12 temporal positions in the archived first-difference BF16 tensors are bitwise identical.
Limitations: this evidence does not verify identical effective step-5 QKV operands, pooled state or full model execution history, and therefore does not prove sole causation of the final H3 artifact. It does prove real overflow and different retained lists for otherwise equal aggregate counts/histograms.

Owner evidence/protocol: Flow PR #99. This is a Core comfy-kitchen issue; a Flow-side threshold workaround or attention-backend gate is not proposed.

Source-level failure mechanism

In CUDA sol_attn_token.cu:

The top histogram bin is clamped/open-ended in hist_bin (line ~52). Pass-2's whole-bin threshold walk (lines ~207–223) can exceed n_tok in this case, despite the normal bin-selection budget rule.
At lines ~318–332, split CTAs atomicAdd(&tok_cnt[group], total), populate tok_idx only when slot < n_tok, and put surplus scores back into the approximate tail. The list membership depends on which CTA reserves the first 256 slots.
sol_token_sort_kernel (lines ~380–402) sorts that selected subset, but does not restore the tokens excluded by scheduling. tokCnt and tokHist can be identical while the selected IDs differ.

The existing upstream test_token_aug_surplus_over_the_budget_stays_in_the_tail already explicitly documents 1,952 reservations for a 256-token budget and asserts that the surplus remaining in the approximate tail is critical for accuracy. Thus dropping the entire overflowing highest bin to force zero admitted tokens is not a fidelity-preserving solution and can regress this test.

Requested fix / acceptance criteria

Please make overflow handling deterministic without sacrificing accurate token-vs-tail accounting:

Retain a deterministic, capacity-limited exact-ID subset (e.g. fixed key-tile/split order with deterministic per-split reservation counts and a prefix quota, rather than a cross-CTA timing-dependent global atomic).
Every candidate not actually retained must still contribute to the appropriate approximate/tail accumulator exactly once. Keep the existing sparse/direct/chunked contracts and token_aug API.
Add an SM80/SM120 CUDA regression forcing 1,000+ candidates in the clamped highest bin, asserting bounded retained capacity, identical retained IDs and identical outputs under repeated launches, plus the existing dense-reference fidelity check.
Verify both primed sol_attn_chunked and cold/direct paths, plus ordinary non-overflow performance. Avoid a broad backend fallback or token augmentation disablement.

I can prepare a dedicated fork/mirror CUDA correction once the required accuracy and performance invariants are agreed, but this issue records the real hardware evidence and test contract without prematurely changing numerical semantics.

Originally created by @xmarre on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/241 ## Summary CUDA Sol-Attn token augmentation has a reproducible **real-model over-budget reservation** case on the installed `comfy-kitchen==0.2.37` H3 chunked-producer path. Two frozen A/A′ replays have **identical reservation-count and score-histogram SHA256 digests** at one attention block but **different retained token-ID digests**. The CUDA implementation sorts the IDs it retained, but its capped list is populated using an `atomicAdd` reservation across independent split CTAs. Sorting cannot make a scheduling-dependent *subset* deterministic once more than `n_tok` slots were requested. This issue distinguishes confirmed over-budget counts and retained-list changes from the still-unproven attribution of the full downstream MiniMax-H3 visual artifact to this one mechanism. ## Real GPU evidence (2026-10-12) - NVIDIA RTX PRO 6000 Blackwell (SM120), PyTorch 2.10.0+cu130, `comfy-kitchen==0.2.37`, H3 50-block transformer. - Native H3 source-sampling A/B/A′ frozen diagnostic: logical prediction 5, block 45, `sol_attn_chunked`, `t=36445`, `h=56`, `token_aug=256`, `groups=15960`. - The observer reads `tokCnt`, `tokHist` and the **initialized portion only** of `tokIdx` from the installed Core workspace after a verified successful Python wrapper return; it **does not** replay or monkeypatch Core. - Both **A** and **A′**: one verified Python wrapper entry and return each; `complete=true`; **12 groups over budget**, **282 exactly at budget**, **max reservations = 2218**, **max top-bin population = 2283**, **max histogram-window population = 29908**. - A and A′ have **identical full `tokCnt` hash** `f166919c...` and **identical full `tokHist` hash** `4066ac26...`, yet the **canonical retained-ID** hashes differ: A `658bd3fd...` vs A′ `9205548f...`. - B (different controlled source prefix): 11 overflow groups, max reservations 2110; different histograms as expected. - A/A′ first actual H3 prediction is bitwise identical; later selected native video prediction first differs at logical step 5 (RMS 0.0185534, max 0.90234375), and final source clean suffix also differs (RMS 0.110349). The protected 12 temporal positions in the archived first-difference BF16 tensors are bitwise identical. - **Limitations:** this evidence does not verify identical effective step-5 QKV operands, pooled state or full model execution history, and therefore does not prove sole causation of the final H3 artifact. It does prove real overflow and different retained lists for otherwise equal aggregate counts/histograms. Owner evidence/protocol: [Flow PR #99](https://github.com/xmarre/MiniMax-H3-Flow-Aligned-Regenerate/pull/99). This is a **Core comfy-kitchen** issue; a Flow-side threshold workaround or attention-backend gate is not proposed. ## Source-level failure mechanism In [CUDA `sol_attn_token.cu`](https://github.com/Comfy-Org/comfy-kitchen/blob/main/comfy_kitchen/backends/cuda/sage_attention/sol_attn_token.cu): 1. The top histogram bin is **clamped/open-ended** in `hist_bin` (line ~52). Pass-2's whole-bin threshold walk (lines ~207–223) can exceed `n_tok` in this case, despite the normal bin-selection budget rule. 2. At lines ~318–332, split CTAs `atomicAdd(&tok_cnt[group], total)`, populate `tok_idx` **only when `slot < n_tok`**, and put surplus scores back into the approximate tail. The list membership depends on which CTA reserves the first 256 slots. 3. `sol_token_sort_kernel` (lines ~380–402) sorts that selected subset, but does **not** restore the tokens excluded by scheduling. `tokCnt` and `tokHist` can be identical while the selected IDs differ. The existing upstream [`test_token_aug_surplus_over_the_budget_stays_in_the_tail`](https://github.com/Comfy-Org/comfy-kitchen/blob/main/tests/test_sol_attn.py) **already explicitly documents 1,952 reservations for a 256-token budget** and asserts that the surplus remaining in the approximate tail is critical for accuracy. Thus dropping the entire overflowing highest bin to force zero admitted tokens is **not** a fidelity-preserving solution and can regress this test. ## Requested fix / acceptance criteria Please make overflow handling **deterministic without sacrificing accurate token-vs-tail accounting**: - Retain a deterministic, capacity-limited exact-ID subset (e.g. fixed key-tile/split order with deterministic per-split reservation counts and a prefix quota, rather than a cross-CTA timing-dependent global atomic). - Every candidate not actually retained must still contribute to the appropriate approximate/tail accumulator **exactly once**. Keep the existing sparse/direct/chunked contracts and `token_aug` API. - Add an SM80/SM120 CUDA regression forcing 1,000+ candidates in the clamped highest bin, asserting bounded retained capacity, identical retained IDs **and** identical outputs under repeated launches, plus the existing dense-reference fidelity check. - Verify both primed `sol_attn_chunked` and cold/direct paths, plus ordinary non-overflow performance. Avoid a broad backend fallback or token augmentation disablement. I can prepare a dedicated fork/mirror CUDA correction once the required accuracy and performance invariants are agreed, but this issue records the real hardware evidence and test contract without prematurely changing numerical semantics.

bsakaguchi commented

2026-10-26 22:19:56 +00:00

Author

Owner

@xmarre commented on GitHub (Oct 12, 2026):

Real-model before/after validation: run 01907 fixes observed nondeterministic admission (2026-10-12)

Owner retested the same MiniMax H3 chunk-2 A/B/A′ witness using corrected comfy-kitchen CUDA numerical PR #242 in the Torch 2.10.0+cu130 Blackwell environment, alongside build #244 and diagnostic matched CUDA13 shared runtime #246. Run 01906 previously had 12 overflow groups in A and A′, equal histogram and reservation hashes but unequal retained-token SHA256 and divergent output at low step 5.

In owner run 01907 the actual Core sol_attn_chunked at logical low step 5/block 45 has T=36445, H=56, token_aug=256, 15960 groups. A and A′ each still produce 12 over-budget groups (max reservation count 2282, max top-bin occupancy 2312), but count, histogram and canonical retained-token hash are now exactly equal. The record reports complete=true, entries=1, calls=1 for every arm, and retained_ids_equal=true for A/A′; B's altered prefix correctly differs with 11 over-budget groups. Both first actual low prediction and clean suffix A/A′ are bitwise equal, RMS=0, maximum=0, reproduction_verified=true, and all 103/103 observed low native/solver boundary records match. The owner separately qualified 7/7 CUDA numerical tests (0 skips) with installed extension via pytest --import-mode=importlib from /tmp. The owner-supplied RAR was independently decoded and all 35 tensor + 2 manifest hashes validated.

Conclusion: compelling controlled Blackwell evidence that #242 corrects the actual over-budget admission nondeterminism reproduced in #241 and removes the associated A/A′ H3 low-sampler replay divergence in this run. Remaining unobserved physical model/cache state cannot be declared isolated, and audiovisual frame/tone seam is not proven fixed by one A render. Keep #242 draft until broader GPU/packaging qualification. The separate runtime linkage #246 has a nonportable absolute INSTALL_RPATH, so do not merge it as a release default. Detailed receipts in Flow PR #99.

@xmarre commented on GitHub (Oct 12, 2026): ## Real-model before/after validation: run 01907 fixes observed nondeterministic admission (2026-10-12) Owner retested the same MiniMax H3 chunk-2 A/B/A′ witness using corrected comfy-kitchen CUDA numerical [PR #242](https://github.com/Comfy-Org/comfy-kitchen/pull/242) in the Torch 2.10.0+cu130 Blackwell environment, alongside build #244 and **diagnostic matched CUDA13 shared runtime** #246. Run 01906 previously had 12 overflow groups in A and A′, *equal* histogram and reservation hashes but *unequal* retained-token SHA256 and divergent output at low step 5. In owner run 01907 the actual Core `sol_attn_chunked` at logical low step 5/block 45 has `T=36445, H=56, token_aug=256`, **15960 groups**. A and A′ each still produce **12 over-budget groups** (max reservation count 2282, max top-bin occupancy 2312), but count, histogram **and canonical retained-token hash** are now **exactly equal**. The record reports `complete=true, entries=1, calls=1` for every arm, and `retained_ids_equal=true` for A/A′; B's altered prefix correctly differs with 11 over-budget groups. Both first actual low prediction and clean suffix A/A′ are **bitwise equal**, RMS=0, maximum=0, `reproduction_verified=true`, and all 103/103 observed low native/solver boundary records match. The owner separately qualified **7/7 CUDA numerical tests** (0 skips) with installed extension via pytest `--import-mode=importlib` from `/tmp`. The owner-supplied RAR was independently decoded and all **35 tensor + 2 manifest hashes** validated. **Conclusion:** compelling controlled Blackwell evidence that #242 corrects the actual over-budget admission nondeterminism reproduced in #241 and removes the associated A/A′ H3 low-sampler replay divergence in this run. Remaining unobserved physical model/cache state cannot be declared isolated, and audiovisual frame/tone seam is **not** proven fixed by one A render. Keep #242 draft until broader GPU/packaging qualification. The separate runtime linkage #246 has a nonportable absolute `INSTALL_RPATH`, so do not merge it as a release default. Detailed receipts in [Flow PR #99](https://github.com/xmarre/MiniMax-H3-Flow-Aligned-Regenerate/pull/99#issuecomment-6118742207).

bsakaguchi referenced this issue

2026-10-26 22:20:34 +00:00

[PR #69] RDNA-aware autotune configs for the INT8 matmul kernels #120