Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #83] [ROCm/Windows] vbar_allocate clamps every VBAR reservation to vram_capacity, exhausting the GPU address space after three models #26

New issue

Open

opened 2026-10-26 22:23:47 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:23:47 +00:00

Owner

Originally created by @doesdev on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/83

Summary

vbar_allocate clamps each VBAR's VA reservation to vram_capacity, so on a large-VRAM device every model reserves the full device capacity worth of address space regardless of its actual size. Three models exhaust the ~256 GiB per-process GPU VA space available on ROCm/Windows, after which allocations fail with hipErrorOutOfMemory while tens of GB are still free.

Changing one multiplier in ComfyUI takes peak usage on a 43 GiB LTX workflow from 26,978 MB to 81,522 MB and turns a hard failure into a clean run.

Environment


GPU	AMD Radeon 8060S (gfx1151, Strix Halo iGPU), 96 GB BIOS VGM carve-out
CPU	AMD Ryzen AI Max+ 395
OS	Windows 11 Pro 10.0.26200, driver 32.0.31035.1003
torch	2.12.0+rocm7.14.0, ROCm 7.14
ComfyUI	0.30.0, comfy-aimdo 0.4.13, comfy-kitchen 0.2.26
Args	`--enable-dynamic-vram --disable-async-offload --enable-manager`
Workload	43.0 GiB LTX checkpoint: 40050 MB diffusion + 11200 MB TE + 1384 MB VAE

Mechanism

src/model-vbar.c:222-227:

size_t nr_pages = VBAR_GET_PAGE_NR_UP(size);
size_t nr_pages_max = VBAR_GET_PAGE_NR(vram_capacity);
if (nr_pages_max < nr_pages) {
    nr_pages = nr_pages_max;
}

ComfyUI calls ModelVBAR(self.model_size() * 10, ...) from comfy/model_patcher.py _vbar_get. With vram_capacity = 110,456 MB the clamp is 3451 pages = 110,432 MB, so any model above roughly a tenth of device capacity reserves 110,432 MB of VA irrespective of size:

VBAR	`model_size() * 10`	reserved after clamp
VideoVAE	13,840 MB	13,856 MB
LTXAVTEModel_	112,000 MB	110,432 MB
LTXAV	400,500 MB	110,432 MB
		234,720 MB = 229.2 GiB

On ROCm/Windows the per-process GPU VA space is ~256 GiB and physical VMM allocations draw from the same budget. That is measured separately with a standalone reproducer and reported as ROCm/rocm-systems#9823. 256 GiB minus 229.2 GiB leaves about 27 GiB for anything to actually be mapped into.

Observed

Two runs, aimdo DEBUG level, differing only in --disable-pinned-memory:

	pinned on	pinned off
`Aimdo Recorded Usage` at failure	26,978 MB	26,786 MB
`Device free` at failure	60,997 MB	82,693 MB
`Total VRAM for VBARs`	26,336 MB	26,144 MB

Free memory differs by 21,696 MB; the failure point differs by 192 MB, which is six 32 MB pages. The ceiling does not track free memory.

budget_deficit() never returns positive in either run (no Deficit: line anywhere in the DEBUG logs), so aimdo is not throttling. The budget_deficit(VBAR_PAGE_SIZE) > 0 short-circuit at src/model-vbar.c:390 does not fire, three_stooges() is actually called, and it returns CUDA_ERROR_OUT_OF_MEMORY for a 32 MB allocation with 60-82 GB free.

Symptoms this produces, in order of appearance in one session:

CUDA error: out of memory at the first sampler step, at a consistent ~27 GB regardless of resolution or workload size.
With a LoRA attached, 432 ERROR lora <weight> CUDA error: out of memory lines, after which the run completes and reports success having silently dropped those patches. This is the worst of the three, because the output looks plausible.
OSError: exception: access violation reading 0x00000000000000E0 from model_vbar.py:52, when a later node (LTXVLatentUpsampler) requests a fourth reservation and hipMemAddressReserve faults instead of returning an error. Same crash as #63 and #45.

Fix that works

comfy/model_patcher.py, _vbar_get: self.model_size() * 10 to self.model_size() * 2.

	`* 10`	`* 2`
total VA reserved	229.2 GiB	102.8 GiB
peak `Aimdo Recorded Usage`	26,978 MB	81,522 MB
peak `Total VRAM for VBARs`	26,336 MB	52,864 MB
LoRA patch errors	432	0
aimdo allocator failures	1	0
`vbars_analyze` warnings	n/a	0
outcome	OOM at step 1, then 0xE0	completes, 256.29 s

81,522 MB compares with 87.2 GB peak for the identical workflow on Fedora 44 on this same machine.

* 2 is not proposed as the fix, only as evidence. The upstream comment on that line says the 10x exists to cover a full FP4 to FP32 upcast, which * 2 does not. Two directions that would preserve the intent:

Budget VA across all live VBARs rather than per-VBAR, so total reservation is bounded by something the platform can supply.
Have vbar_allocate report the clamped size back to Python instead of clamping silently, so the caller can react.

Note on the clamp and `max_size`

vbar_allocate clamps the reservation but comfy_aimdo/model_vbar.py:56 stores the caller's unclamped size as self.max_size, and alloc() bounds-checks against that. When the clamp engages, Python believes it owns more address space than was reserved. It does not trigger here because offsets stay well under the clamped size, and it disappears at * 2 since nothing clamps. Mentioning it only because it is adjacent to the same code; happy to open it separately if preferred.

#63, #45: the 0xE0 crash on RDNA4 and gfx1100. Symptom 3 above. Root cause of the crash itself is ROCm/rocm-systems#6051 (closed, unmerged).
ROCm/rocm-systems#6191: VA state corruption across reserve/free cycles on Windows, open, AMD-reproduced.
PR #67 added a VA pool for VramBuffer on ROCm/Windows to avoid the same fault. model-vbar.c has no equivalent, so each model load still reserves and frees its own window. Reducing the per-VBAR size addresses more of the problem than pooling does, but the two compose.

Originally created by @doesdev on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/83 ### Summary `vbar_allocate` clamps each VBAR's VA reservation to `vram_capacity`, so on a large-VRAM device every model reserves the full device capacity worth of address space regardless of its actual size. Three models exhaust the ~256 GiB per-process GPU VA space available on ROCm/Windows, after which allocations fail with `hipErrorOutOfMemory` while tens of GB are still free. Changing one multiplier in ComfyUI takes peak usage on a 43 GiB LTX workflow from 26,978 MB to 81,522 MB and turns a hard failure into a clean run. ### Environment | | | |---|---| | GPU | AMD Radeon 8060S (gfx1151, Strix Halo iGPU), 96 GB BIOS VGM carve-out | | CPU | AMD Ryzen AI Max+ 395 | | OS | Windows 11 Pro 10.0.26200, driver 32.0.31035.1003 | | torch | 2.12.0+rocm7.14.0, ROCm 7.14 | | ComfyUI | 0.30.0, comfy-aimdo 0.4.13, comfy-kitchen 0.2.26 | | Args | `--enable-dynamic-vram --disable-async-offload --enable-manager` | | Workload | 43.0 GiB LTX checkpoint: 40050 MB diffusion + 11200 MB TE + 1384 MB VAE | ### Mechanism `src/model-vbar.c:222-227`: ```c size_t nr_pages = VBAR_GET_PAGE_NR_UP(size); size_t nr_pages_max = VBAR_GET_PAGE_NR(vram_capacity); if (nr_pages_max < nr_pages) { nr_pages = nr_pages_max; } ``` ComfyUI calls `ModelVBAR(self.model_size() * 10, ...)` from `comfy/model_patcher.py` `_vbar_get`. With `vram_capacity` = 110,456 MB the clamp is 3451 pages = 110,432 MB, so any model above roughly a tenth of device capacity reserves 110,432 MB of VA irrespective of size: | VBAR | `model_size() * 10` | reserved after clamp | |---|---|---| | VideoVAE | 13,840 MB | 13,856 MB | | LTXAVTEModel_ | 112,000 MB | **110,432 MB** | | LTXAV | 400,500 MB | **110,432 MB** | | | | **234,720 MB = 229.2 GiB** | On ROCm/Windows the per-process GPU VA space is ~256 GiB and physical VMM allocations draw from the same budget. That is measured separately with a standalone reproducer and reported as ROCm/rocm-systems#9823. 256 GiB minus 229.2 GiB leaves about 27 GiB for anything to actually be mapped into. ### Observed Two runs, aimdo DEBUG level, differing only in `--disable-pinned-memory`: | | pinned on | pinned off | |---|---|---| | `Aimdo Recorded Usage` at failure | 26,978 MB | 26,786 MB | | `Device free` at failure | 60,997 MB | 82,693 MB | | `Total VRAM for VBARs` | 26,336 MB | 26,144 MB | Free memory differs by 21,696 MB; the failure point differs by 192 MB, which is six 32 MB pages. The ceiling does not track free memory. `budget_deficit()` never returns positive in either run (no `Deficit:` line anywhere in the DEBUG logs), so aimdo is not throttling. The `budget_deficit(VBAR_PAGE_SIZE) > 0` short-circuit at `src/model-vbar.c:390` does not fire, `three_stooges()` is actually called, and it returns `CUDA_ERROR_OUT_OF_MEMORY` for a 32 MB allocation with 60-82 GB free. Symptoms this produces, in order of appearance in one session: 1. `CUDA error: out of memory` at the first sampler step, at a consistent ~27 GB regardless of resolution or workload size. 2. With a LoRA attached, 432 `ERROR lora <weight> CUDA error: out of memory` lines, after which **the run completes and reports success** having silently dropped those patches. This is the worst of the three, because the output looks plausible. 3. `OSError: exception: access violation reading 0x00000000000000E0` from `model_vbar.py:52`, when a later node (`LTXVLatentUpsampler`) requests a fourth reservation and `hipMemAddressReserve` faults instead of returning an error. Same crash as #63 and #45. ### Fix that works `comfy/model_patcher.py`, `_vbar_get`: `self.model_size() * 10` to `self.model_size() * 2`. | | `* 10` | `* 2` | |---|---|---| | total VA reserved | 229.2 GiB | 102.8 GiB | | peak `Aimdo Recorded Usage` | 26,978 MB | **81,522 MB** | | peak `Total VRAM for VBARs` | 26,336 MB | **52,864 MB** | | LoRA patch errors | 432 | **0** | | aimdo allocator failures | 1 | **0** | | `vbars_analyze` warnings | n/a | **0** | | outcome | OOM at step 1, then 0xE0 | completes, 256.29 s | 81,522 MB compares with 87.2 GB peak for the identical workflow on Fedora 44 on this same machine. `* 2` is not proposed as the fix, only as evidence. The upstream comment on that line says the 10x exists to cover a full FP4 to FP32 upcast, which `* 2` does not. Two directions that would preserve the intent: - Budget VA across all live VBARs rather than per-VBAR, so total reservation is bounded by something the platform can supply. - Have `vbar_allocate` report the clamped size back to Python instead of clamping silently, so the caller can react. ### Note on the clamp and `max_size` `vbar_allocate` clamps the reservation but `comfy_aimdo/model_vbar.py:56` stores the caller's unclamped `size` as `self.max_size`, and `alloc()` bounds-checks against that. When the clamp engages, Python believes it owns more address space than was reserved. It does not trigger here because offsets stay well under the clamped size, and it disappears at `* 2` since nothing clamps. Mentioning it only because it is adjacent to the same code; happy to open it separately if preferred. ### Related - #63, #45: the 0xE0 crash on RDNA4 and gfx1100. Symptom 3 above. Root cause of the crash itself is ROCm/rocm-systems#6051 (closed, unmerged). - ROCm/rocm-systems#6191: VA state corruption across reserve/free cycles on Windows, open, AMD-reproduced. - PR #67 added a VA pool for `VramBuffer` on ROCm/Windows to avoid the same fault. `model-vbar.c` has no equivalent, so each model load still reserves and frees its own window. Reducing the per-VBAR size addresses more of the problem than pooling does, but the two compose.

bsakaguchi commented

2026-10-26 22:23:48 +00:00

Author

Owner

@coderabbitai[bot] commented on GitHub (Aug 7, 2026):

🔗 Related PRs

Comfy-Org/comfy-aimdo#62 - Dev/double sync [closed]
Comfy-Org/comfy-aimdo#72 - Added support for MCDM and GPU-PV through the DXCore API [open]
Comfy-Org/comfy-aimdo#78 - Implement NVML pressure and async pre-free clamping [merged]
Comfy-Org/comfy-aimdo#80 - Refs/heads/dev/threaded loaded errors [merged]

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

@coderabbitai[bot] commented on GitHub (Aug 7, 2026):  <details> <summary>🔗 Related PRs</summary> Comfy-Org/comfy-aimdo#62 - Dev/double sync [closed] Comfy-Org/comfy-aimdo#72 - Added support for MCDM and GPU-PV through the DXCore API [open] Comfy-Org/comfy-aimdo#78 - Implement NVML pressure and async pre-free clamping [merged] Comfy-Org/comfy-aimdo#80 - Refs/heads/dev/threaded loaded errors [merged] </details> --- <details> <summary> 🧪 Issue enrichment is currently in open beta.</summary> You can configure auto-planning by selecting labels in the issue_enrichment configuration. To disable automatic issue enrichment, add the following to your `.coderabbit.yaml`: ```yaml issue_enrichment: auto_enrich: enabled: false ``` </details> 💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit)!

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#26

No description provided.

Rows
Columns