Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #117] vbar_fault() resident detection for a VAE degrades irreversibly after a multi-segment continuity generation (26x H2D memcpy blowup, GPU compute unaffected) #44

New issue

Open

opened 2026-10-26 22:24:01 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:24:01 +00:00

Owner

Originally created by @lilam9110 on GitHub (Sep 17, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/117

Summary

vbar_fault()'s residency detection for a VAE model degrades irreversibly (until process restart) after a multi-segment "continuity" generation completes. Once triggered, every subsequent decode on that VAE — regardless of task type — sees its resident-cache hit rate collapse from ~100% to ~0%, causing the same weights to be re-transferred Host→Device on nearly every forward pass. GPU kernel compute time is unaffected; the entire slowdown is H2D memcpy volume. We've root-caused this down to the vbar_fault() boundary (compiled aimdo.so, no source access) across an 11-round investigation and can supply the full evidence trail.

Environment

OS: Ubuntu 22.04.5, Linux 6.8.0
GPU: 2x NVIDIA RTX 4090 D (48GB), CUDA 13.1, driver 610.57.04 (multi-GPU: UNet on cuda:0, CLIP/VAE on cuda:1 via SelectModelDevice/SelectCLIPDevice/SelectVAEDevice)
comfy-aimdo: 0.5.3
ComfyUI: recent master, --use-ck-attention --disable-cuda-malloc, no --cache-ram override (default RAMPressureCache)
Model: MiniMax H3 (custom node pack ComfyUI_MiniMaxH3_Director), VAE = minimax_h3_video_vae_fp16.safetensors, VBAR/DynamicVRAM enabled (default)

Symptom

Same VAE, same resolution, same frame count, in the same long-lived ComfyUI process: after one "continuity" submission (multi-segment generation with a motion-context latent pin carried in-graph across segments) completes, decode time on every following submission — regardless of whether it uses continuity again — jumps from baseline ~15-25s (90-141 frames) / ~43s (243 frames) to 130-285s+, and keeps escalating across subsequent submissions. This persists for the remaining life of the process (observed >12h / dozens of submissions without self-recovery) and only clears on restart.

Reliable repro sequence (confirmed 3x independently, near-identical numbers each time)

2x baseline single-segment generation (any task type) — confirms healthy baseline (~15-21s decode)
3x continuity submission, each internally chaining 2-3 segments via a motion-context latent pin (kept in-graph memory, never round-tripped to disk/pixels between segments) — these themselves decode at flat, healthy per-segment times (no escalation within them)
Immediately after step 2, submit one more generation (any type, continuity or not) — decode time on this submission (and every subsequent one) begins escalating: e.g. 41.534s → 62.919s → 129.623s across 3 internal segments of that single submission, then 207.4s / 284.4s on the next.

Multiple independent repro runs (different days, fresh processes) landed within a few seconds of each other on segment 3 (~129.6-130.9s), strongly suggesting a deterministic trigger, not noise.

Root-cause chain (what we verified, with tooling)

py-spy (8x stack dumps + flamegraph during a 130s degraded decode): 100% of sampled time is in normal decode code paths (TransformerBlock/Attention/rms_norm/ops.py::forward_comfy_cast_weights/cast_modules_with_vbar). No Python-level deadlock, no stuck loop. Bottoms out at comfy_kitchen's _int8_attention_cuda — a compiled extension py-spy can't see into.
nsys (kernel-level comparison, healthy vs degraded window, same process): GPU compute kernel time is essentially unchanged (230.838s vs 230.334s total, ratio 0.998). But [CUDA memcpy Host-to-Device]:
- call count: 8,993 → 77,184 (8.58x)
- total bytes: 64.4 GB → 1.70 TB (26.35x)
- total time: 10.85s → 259.71s (23.9x)
- Degraded window uniquely shows 474x cudaHostRegister + 98x non-trivial cudaMalloc calls (0 in healthy window) — i.e. fresh host-side buffer allocation/registration, not reuse.

Static read of comfy/ops.py::cast_modules_with_vbar: the resident check is

signature = comfy_aimdo.model_vbar.vbar_fault(s._v)
resident = comfy_aimdo.model_vbar.vbar_signature_compare(signature, s._v_signature)
if resident:
    continue  # zero-copy fast path
# else: materialize_meta_param -> host->device copy (this is the H2D memcpy nsys caught)

xfer_source in the slow path is always the module's own [weight, bias] (or its pin) — confirmed no extraneous data is being transferred; it's the same weights being re-transferred on every fault.

Debug-probe instrumentation (temporary logging around the resident check, isolated worktree): cuda:1 VAE resident hit rate is 100% (5874/5874) in a clean/healthy window, then collapses monotonically across the degraded submission's 3 internal segments: 40.5% → 25.4% → 0.2%, tracking almost perfectly with the decode-time escalation (41.7s / 63.0s / 129.6s) in the same run.
Ruled out (each with a controlled A/B on the exact same repro sequence, real numbers, not just code reading):
- comfy_extras/nodes_multigpu.py::_retarget_patcher repeatedly reassigning offload_device — UNet goes through the identical code path every call and never degrades; also confirmed offload_device is discarded (ModelPatcherDynamic.unpatch_model hardcodes device_to=None) and never read by the native layer.
- Any Python-side pin/eviction bookkeeping accumulating over calls (comfy/model_patcher.py:2072-2122's "AIMDO free" logging never fires once across the whole trigger+degrade window — 0 hits in a 341k-line DEBUG log).
- nvml_pressure (--disable-nvml-pressure): degradation still reproduces at the same magnitude.
- --disable-pinned-memory (tried based on a similar-looking closed issue, #104/comfy-org/ComfyUI#16062): degradation still reproduces, if anything slightly worse (52.5/77.3/162.2s vs historical 41.5-52.5/62.9-77.3/129.6-162.2s) — this tells us our case is a different mechanism from #104's pinned-memory-pool sizing issue, even though the surface symptom (VAE-decode-stage slowdown that gets worse on repeat) looks similar.
- comfy/model_management.py:1386's process-wide LARGEST_AIMDO_CASTED_WEIGHT (not device-scoped, read/written cross-device by ops.py:143/146-147) — plausible cross-device-interference candidate, but disproved by code flow: ensure_offload_stream (the only reader/writer) is only ever called after resident has already been determined False, so it structurally cannot be the cause of the resident check itself failing. Confirmed with the same debug probe: resident hit-rate collapse is identical regardless of whether this global gets clobbered by the other GPU's UNet.

Where we're stuck

vbar_fault() itself (.venv/…/comfy_aimdo/model_vbar.py → lib.vbar_fault, a ctypes call into the compiled aimdo.so/aimdo_rocm.so) is where the actual "is this signature still resident" decision is made. We have no source access to that boundary. We also tried --verbose DEBUG hoping the native set_log_callback hook would surface fault/evict diagnostics (per _LOG_LEVELS mapping native levels 5-7 → logging.DEBUG) — across the full trigger+degrade window we got exactly two one-time init messages and nothing else; no fault/evict/resident diagnostic logging exists at any verbosity we could reach.

What we haven't isolated yet

Whether "continuity" is strictly necessary to trigger this, or whether any sufficiently memory-pressuring submission sequence would do it (continuity pushes the UNet's GPU to ~46.5-46.9GB / 48GB cap during the pin-holding segments; we haven't run a non-continuity high-pressure control).
We also tried an experimental "one segment per independent ComfyUI submission, motion-context state round-tripped through disk between them" variant (mimicking the pattern in NikoDemon80/ComfyUI-H3-Motion-Context) hoping to avoid whatever continuity does — it did not avoid the degradation (decode escalated 76→100.7→129.6s across 3 fresh independent submissions, landing in the same historical range). So the trigger isn't specific to "many segments in one submission" either.

What we can provide on request

Full evidence trail — 11(+1) rounds of logs, nsys .nsys-rep files + CLI stats exports, py-spy dumps/flamegraph, debug-probe output, GPU/host-memory traces — is organized and citable by round number. Happy to share the relevant subset (or all of it) if useful for triage. This report intentionally leaves out an unrelated LARGEST_AIMDO_CASTED_WEIGHT cross-device-scoping cleanup we noticed along the way (confirmed harmless to this bug, comfy/model_management.py:1386 isn't device-keyed) — mentioning only in case it's an easy separate fix.

Originally created by @lilam9110 on GitHub (Sep 17, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/117 ## Summary `vbar_fault()`'s residency detection for a VAE model degrades irreversibly (until process restart) after a multi-segment "continuity" generation completes. Once triggered, **every subsequent decode on that VAE** — regardless of task type — sees its resident-cache hit rate collapse from ~100% to ~0%, causing the same weights to be re-transferred Host→Device on nearly every forward pass. GPU kernel compute time is unaffected; the entire slowdown is H2D memcpy volume. We've root-caused this down to the `vbar_fault()` boundary (compiled `aimdo.so`, no source access) across an 11-round investigation and can supply the full evidence trail. ## Environment - OS: Ubuntu 22.04.5, Linux 6.8.0 - GPU: 2x NVIDIA RTX 4090 D (48GB), CUDA 13.1, driver 610.57.04 (multi-GPU: UNet on cuda:0, CLIP/VAE on cuda:1 via `SelectModelDevice`/`SelectCLIPDevice`/`SelectVAEDevice`) - comfy-aimdo: 0.5.3 - ComfyUI: recent master, `--use-ck-attention --disable-cuda-malloc`, no `--cache-ram` override (default `RAMPressureCache`) - Model: MiniMax H3 (custom node pack `ComfyUI_MiniMaxH3_Director`), VAE = `minimax_h3_video_vae_fp16.safetensors`, VBAR/DynamicVRAM enabled (default) ## Symptom Same VAE, same resolution, same frame count, **in the same long-lived ComfyUI process**: after one "continuity" submission (multi-segment generation with a motion-context latent pin carried in-graph across segments) completes, decode time on **every following submission** — regardless of whether it uses continuity again — jumps from baseline ~15-25s (90-141 frames) / ~43s (243 frames) to 130-285s+, and keeps escalating across subsequent submissions. This persists for the remaining life of the process (observed >12h / dozens of submissions without self-recovery) and only clears on restart. ## Reliable repro sequence (confirmed 3x independently, near-identical numbers each time) 1. 2x baseline single-segment generation (any task type) — confirms healthy baseline (~15-21s decode) 2. 3x continuity submission, each internally chaining 2-3 segments via a motion-context latent pin (kept in-graph memory, never round-tripped to disk/pixels between segments) — these themselves decode at flat, healthy per-segment times (no escalation within them) 3. Immediately after step 2, submit one more generation (any type, continuity or not) — decode time on this submission (and every subsequent one) begins escalating: e.g. `41.534s → 62.919s → 129.623s` across 3 internal segments of that single submission, then `207.4s / 284.4s` on the next. Multiple independent repro runs (different days, fresh processes) landed within a few seconds of each other on segment 3 (~129.6-130.9s), strongly suggesting a deterministic trigger, not noise. ## Root-cause chain (what we verified, with tooling) 1. **py-spy** (8x stack dumps + flamegraph during a 130s degraded decode): 100% of sampled time is in normal decode code paths (`TransformerBlock`/`Attention`/`rms_norm`/`ops.py::forward_comfy_cast_weights`/`cast_modules_with_vbar`). No Python-level deadlock, no stuck loop. Bottoms out at `comfy_kitchen`'s `_int8_attention_cuda` — a compiled extension py-spy can't see into. 2. **nsys** (kernel-level comparison, healthy vs degraded window, same process): GPU compute kernel time is essentially unchanged (230.838s vs 230.334s total, ratio 0.998). But `[CUDA memcpy Host-to-Device]`: - call count: 8,993 → 77,184 (**8.58x**) - total bytes: 64.4 GB → 1.70 TB (**26.35x**) - total time: 10.85s → 259.71s (**23.9x**) - Degraded window uniquely shows 474x `cudaHostRegister` + 98x non-trivial `cudaMalloc` calls (0 in healthy window) — i.e. fresh host-side buffer allocation/registration, not reuse. 3. **Static read of `comfy/ops.py::cast_modules_with_vbar`**: the resident check is ```python signature = comfy_aimdo.model_vbar.vbar_fault(s._v) resident = comfy_aimdo.model_vbar.vbar_signature_compare(signature, s._v_signature) if resident: continue # zero-copy fast path # else: materialize_meta_param -> host->device copy (this is the H2D memcpy nsys caught) ``` `xfer_source` in the slow path is always the module's own `[weight, bias]` (or its pin) — confirmed no extraneous data is being transferred; it's the **same weights being re-transferred on every fault**. 4. **Debug-probe instrumentation** (temporary logging around the resident check, isolated worktree): `cuda:1` VAE resident hit rate is 100% (5874/5874) in a clean/healthy window, then collapses monotonically across the degraded submission's 3 internal segments: **40.5% → 25.4% → 0.2%**, tracking almost perfectly with the decode-time escalation (41.7s / 63.0s / 129.6s) in the same run. 5. Ruled out (each with a controlled A/B on the exact same repro sequence, real numbers, not just code reading): - `comfy_extras/nodes_multigpu.py::_retarget_patcher` repeatedly reassigning `offload_device` — UNet goes through the identical code path every call and never degrades; also confirmed `offload_device` is discarded (`ModelPatcherDynamic.unpatch_model` hardcodes `device_to=None`) and never read by the native layer. - Any Python-side pin/eviction bookkeeping accumulating over calls (`comfy/model_patcher.py:2072-2122`'s "AIMDO free" logging never fires once across the whole trigger+degrade window — 0 hits in a 341k-line DEBUG log). - `nvml_pressure` (`--disable-nvml-pressure`): degradation still reproduces at the same magnitude. - `--disable-pinned-memory` (tried based on a similar-looking closed issue, #104/comfy-org/ComfyUI#16062): degradation still reproduces, if anything slightly worse (52.5/77.3/162.2s vs historical 41.5-52.5/62.9-77.3/129.6-162.2s) — this tells us our case is a *different* mechanism from #104's pinned-memory-pool sizing issue, even though the surface symptom (VAE-decode-stage slowdown that gets worse on repeat) looks similar. - `comfy/model_management.py:1386`'s process-wide `LARGEST_AIMDO_CASTED_WEIGHT` (not device-scoped, read/written cross-device by `ops.py:143/146-147`) — plausible cross-device-interference candidate, but disproved by code flow: `ensure_offload_stream` (the only reader/writer) is only ever called *after* `resident` has already been determined `False`, so it structurally cannot be the cause of the resident check itself failing. Confirmed with the same debug probe: resident hit-rate collapse is identical regardless of whether this global gets clobbered by the other GPU's UNet. ## Where we're stuck `vbar_fault()` itself (`.venv/…/comfy_aimdo/model_vbar.py` → `lib.vbar_fault`, a ctypes call into the compiled `aimdo.so`/`aimdo_rocm.so`) is where the actual "is this signature still resident" decision is made. We have no source access to that boundary. We also tried `--verbose DEBUG` hoping the native `set_log_callback` hook would surface fault/evict diagnostics (per `_LOG_LEVELS` mapping native levels 5-7 → `logging.DEBUG`) — across the full trigger+degrade window we got exactly two one-time init messages and nothing else; no fault/evict/resident diagnostic logging exists at any verbosity we could reach. ## What we haven't isolated yet - Whether "continuity" is strictly necessary to trigger this, or whether any sufficiently memory-pressuring submission sequence would do it (continuity pushes the UNet's GPU to ~46.5-46.9GB / 48GB cap during the pin-holding segments; we haven't run a non-continuity high-pressure control). - We also tried an experimental "one segment per independent ComfyUI submission, motion-context state round-tripped through disk between them" variant (mimicking the pattern in NikoDemon80/ComfyUI-H3-Motion-Context) hoping to avoid whatever continuity does — it did **not** avoid the degradation (decode escalated 76→100.7→129.6s across 3 fresh independent submissions, landing in the same historical range). So the trigger isn't specific to "many segments in one submission" either. ## What we can provide on request Full evidence trail — 11(+1) rounds of logs, `nsys` `.nsys-rep` files + CLI stats exports, `py-spy` dumps/flamegraph, debug-probe output, GPU/host-memory traces — is organized and citable by round number. Happy to share the relevant subset (or all of it) if useful for triage. This report intentionally leaves out an unrelated `LARGEST_AIMDO_CASTED_WEIGHT` cross-device-scoping cleanup we noticed along the way (confirmed harmless to this bug, `comfy/model_management.py:1386` isn't device-keyed) — mentioning only in case it's an easy separate fix.

bsakaguchi commented

2026-10-26 22:24:02 +00:00

Author

Owner

@34262315716 commented on GitHub (Sep 19, 2026):

Same shape of failure, different stage — at the text encoder instead of the VAE. And a note about where the source code has been living this whole time.

Before the technical part, three observations offered in the spirit of friendly needle rather than complaint:

1. @lilam9110's report says "we have no source access to that boundary." That boundary is src/model-vbar.c — in this repository. It has been sitting here the entire time. We spent a large part of an evening treating aimdo.dll as an opaque black box before it occurred to us to open the repo the DLL came from. To be fair, we got there in the end; we just took a fifteen-hour scenic route past a file that was one click away. Consider that a shared character-building exercise.

2. vbar_free_memory() is extremely polite. When it does nothing, it returns 0. It does not tell you it did nothing. It does not tell you it cannot do anything. It does not mention that watermark has already reached watermark_limit — which, after a vbars_reset_watermark_limits(), means it can never do anything again. One log(WARNING, "watermark at limit, nothing to free") would have saved us several hours and one process abort. A silent no-op that looks identical to a successful release is a rough thing to debug from the outside.

3. @rattus128 — you wrote in the ComfyUI issue thread, "I have no reproducer of that reported hang to do anything about it." Your own tracker had issue #117 open for two days, with nsys-CLI exports, py-spy flamegraphs, an eleven-round investigation and a monotonic degradation curve. The reproducer was, so to speak, already in the building. We'd have found it much sooner if the trail had pointed here first — which is on us, but the signpost was available.

With that out of the way, the actual content, which we hope is useful.

What we observe

comfy-aimdo 0.5.3, ComfyUI 0.36.0, Windows, RTX 4070 Laptop (8 GB VRAM / 32 GB RAM), MiniMax H3 via ComfyUI_MiniMaxH3_Director.

Same shape as the original report, different stage — text encoder rather than VAE:

Segment 1 of a run: text-encoder conditioning 38.0 s
Segment 2: 151 s → 196 s → or simply never completes (killed at 380 s+)
comfy_aimdo.control.get_total_vram_usage() sits at 4.02–4.83 GB and does not move across segment boundaries
true_free_vram collapses from 6.77 GB to 2.72 GB
During the stall: utilization.gpu 100 % but power.draw 27 W out of 140 W, memory-controller utilization 3 %, disk ~0 — spinning, not computing
py-spy: 100 % of sampled time in read_file_to_device → cast_modules_with_vbar → cast_bias_weight, i.e. the H2D slow path taken for the same weights over and over

What we found in `src/model-vbar.c`

1. vbar_fault() gates on the watermark before it consults residency at all:

if (page_end > mv->watermark) {
    log(VVERBOSE, "VBAR Allocation is above watermark\n");
    return VBAR_FAULT_OOM;
}

Python-side that becomes fault() -> None → ops.py computes resident = False → the layer takes the full host→device copy. Any vbar whose watermark has been driven below the pages being requested therefore reports "not resident" permanently, regardless of what is actually mapped.

2. vbar_free_memory() is a permanent no-op once the watermark reaches zero:

for (; pages_to_free && mv->watermark > mv->watermark_limit; mv->watermark--) {
    if (mod1(mv, mv->watermark - 1, true, false)) { pages_to_free--; pages_freed++; }
}
return (uint64_t)pages_freed * VBAR_PAGE_SIZE;

0 > 0 is false, so the loop body never executes and it returns 0. vbars_reset_watermark_limits() sets watermark_limit = 0 on every vbar, and vbar_free_memory(large) itself drains watermark down to that limit. So calling free_memory(1 << 62) after a vbars_reset_watermark_limits() is a state change that calling free_memory() again cannot undo.

3. Only vbar_prioritize() restores the watermark:

void vbar_prioritize(...) {
    remove_vbar(mv); insert_vbar(mv);
    mv->watermark = mv->nr_pages;      /* the only place this goes back up */
}

ComfyUI calls it from model_patcher.py on model load. Anything kept loaded across a submission boundary is never re-prioritized, so a drained watermark stays drained.

That matches the reported "degrades irreversibly (until process restart)" and the 100 % → 40.5 % → 25.4 % → 0.2 % hit-rate curve: pages get evicted, the watermark drops with them, and from then on the fault returns OOM before residency is even consulted.

What drains them for us: the prefetch queues

The pages get pinned. vbar_fault() ends with:

for (...) { rp->pin_count++; }

and mod1(..., do_free=true, do_unpin=false) will not free a page with pin_count != 0. Meanwhile comfy/model_prefetch.py keeps queues holding module references and faulted _v_block ranges. In our setup those queues were never cleared between segments, so the pinned pages stayed pinned, the encoder's watermark got drained by the next free_memory() attempt, and every subsequent fault on that vbar returned OOM.

Workaround that fixed it for us (one line)

ComfyUI's per-node finally (execution.py:546-554) does four things:

comfy_aimdo.control.analyze()                      # only when console level is DEBUG
comfy.model_prefetch.cleanup_prefetch_queues()     # ← this one
comfy.model_management.reset_cast_buffers()
comfy_aimdo.model_vbar.vbars_reset_watermark_limits()

A node that loops over segments inside a single node execution gets that block once for the whole job, not between segments. So we call the second line ourselves at each segment boundary.

Result, three segments, same process:

	before	after
seg 1 / 2 / 3 conditioning	38.0 s / 151–196 s / stall	38.0 s / 32.1 s / 41.9 s
`aimdo_vram` at segment boundaries	stuck at 4.02 G	0.05 G, every time (7/7)
`true_free_vram`	2.72 G	6.69 G, all 7 samples
sampling rate	58 → 104 s/it	60.6 / 54.7 / 65.8 / 62.9 s/it

For calibration against the original report's escalation (41.5 → 62.9 → 129.6 → 207.4 → 284.4 s), our segment 3 — the one that should have blown up — came in at 41.9 s.

One loose end we couldn't explain

With every vbar reporting watermark=0 and loaded_size() == 0, get_total_vram_usage() still reports 4.02 GB. If every page is unmapped and resident_count is 0, we can't account for that number from src/model-vbar.c alone — mod1() decrements total_vram_usage on every free, so it shouldn't be vbar pages. Possibly a different pool counted by the same function; we ran out of evening before reading control.c properly. Not blocking for us, but if it is a real accounting leak it may matter here.

Happy to supply logs, the full command lines of the runs that did and didn't reproduce, or re-run anything on request.

@34262315716 commented on GitHub (Sep 19, 2026): Same shape of failure, different stage — at the text encoder instead of the VAE. And a note about where the source code has been living this whole time. Before the technical part, three observations offered in the spirit of friendly needle rather than complaint: **1. @lilam9110's report says "we have no source access to that boundary."** That boundary is `src/model-vbar.c` — in this repository. It has been sitting here the entire time. We spent a large part of an evening treating `aimdo.dll` as an opaque black box before it occurred to us to open the repo the DLL came from. To be fair, we got there in the end; we just took a fifteen-hour scenic route past a file that was one click away. Consider that a shared character-building exercise. **2. `vbar_free_memory()` is extremely polite.** When it does nothing, it returns `0`. It does not tell you it did nothing. It does not tell you it *cannot* do anything. It does not mention that `watermark` has already reached `watermark_limit` — which, after a `vbars_reset_watermark_limits()`, means it can never do anything again. One `log(WARNING, "watermark at limit, nothing to free")` would have saved us several hours and one process abort. A silent no-op that looks identical to a successful release is a rough thing to debug from the outside. **3. @rattus128** — you wrote in the ComfyUI issue thread, *"I have no reproducer of that reported hang to do anything about it."* Your own tracker had issue #117 open for two days, with nsys-CLI exports, py-spy flamegraphs, an eleven-round investigation and a monotonic degradation curve. The reproducer was, so to speak, already in the building. We'd have found it much sooner if the trail had pointed here first — which is on us, but the signpost was available. With that out of the way, the actual content, which we hope is useful. ## What we observe comfy-aimdo 0.5.3, ComfyUI 0.36.0, Windows, RTX 4070 Laptop (8 GB VRAM / 32 GB RAM), MiniMax H3 via `ComfyUI_MiniMaxH3_Director`. Same shape as the original report, different stage — text encoder rather than VAE: - Segment 1 of a run: text-encoder conditioning **38.0 s** - Segment 2: **151 s → 196 s → or simply never completes** (killed at 380 s+) - `comfy_aimdo.control.get_total_vram_usage()` sits at **4.02–4.83 GB** and does not move across segment boundaries - `true_free_vram` collapses from **6.77 GB** to **2.72 GB** - During the stall: `utilization.gpu` 100 % but `power.draw` 27 W out of 140 W, memory-controller utilization 3 %, disk ~0 — spinning, not computing - py-spy: 100 % of sampled time in `read_file_to_device` → `cast_modules_with_vbar` → `cast_bias_weight`, i.e. the H2D slow path taken for the same weights over and over ## What we found in `src/model-vbar.c` **1. `vbar_fault()` gates on the watermark before it consults residency at all:** ```c if (page_end > mv->watermark) { log(VVERBOSE, "VBAR Allocation is above watermark\n"); return VBAR_FAULT_OOM; } ``` Python-side that becomes `fault() -> None` → `ops.py` computes `resident = False` → the layer takes the full host→device copy. **Any vbar whose watermark has been driven below the pages being requested therefore reports "not resident" permanently, regardless of what is actually mapped.** **2. `vbar_free_memory()` is a permanent no-op once the watermark reaches zero:** ```c for (; pages_to_free && mv->watermark > mv->watermark_limit; mv->watermark--) { if (mod1(mv, mv->watermark - 1, true, false)) { pages_to_free--; pages_freed++; } } return (uint64_t)pages_freed * VBAR_PAGE_SIZE; ``` `0 > 0` is false, so the loop body never executes and it returns 0. `vbars_reset_watermark_limits()` sets `watermark_limit = 0` on **every** vbar, and `vbar_free_memory(large)` itself drains `watermark` down to that limit. So calling `free_memory(1 << 62)` after a `vbars_reset_watermark_limits()` is a state change that calling `free_memory()` again cannot undo. **3. Only `vbar_prioritize()` restores the watermark:** ```c void vbar_prioritize(...) { remove_vbar(mv); insert_vbar(mv); mv->watermark = mv->nr_pages; /* the only place this goes back up */ } ``` ComfyUI calls it from `model_patcher.py` on model **load**. Anything kept loaded across a submission boundary is never re-prioritized, so a drained watermark stays drained. That matches the reported "degrades irreversibly (until process restart)" and the 100 % → 40.5 % → 25.4 % → 0.2 % hit-rate curve: pages get evicted, the watermark drops with them, and from then on the fault returns OOM before residency is even consulted. ## What drains them for us: the prefetch queues The pages get pinned. `vbar_fault()` ends with: ```c for (...) { rp->pin_count++; } ``` and `mod1(..., do_free=true, do_unpin=false)` will not free a page with `pin_count != 0`. Meanwhile `comfy/model_prefetch.py` keeps queues holding module references and faulted `_v_block` ranges. In our setup those queues were never cleared between segments, so the pinned pages stayed pinned, the encoder's watermark got drained by the next `free_memory()` attempt, and every subsequent fault on that vbar returned OOM. ## Workaround that fixed it for us (one line) ComfyUI's per-node `finally` (`execution.py:546-554`) does four things: ```python comfy_aimdo.control.analyze() # only when console level is DEBUG comfy.model_prefetch.cleanup_prefetch_queues() # ← this one comfy.model_management.reset_cast_buffers() comfy_aimdo.model_vbar.vbars_reset_watermark_limits() ``` A node that loops over segments **inside a single node execution** gets that block once for the whole job, not between segments. So we call the second line ourselves at each segment boundary. Result, three segments, same process: | | before | after | |---|---|---| | seg 1 / 2 / 3 conditioning | 38.0 s / 151–196 s / stall | 38.0 s / **32.1 s** / **41.9 s** | | `aimdo_vram` at segment boundaries | stuck at **4.02 G** | **0.05 G**, every time (7/7) | | `true_free_vram` | 2.72 G | **6.69 G**, all 7 samples | | sampling rate | 58 → 104 s/it | 60.6 / 54.7 / 65.8 / 62.9 s/it | For calibration against the original report's escalation (41.5 → 62.9 → 129.6 → 207.4 → 284.4 s), our segment 3 — the one that should have blown up — came in at 41.9 s. ## One loose end we couldn't explain With every vbar reporting `watermark=0` and `loaded_size() == 0`, `get_total_vram_usage()` still reports **4.02 GB**. If every page is unmapped and `resident_count` is 0, we can't account for that number from `src/model-vbar.c` alone — `mod1()` decrements `total_vram_usage` on every free, so it shouldn't be vbar pages. Possibly a different pool counted by the same function; we ran out of evening before reading `control.c` properly. Not blocking for us, but if it is a real accounting leak it may matter here. Happy to supply logs, the full command lines of the runs that did and didn't reproduce, or re-run anything on request.

bsakaguchi referenced this issue

2026-10-26 22:24:25 +00:00

[PR #44] [CLOSED] fix: use module attributes to detect vendor directly #84

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#44

No description provided.

Rows
Columns