Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #118] Windows, 16 GB GPU, SATA models (fast_disk=False): model start stalls for minutes at 100% util / ~60 W with VRAM pinned at 15.4–15.8 GB after "0 models unloaded" (MiniMax H3) #43

New issue

Open

opened 2026-10-26 22:24:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:24:00 +00:00

Owner

Originally created by @mark9009 on GitHub (Sep 28, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/118

Summary

With dynamic VRAM on (default), a model that starts executing while VRAM is almost full sometimes stalls for 1.5–20 minutes before doing any work. nvidia-smi shows 100 % utilization at ~57–63 W (the same card draws 210–245 W while sampling normally). During the stall disk reads, paging and RAM usage stay flat. When the stall ends, sampling runs at its normal speed (≈33–36 s/it). It affects the video VAE (encode) as well as the DiT, it lands on a random clip, and it can happen in a freshly started process. Right before one stall, ComfyUI logs 0 models unloaded while VRAM is at 15.8 / 16 GB, so something the model manager cannot see is still holding that memory.

This looks close to #117 (lost residency after multi-segment generation). The "~4 GB get_total_vram_usage() with every vbar at watermark 0" loose end in that thread may be the same thing we see.

Environment

Windows 10 Pro 19045, RTX 4070 Ti SUPER 16 GB, driver 610.62, 64 GB RAM
ComfyUI 0.36.0 (a8686f2b, 2026-09-18), comfy-aimdo 0.5.5, comfy-kitchen 0.2.35, torch 2.9.1+cu128, sageattention 2.2.0
Launch: --windows-standalone-build --use-sage-attention --listen --disable-pinned-memory
(without --disable-pinned-memory we hit hostbuf_file_reader_read failed, the same as ComfyUI#16223)
Models on a SATA SSD (Samsung 870 QVO), so Model storage policy: fast_disk=False
MiniMax H3 ref2va fp8_scaled (19983 MB staged) + Qwen3VL-32B int8_convrot text encoder (25140 MB staged) + H3 video VAE fp16 (4965 MB staged). Also reproduced with the 14 GB H3 ErosMax checkpoint and the 15 GB nvfp4 text encoder.

Workload

A custom node renders a long video as several H3 clips inside one node execution. For each clip it runs VAE encode (reference video), text encoder, DiT (4 steps), then VAE decode. Between phases it calls unload_all_models() + soft_empty_cache().

Measurements (5 s sampling of nvidia-smi / perf counters)

event	GPU util	power	VRAM
normal sampling	100 %	210–245 W	9–15 GB
stall before first DiT step (clip 1, fresh process)	100 %	59–63 W	15.4 GB
stall in VAE encode (clip 3, right after `0 models unloaded`)	100 %	57–61 W	15.8 GB

Stall durations we logged today: 6:45, 4:00, 1:25, 3:50, and one still stuck after 20 minutes that we had to kill.

Before we changed the node, the second job in the same process was slow from its first clip onwards: sampling 12 min instead of 2:40, reference VAE encode 8 min instead of 36 s, and then it hung. Unloading all models at the start of each node execution fixed that part (both jobs now take 15–17 min instead of hanging), which matches the "vbar_prioritize() only on load" analysis in #117. The random stalls described above still happen.

Also: `--disable-dynamic-vram` is not a workaround here

With --disable-dynamic-vram (with or without --disable-pinned-memory) the process dies while loading the models, right after the VAEs: 0xc0000005 in torch_cpu.dll. This looks like ComfyUI#15424 / #15397 (large mixed-precision quantized safetensors). With these quantized H3 files there is currently no stable way to turn aimdo off on Windows.

#117 (lost residency after multi-segment generation)
Comfy-Org/ComfyUI#16223 (hostbuf_read_file_slice failures, fast_disk=False path, 64 GB machine)
Comfy-Org/ComfyUI#15424, #15397 (access violation loading large quantized text encoders)
Comfy-Org/ComfyUI#15488 (H3 + 64 GB RAM instability on 16 GB Windows cards)

We can supply full logs, the probe CSV, or run a debug build / extra logging.

Originally created by @mark9009 on GitHub (Sep 28, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/118 ### Summary With dynamic VRAM on (default), a model that starts executing while VRAM is almost full sometimes stalls for 1.5–20 minutes before doing any work. `nvidia-smi` shows **100 % utilization at ~57–63 W** (the same card draws 210–245 W while sampling normally). During the stall disk reads, paging and RAM usage stay flat. When the stall ends, sampling runs at its normal speed (≈33–36 s/it). It affects the video VAE (encode) as well as the DiT, it lands on a random clip, and it can happen in a freshly started process. Right before one stall, ComfyUI logs `0 models unloaded` while VRAM is at 15.8 / 16 GB, so something the model manager cannot see is still holding that memory. This looks close to #117 (lost residency after multi-segment generation). The "~4 GB `get_total_vram_usage()` with every vbar at watermark 0" loose end in that thread may be the same thing we see. ### Environment - Windows 10 Pro 19045, RTX 4070 Ti SUPER 16 GB, driver 610.62, 64 GB RAM - ComfyUI 0.36.0 (`a8686f2b`, 2026-09-18), comfy-aimdo 0.5.5, comfy-kitchen 0.2.35, torch 2.9.1+cu128, sageattention 2.2.0 - Launch: `--windows-standalone-build --use-sage-attention --listen --disable-pinned-memory` (without `--disable-pinned-memory` we hit `hostbuf_file_reader_read failed`, the same as ComfyUI#16223) - Models on a SATA SSD (Samsung 870 QVO), so `Model storage policy: fast_disk=False` - MiniMax H3 ref2va fp8_scaled (19983 MB staged) + Qwen3VL-32B int8_convrot text encoder (25140 MB staged) + H3 video VAE fp16 (4965 MB staged). Also reproduced with the 14 GB H3 ErosMax checkpoint and the 15 GB nvfp4 text encoder. ### Workload A custom node renders a long video as several H3 clips inside **one** node execution. For each clip it runs VAE encode (reference video), text encoder, DiT (4 steps), then VAE decode. Between phases it calls `unload_all_models()` + `soft_empty_cache()`. ### Measurements (5 s sampling of nvidia-smi / perf counters) | event | GPU util | power | VRAM | disk read | paging | |---|---|---|---|---|---| | normal sampling | 100 % | 210–245 W | 9–15 GB | 0 | 0 | | **stall before first DiT step** (clip 1, fresh process) | 100 % | 59–63 W | 15.4 GB | 0 | 0 | | **stall in VAE encode** (clip 3, right after `0 models unloaded`) | 100 % | 57–61 W | 15.8 GB | 0 | 0 | Stall durations we logged today: 6:45, 4:00, 1:25, 3:50, and one still stuck after 20 minutes that we had to kill. Before we changed the node, the **second** job in the same process was slow from its first clip onwards: sampling 12 min instead of 2:40, reference VAE encode 8 min instead of 36 s, and then it hung. Unloading all models at the start of each node execution fixed that part (both jobs now take 15–17 min instead of hanging), which matches the "`vbar_prioritize()` only on load" analysis in #117. The random stalls described above still happen. ### Also: `--disable-dynamic-vram` is not a workaround here With `--disable-dynamic-vram` (with or without `--disable-pinned-memory`) the process dies while loading the models, right after the VAEs: `0xc0000005` in `torch_cpu.dll`. This looks like ComfyUI#15424 / #15397 (large mixed-precision quantized safetensors). With these quantized H3 files there is currently no stable way to turn aimdo off on Windows. ### Related - #117 (lost residency after multi-segment generation) - Comfy-Org/ComfyUI#16223 (`hostbuf_read_file_slice` failures, fast_disk=False path, 64 GB machine) - Comfy-Org/ComfyUI#15424, #15397 (access violation loading large quantized text encoders) - Comfy-Org/ComfyUI#15488 (H3 + 64 GB RAM instability on 16 GB Windows cards) We can supply full logs, the probe CSV, or run a debug build / extra logging.

bsakaguchi referenced this issue

2026-10-26 22:24:23 +00:00

[PR #43] [MERGED] Fix ModelVBAR __del__ shutdown error (control is None) #81

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#43

No description provided.

Rows
Columns