Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #25] bug & fix: ModelVBAR crashes on NVIDIA GRID / vGPU instead of falling back #7

New issue

Open

opened 2026-10-26 22:23:28 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 22:23:28 +00:00

Owner

Originally created by @gregdotcom on GitHub (Mar 17, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/25

Hey,
i ran into a MemoryError on my setup even though I have enough VRAM.
Took a while to figure out but the issue is that VBAR allocation just fails silently on virtualized GPUs (vGPU environments) and then throws a hard exception instead of falling back gracefully.

MemoryError: VBAR allocation failed File "execution.py", line 525, in execute File "execution.py", line 334, in get_output_data File "custom_nodes/rgthree-comfy/py/power_prompt_simple.py", line 39, in main conditioning = CLIPTextEncode().encode(opt_clip, prompt)[0] File "nodes.py", line 80, in encode return (clip.encode_from_tokens_scheduled(tokens), ) File "comfy/sd.py", line 375, in encode_from_tokens self.load_model(tokens) File "comfy/model_management.py", line 819, in load_models_gpu loaded_model.model_load(lowvram_model_memory, ...) File "comfy/model_patcher.py", line 1495, in _vbar_get vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index) File "comfy_aimdo/model_vbar.py", line 51, in __init__ raise MemoryError("VBAR allocation failed")

My setup: GRID RTX6000-24Q (24GB vGPU), Ubuntu, CUDA 12.8
VBAR cannot allocate because the underlying low-level CUDA memory ops aren't fully supported in virtualized environments.

What I changed (i got help from AI to find a working fix)

ModelVBAR.__init__ no longer raises MemoryError when allocation fails, instead sets _disabled = True and emits a RuntimeWarning
All methods (alloc, fault, prioritize, deprioritize, loaded_size, etc.) check _disabled and return safe dummy values instead of crashing
vbar_fault() and vbar_unpin() module-level functions also handle None alloc gracefully now

ComfyUI runs fine without VBAR on these setups anyway since the optimization doesn't really work in a virtualized context, so this just makes it degrade nicely instead of hard crashing.

Or do i miss something else?

Originally created by @gregdotcom on GitHub (Mar 17, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/25 Hey, i ran into a MemoryError on my setup even though I have enough VRAM. Took a while to figure out but the issue is that VBAR allocation just fails silently on virtualized GPUs (vGPU environments) and then throws a hard exception instead of falling back gracefully. `MemoryError: VBAR allocation failed File "execution.py", line 525, in execute File "execution.py", line 334, in get_output_data File "custom_nodes/rgthree-comfy/py/power_prompt_simple.py", line 39, in main conditioning = CLIPTextEncode().encode(opt_clip, prompt)[0] File "nodes.py", line 80, in encode return (clip.encode_from_tokens_scheduled(tokens), ) File "comfy/sd.py", line 375, in encode_from_tokens self.load_model(tokens) File "comfy/model_management.py", line 819, in load_models_gpu loaded_model.model_load(lowvram_model_memory, ...) File "comfy/model_patcher.py", line 1495, in _vbar_get vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index) File "comfy_aimdo/model_vbar.py", line 51, in __init__ raise MemoryError("VBAR allocation failed")` My setup: GRID RTX6000-24Q (24GB vGPU), Ubuntu, CUDA 12.8 VBAR cannot allocate because the underlying low-level CUDA memory ops aren't fully supported in virtualized environments. ### What I changed (i got help from AI to find a working fix) - `ModelVBAR.__init__` no longer raises `MemoryError` when allocation fails, instead sets `_disabled = True` and emits a `RuntimeWarning` - All methods (`alloc`, `fault`, `prioritize`, `deprioritize`, `loaded_size`, etc.) check `_disabled` and return safe dummy values instead of crashing - `vbar_fault()` and `vbar_unpin()` module-level functions also handle `None` alloc gracefully now ComfyUI runs fine without VBAR on these setups anyway since the optimization doesn't really work in a virtualized context, so this just makes it degrade nicely instead of hard crashing. Or do i miss something else?

bsakaguchi commented

2026-10-26 22:23:29 +00:00

Author

Owner

@rattus128 commented on GitHub (Mar 18, 2026):

This wheel has a built is self-test for startup that will disable the system wholesale if it can get those functions to work:

https://github.com/Comfy-Org/comfy-aimdo/actions/runs/23231132334

Give it a go. It will wholesale disable the dynamic vram system when it fails those APIs.

The degradation you experience is actually full offloading as with no vbar a ModelPatcherDynamic cannot load the model to the GPU at all. The performance of this is often actually quite good and we have had a user following using the --novram mode which is very similar in behaviour (this switches off aimdo though).

If you want the best of both worlds, where you have the --novram model load behaviour but with the dynamic VRAM features we should actually support that in core comfy rather than aimdo, as we can just make your no-vbar dynamic VRAM a mode.

It has to be opt-in as users will assume loaded models we cant fallback to non-loaded models as a default.

@rattus128 commented on GitHub (Mar 18, 2026): This wheel has a built is self-test for startup that will disable the system wholesale if it can get those functions to work: https://github.com/Comfy-Org/comfy-aimdo/actions/runs/23231132334 Give it a go. It will wholesale disable the dynamic vram system when it fails those APIs. The degradation you experience is actually full offloading as with no vbar a ModelPatcherDynamic cannot load the model to the GPU at all. The performance of this is often actually quite good and we have had a user following using the --novram mode which is very similar in behaviour (this switches off aimdo though). If you want the best of both worlds, where you have the --novram model load behaviour but with the dynamic VRAM features we should actually support that in core comfy rather than aimdo, as we can just make your no-vbar dynamic VRAM a mode. It has to be opt-in as users will assume loaded models we cant fallback to non-loaded models as a default.

bsakaguchi commented

2026-10-26 22:23:30 +00:00

Author

Owner

@gregdotcom commented on GitHub (Mar 29, 2026):

Thanks for the new build, working fine now.

Here's the log of the running with your build:

CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10011400000, size=2048k, stream=(nil)
Requested to load Flux2TEModel_
loaded completely; 10221.02 MB usable, 8263.34 MB loaded, full load: True
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10108000000, size=49152k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x101c8000000, size=49152k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10008000000, size=65536k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x100c4000000, size=98304k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10258000000, size=196608k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10264000000, size=196608k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257800000, size=2048k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257a00000, size=2048k, stream=(nil)
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.float8_e4m3fn, manual cast: torch.float16
model_type FLUX
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10008000000, size=65536k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x100c4000000, size=65536k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10258000000, size=196608k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10264000000, size=196608k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10270000000, size=196608k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257800000, size=2048k, stream=(nil)
aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257a00000, size=2048k, stream=(nil)
Requested to load Flux2
Unloaded partially: 2420.34 MB freed, 5843.00 MB remains loaded, 336.00 MB buffer reserved, lowvram patches: 0
loaded completely; 12462.79 MB usable, 8996.01 MB loaded, full load: True

I also tested the latest version of comfy-aimdo with the flag --novram but as expected that falls back to CPU only:

CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16

Just noticed I'm probably not alone with this – looks like there might be other with this issue. It seems to happen with PyTorch >2.8.x: Comfy-Org/ComfyUI#13015

I will link the findings there too, maybe it helps track down a proper fix.
It's bit above my technical level to dig deeper into it myself but happy to test things if needed.

@gregdotcom commented on GitHub (Mar 29, 2026): Thanks for the new build, working fine now. Here's the log of the running with your build: ``` CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16 aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10011400000, size=2048k, stream=(nil) Requested to load Flux2TEModel_ loaded completely; 10221.02 MB usable, 8263.34 MB loaded, full load: True aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10108000000, size=49152k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x101c8000000, size=49152k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10008000000, size=65536k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x100c4000000, size=98304k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10258000000, size=196608k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10264000000, size=196608k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257800000, size=2048k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257a00000, size=2048k, stream=(nil) Found quantization metadata version 1 Detected mixed precision quantization Using mixed precision operations model weight dtype torch.float8_e4m3fn, manual cast: torch.float16 model_type FLUX aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10008000000, size=65536k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x100c4000000, size=65536k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10258000000, size=196608k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10264000000, size=196608k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10270000000, size=196608k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257800000, size=2048k, stream=(nil) aimdo: src/pyt-cu-plug-alloc-async.c:72:VVERBOSE:Freed: ptr=0x10257a00000, size=2048k, stream=(nil) Requested to load Flux2 Unloaded partially: 2420.34 MB freed, 5843.00 MB remains loaded, 336.00 MB buffer reserved, lowvram patches: 0 loaded completely; 12462.79 MB usable, 8996.01 MB loaded, full load: True ``` I also tested the latest version of comfy-aimdo with the flag `--novram` but as expected that falls back to CPU only: ``` CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16 ``` Just noticed I'm probably not alone with this – looks like there might be other with this issue. It seems to happen with PyTorch >2.8.x: [Comfy-Org/ComfyUI#13015](https://github.com/Comfy-Org/ComfyUI/issues/13015) I will link the findings there too, maybe it helps track down a proper fix. It's bit above my technical level to dig deeper into it myself but happy to test things if needed.

bsakaguchi commented

2026-10-26 22:23:30 +00:00

Author

Owner

@sicp-dnegvp commented on GitHub (Mar 31, 2026):

Any chance this can get fixed soon? Running into this as well

Total VRAM 24355 MB, total RAM 128652 MB
pytorch version: 2.10.0+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA A40-24Q : native
Using async weight offloading with 2 streams
Enabled pinned memory 115786.0
working around nvidia conv3d memory bug.
Using Flash Attention
aimdo: src/control.c:68:INFO:comfy-aimdo inited for GPU: NVIDIA A40-24Q (VRAM: 24355 MB)
DynamicVRAM support detected and enabled
Python version: 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]
ComfyUI version: 0.18.1
comfy-aimdo version: 0.2.12
comfy-kitchen version: 0.2.8
ComfyUI frontend version: 1.42.8

Requested to load AutoencoderKL
aimdo: src/model-vbar.c:200:ERROR:Could not reseve Virtual Address space for VBAR
!!! Exception during processing !!! VBAR allocation failed
Traceback (most recent call last):
  File "/opt/ComfyUI/execution.py", line 534, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/execution.py", line 334, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/execution.py", line 308, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/opt/ComfyUI/execution.py", line 296, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "/opt/ComfyUI/nodes.py", line 365, in encode
    t = vae.encode(pixels)
        ^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/sd.py", line 1049, in encode
    model_management.raise_non_oom(e)
  File "/opt/ComfyUI/comfy/model_management.py", line 289, in raise_non_oom
    raise e
  File "/opt/ComfyUI/comfy/sd.py", line 1031, in encode
    model_management.load_models_gpu([self.patcher], memory_required=memory_used, force_full_load=self.disable_offload)
  File "/opt/ComfyUI/comfy/model_management.py", line 820, in load_models_gpu
    loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
  File "/opt/ComfyUI/comfy/model_management.py", line 582, in model_load
    self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
  File "/opt/ComfyUI/comfy/model_management.py", line 613, in model_use_more_vram
    return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/model_patcher.py", line 1707, in partially_load
    raise e
  File "/opt/ComfyUI/comfy/model_patcher.py", line 1704, in partially_load
    self.load(device_to, dirty=dirty)
  File "/opt/ComfyUI/comfy/model_patcher.py", line 1538, in load
    vbar = self._vbar_get(create=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/model_patcher.py", line 1492, in _vbar_get
    vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/comfy_aimdo/model_vbar.py", line 51, in __init__
    raise MemoryError("VBAR allocation failed")
MemoryError: VBAR allocation failed

@sicp-dnegvp commented on GitHub (Mar 31, 2026): Any chance this can get fixed soon? Running into this as well ``` Total VRAM 24355 MB, total RAM 128652 MB pytorch version: 2.10.0+cu128 Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA A40-24Q : native Using async weight offloading with 2 streams Enabled pinned memory 115786.0 working around nvidia conv3d memory bug. Using Flash Attention aimdo: src/control.c:68:INFO:comfy-aimdo inited for GPU: NVIDIA A40-24Q (VRAM: 24355 MB) DynamicVRAM support detected and enabled Python version: 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] ComfyUI version: 0.18.1 comfy-aimdo version: 0.2.12 comfy-kitchen version: 0.2.8 ComfyUI frontend version: 1.42.8 ``` ``` Requested to load AutoencoderKL aimdo: src/model-vbar.c:200:ERROR:Could not reseve Virtual Address space for VBAR !!! Exception during processing !!! VBAR allocation failed Traceback (most recent call last): File "/opt/ComfyUI/execution.py", line 534, in execute output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/ComfyUI/execution.py", line 334, in get_output_data return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/ComfyUI/execution.py", line 308, in _async_map_node_over_list await process_inputs(input_dict, i) File "/opt/ComfyUI/execution.py", line 296, in process_inputs result = f(**inputs) ^^^^^^^^^^^ File "/opt/ComfyUI/nodes.py", line 365, in encode t = vae.encode(pixels) ^^^^^^^^^^^^^^^^^^ File "/opt/ComfyUI/comfy/sd.py", line 1049, in encode model_management.raise_non_oom(e) File "/opt/ComfyUI/comfy/model_management.py", line 289, in raise_non_oom raise e File "/opt/ComfyUI/comfy/sd.py", line 1031, in encode model_management.load_models_gpu([self.patcher], memory_required=memory_used, force_full_load=self.disable_offload) File "/opt/ComfyUI/comfy/model_management.py", line 820, in load_models_gpu loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights) File "/opt/ComfyUI/comfy/model_management.py", line 582, in model_load self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights) File "/opt/ComfyUI/comfy/model_management.py", line 613, in model_use_more_vram return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/ComfyUI/comfy/model_patcher.py", line 1707, in partially_load raise e File "/opt/ComfyUI/comfy/model_patcher.py", line 1704, in partially_load self.load(device_to, dirty=dirty) File "/opt/ComfyUI/comfy/model_patcher.py", line 1538, in load vbar = self._vbar_get(create=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/ComfyUI/comfy/model_patcher.py", line 1492, in _vbar_get vbar = comfy_aimdo.model_vbar.ModelVBAR(self.model_size() * 10, self.load_device.index) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/comfy_aimdo/model_vbar.py", line 51, in __init__ raise MemoryError("VBAR allocation failed") MemoryError: VBAR allocation failed ```

bsakaguchi referenced this issue

2026-10-26 22:23:57 +00:00

[GH-ISSUE #100] Blackwell (RTX 5060 Ti): intermittent Fault failed: 2 (cuMemSetAccess: device not ready) with DynamicVRAM/VBAR #36

bsakaguchi referenced this issue

2026-10-26 22:24:12 +00:00

[PR #7] [MERGED] Prs/cleanup #58