mirror of
https://github.com/Comfy-Org/comfy-aimdo.git
synced 2026-11-04 23:37:21 +00:00
[GH-ISSUE #100] Blackwell (RTX 5060 Ti): intermittent Fault failed: 2 (cuMemSetAccess: device not ready) with DynamicVRAM/VBAR #36
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/comfy-aimdo#36
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @joq100 on GitHub (Aug 31, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/100
Summary
I'm seeing intermittent crashes with DynamicVRAM/VBAR on a Blackwell GPU (RTX 5060 Ti 16 GB). The crash disappears completely when DynamicVRAM (or Smart Memory) is disabled, which seems to isolate the issue to AIMDO/VBAR rather than ComfyUI, the model, or VRAM exhaustion.
Environment
Launch command:
Note:
--cuda-device 1is intentional because this is a dual-GPU system.Error
The crash always happens during "Model Initializing..." before the first sampling step.
Relevant log:
The traceback points to:
Reproduction
The issue is intermittent.
Typical pattern:
Fault failed: 2I can reproduce this repeatedly after generating multiple images with Krea2.
What I tested
This strongly suggests the problem is related to DynamicVRAM/VBAR.
Additional notes
--disable-smart-memoryand--disable-dynamic-vrameliminate it.the full ComfyUI error report and workflow.
Node threw an error during execution.ComfyUI Error Report
Error Details
Stack Trace
System Information
Devices
Logs
Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
Additional Context
(Please add any additional context or steps to reproduce the error here)
@joq100 commented on GitHub (Sep 5, 2026):
Update: not Blackwell-specific, still present on 0.5.2, and a retry makes it go away
Some new data since the original report.
Not specific to Blackwell or to the RTX 5060 Ti. I reproduced the identical failure on an RTX 4060 Ti (Ada) in the same machine — same error, same line, same frequency. The original report was on the 5060 Ti launched with
--cuda-device 1; the 4060 Ti runs had no--cuda-deviceflag at all. The current issue title is misleading and I'm happy to retitle it to something like "Intermittent cuMemSetAccess NOT_READY (600) in vbar_fault on model swap".Not related to multi-GPU device selection. It occurs with a single visible CUDA device.
Still present on comfy-aimdo 0.5.2 with ComfyUI 0.34.3.
The precondition appears to be forced eviction. My workflow loads
Krea2TEModel_(4999 MB staged) and thenKrea2(12864 MB staged) — about 17.8 GB of models on a 16 GB card, so the text encoder has to be evicted before the diffusion model can be faulted in. The failure always lands on the first fault after that eviction, 1.1 to 1.5 seconds afterModel Initializing, at roughly one run in five. I compared the inter-event timings of five failing runs against twenty passing ones and found nothing that distinguishes them, which is consistent with a race rather than a state-dependent bug.Error 600 is
CUDA_ERROR_NOT_READY, a transient asynchronous status rather than an allocation failure.plat.h:176treats it as fatal andmodel-vbar.c:393then reports "VRAM Allocation failed (non OOM)". Re-queueing the same prompt always succeeds.A retry makes it go away. I patched
comfy_aimdo/model_vbar.pyso thatModelVBAR.fault()retries onlyVBAR_FAULT_ERROR—VBAR_FAULT_OOMstill returnsNoneunchanged — up to 4 times with 5/10/15/20 ms backoff, reallocating the signature buffer on each attempt:With this in place I ran several generations with zero
Fault failed: 2.This is obviously a user-side workaround and not the right fix — the wait presumably belongs down near the
cuMemSetAccesscall, where the unmap from the eviction can be awaited properly instead of slept through. Happy to test a patched build.Related: Comfy-Org/ComfyUI#12943 reports the same signature and has been open since March.
@Davikar commented on GitHub (Sep 20, 2026):
I started getting this error randomly with 0.5.5 with the second sampler in a dual sampler setup. Disconnecting and reconnecting an input to the ksampler stops it from happening again for some reason...
@HsiangNat commented on GitHub (Oct 11, 2026):
Same intermittent failure here on an Ada GPU (RTX 4060 Laptop), with the latest NVIDIA driver, so this does not look Blackwell-specific or driver-version-specific.
Environment
--enable-assets(no--disable-dynamic-vram, DynamicVRAM active)Workflow
Batch img2img driven by a folder image batch loader (one image per queued prompt): RTX Video Super Resolution (2x) -> VAEEncode -> KSampler (euler/beta, 2 steps, denoise 0.2, cfg 1) -> VAEDecode -> SaveImage.
Models: Krea2 int8 (about 12.9 GB staged, "12864MB Staged"), text encoder about 5 GB ("4999MB Staged"), WanVAE. The diffusion model is much larger than my 8 GB of VRAM, so weights are constantly streamed through DynamicVRAM.
Symptom
Some prompts fail at the very start of sampling. The progress bar sits at
0/2 ... Model Initializing ...for about 13.5 s, then:The traceback ends in
comfy_aimdo/model_vbar.py(vbar_fault) <-comfy/ops.py(cast_modules_with_vbar), called from the Krea2 MLP layer.It is not input-dependent: the failing images have the same format and size (1408x768 JPG) as the ones that succeed, and re-running a failed image on its own works. It is not an OOM either (about 7.4 GB free at startup).
Results
Before updating the driver (older driver, DynamicVRAM on): 7 images queued, 2 failed (#3 and #7).
After updating to the latest driver, 617.42 (DynamicVRAM on): I first ran a 9-image batch with no failures and thought the update had fixed it, but a larger 13-image batch failed again:
So the driver update did not fix it. The 9-image run was just a lucky/small sample.
Observation
In both runs the failures are exactly 4 prompts apart (#3, #7 before; #5, #9, #13 after), and every failure happens at the same place (first KSampler step, right after
Model Krea2 prepared for dynamic VRAM loading). That looks more like a periodic/stateful condition than random flakiness, in case it helps narrow things down.Log excerpt (username and local paths redacted)
Startup arguments and failing prompt (G05)
The other two failures (G09 and G13): identical error lines
Other things that may be relevant
--enable-assets) was active, pausing/resuming between prompts.I'm not able to run further tests myself, so I'm reporting this as an additional data point for the same failure on a different GPU generation and with an up-to-date driver.