[GH-ISSUE #94 ] Krea2 Prompt Enhancement causes NaN when enabling Dynamic VRAM on a gfx1030 (RDNA2) card. #32

Author

Owner

@0xDELUXA commented on GitHub (Aug 25, 2026):

Which ROCm version are you using? On ROCm 7.14+, dynamic VRAM is enabled by default. If you aren’t on a recent version, you should update.

@0xDELUXA commented on GitHub (Aug 25, 2026): Which ROCm version are you using? On ROCm 7.14+, dynamic VRAM is enabled by default. If you aren’t on a recent version, you should update.

bsakaguchi commented

Author

Owner

@derfasthirnlosenick commented on GitHub (Aug 25, 2026):

Which ROCm version are you using? On ROCm 7.14+, dynamic VRAM is enabled by default. If you aren’t on a recent version, you should update.

7.14, so I have the choice of disabling dynamic veam and increased vram headroom. The latter gave me the NaN error as well after 10ish gens so something seems to be off still.

@derfasthirnlosenick commented on GitHub (Aug 25, 2026): > Which ROCm version are you using? On ROCm 7.14+, dynamic VRAM is enabled by default. If you aren’t on a recent version, you should update. 7.14, so I have the choice of disabling dynamic veam and increased vram headroom. The latter gave me the NaN error as well after 10ish gens so something seems to be off still.

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 25, 2026):

I'm not on RDNA2, so I can't really investigate this issue. It works fine on my side on RDNA4.

@0xDELUXA commented on GitHub (Aug 25, 2026): I'm not on RDNA2, so I can't really investigate this issue. It works fine on my side on RDNA4.

bsakaguchi commented

Author

Owner

@derfasthirnlosenick commented on GitHub (Aug 25, 2026):

I'm not on RDNA2, so I can't really investigate this issue. It works fine on my side on RDNA4.

Which his an interesting point in itself - I'll check if there's different paths for rdna2 vs 4. In any case, the reserve-vram already helps to get it to "usable", so I'll try to track it down once I have a bit of time.

@derfasthirnlosenick commented on GitHub (Aug 25, 2026): > I'm not on RDNA2, so I can't really investigate this issue. It works fine on my side on RDNA4. Which his an interesting point in itself - I'll check if there's different paths for rdna2 vs 4. In any case, the reserve-vram already helps to get it to "usable", so I'll try to track it down once I have a bit of time.

bsakaguchi commented

Author

Owner

@MillanSiS commented on GitHub (Aug 31, 2026):

Facing similar issue with dynamic vram enabled on gfx1031 rdna2. For me Krea2 turbo fp8 generating black images with any prompt. Klein 9b distill fp8 generates 1-2 somewhat correct images after startup then produces only garbage images. Zimage turbo fp8 somehow generates good images but haven't tested extensively. Using device specific wheels in venv from AMD (torch[device-gfx1031], ROCM 10.0.0 as dependency). same issue occurs with ROCM 7.15 as well. With --disable-dynamic-vram (since comfyui 0.34.0) all works fine

@MillanSiS commented on GitHub (Aug 31, 2026): Facing similar issue with dynamic vram enabled on gfx1031 rdna2. For me Krea2 turbo fp8 generating black images with any prompt. Klein 9b distill fp8 generates 1-2 somewhat correct images after startup then produces only garbage images. Zimage turbo fp8 somehow generates good images but haven't tested extensively. Using device specific wheels in venv from AMD (torch[device-gfx1031], ROCM 10.0.0 as dependency). same issue occurs with ROCM 7.15 as well. With --disable-dynamic-vram (since comfyui 0.34.0) all works fine

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 31, 2026):

Do you use any other environment variables or flags?

@0xDELUXA commented on GitHub (Aug 31, 2026): Do you use any other environment variables or flags?

bsakaguchi commented

Author

Owner

@MillanSiS commented on GitHub (Aug 31, 2026):

Do you use any other environment variables or flags?

I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue

@MillanSiS commented on GitHub (Aug 31, 2026): > Do you use any other environment variables or flags? I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 31, 2026):

RDNA2 support isn't the main focus, dynamic VRAM works better on 3 and 4.

@0xDELUXA commented on GitHub (Aug 31, 2026): RDNA2 support isn't the main focus, dynamic VRAM works better on 3 and 4.

bsakaguchi commented

Author

Owner

@derfasthirnlosenick commented on GitHub (Aug 31, 2026):

Do you use any other environment variables or flags?

I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue

Can confirm, the reserve-vram stopped helping and artifacts started appearing.

Claude suggested that there might be an issue in src/plat.h three_stooges that can cause incorrect mapping to be declared "safe", which could be fixed by adding a cuCtxSynchronize after the set access. That worked for me a few days ago but I didn't get around creating a pull request.

@derfasthirnlosenick commented on GitHub (Aug 31, 2026): > > Do you use any other environment variables or flags? > > I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue Can confirm, the reserve-vram stopped helping and artifacts started appearing. Claude suggested that there might be an issue in src/plat.h three_stooges that can cause incorrect mapping to be declared "safe", which could be fixed by adding a cuCtxSynchronize after the set access. That worked for me a few days ago but I didn't get around creating a pull request.

bsakaguchi commented

Author

Owner

@MillanSiS commented on GitHub (Aug 31, 2026):

Do you use any other environment variables or flags?

I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue

Can confirm, the reserve-vram stopped helping and artifacts started appearing.

Claude suggested that there might be an issue in src/plat.h three_stooges that can cause incorrect mapping to be declared "safe", which could be fixed by adding a cuCtxSynchronize after the set access. That worked for me a few days ago but I didn't get around creating a pull request.

any noticeable impact on the performance advantage of dynamic vram ?

@MillanSiS commented on GitHub (Aug 31, 2026): > > > Do you use any other environment variables or flags? > > > > > > I only have export HF_HOME="$XDG_CACHE_HOME/huggingface" export TORCH_HOME="$XDG_CACHE_HOME/torch" and start with 'python main.py --fast-disk'. By default it uses quad cross attention but same thing occurs with pytorch attention as well. The --reserve-vram 2 mentioned by derfasthirnlosenick doesn't help either. Sometimes i get random artifacts on the display Only when prompt executes with dynamic vram enabled and no my gpu isnt failing. Probably a memory corruption issue > > Can confirm, the reserve-vram stopped helping and artifacts started appearing. > > Claude suggested that there might be an issue in src/plat.h three_stooges that can cause incorrect mapping to be declared "safe", which could be fixed by adding a cuCtxSynchronize after the set access. That worked for me a few days ago but I didn't get around creating a pull request. any noticeable impact on the performance advantage of dynamic vram ?

bsakaguchi commented

Author

Owner

@derfasthirnlosenick commented on GitHub (Aug 31, 2026):

For me the main benefit is that I can actually run krea 2

@derfasthirnlosenick commented on GitHub (Aug 31, 2026): For me the main benefit is that I can actually run krea 2

bsakaguchi commented

Author

Owner

@patientx commented on GitHub (Aug 31, 2026):

"Prompt enhancement" doesn't work any good with rdna2 at all, as far as I know it is different then using the models as clip , it treats it as an llm model which is extra slow. Personally I would just use Claude etc or if you want local prompt enhancements, try qwenvl with 4 bit setting ( needs bitsandbytes) which let's me for example use this model (in nf4 quant) : https://huggingface.co/huihui-ai/Huihui-Qwen3-VL-8B-Instruct-abliterated which does both image and video description , and prompt enhancement.

I am using a modified version of the node though and I realized now they (the original) has added some memory improvements and new model support. For me quantized version of the above model works very well for all those scenarios though.

the QwenVL node : https://github.com/1038lab/ComfyUI-QwenVL
my fork : https://github.com/patientx/ComfyUI-QwenVL

My fork has saving and loading the model in nvfp4 format too, which I've actually made a quant of the above model, and just uploaded it to my huggingface account. It doesn't have automatic downloading but you can just download all the files in there to this folder (create if not present) comfyui\models\LLM\Qwen-VL\Huihui-Qwen3-VL-8B-Instruct-abliterated-bnb-nf4 which then should show up in the list of models you can use at the bottom of the loader.

You can download other models and save them as nvfp4 -thus fits into memory and runs well- and get rid of the original models like this.

@patientx commented on GitHub (Aug 31, 2026): "Prompt enhancement" doesn't work any good with rdna2 at all, as far as I know it is different then using the models as clip , it treats it as an llm model which is extra slow. Personally I would just use Claude etc or if you want local prompt enhancements, try qwenvl with 4 bit setting ( needs bitsandbytes) which let's me for example use this model (in nf4 quant) : `https://huggingface.co/huihui-ai/Huihui-Qwen3-VL-8B-Instruct-abliterated` which does both image and video description , and prompt enhancement. <img width="221" height="379" alt="Image" src="https://github.com/user-attachments/assets/8fd5d9ed-7f46-4b79-8d5a-4286983e4b6a" /> I am using a modified version of the node though and I realized now they (the original) has added some memory improvements and new model support. For me quantized version of the above model works very well for all those scenarios though. the QwenVL node : https://github.com/1038lab/ComfyUI-QwenVL my fork : https://github.com/patientx/ComfyUI-QwenVL My fork has saving and loading the model in nvfp4 format too, which I've actually made a quant of the above model, and just uploaded it to [my huggingface account](https://huggingface.co/patientxtr/Huihui-Qwen3-VL-8B-Instruct-abliterated-bnb-nf4). It doesn't have automatic downloading but you can just download all the files in there to this folder (create if not present) `comfyui\models\LLM\Qwen-VL\Huihui-Qwen3-VL-8B-Instruct-abliterated-bnb-nf4` which then should show up in the list of models you can use at the bottom of the loader. <img width="1054" height="697" alt="Image" src="https://github.com/user-attachments/assets/dc5e8f52-f92a-46cc-92ae-ddec46d186cb" /> <img width="943" height="456" alt="Image" src="https://github.com/user-attachments/assets/974bc908-2851-4649-bf89-a73a8925328f" /> You can download other models and save them as nvfp4 -thus fits into memory and runs well- and get rid of the original models like this.

bsakaguchi commented

Author

Owner

@derfasthirnlosenick commented on GitHub (Aug 31, 2026):

"Prompt enhancement" doesn't work any good with rdna2 at all,

I don't have any issues at all with the enhancement part.
Sure, it's not fast (takes a few sec), but nothing is on RDNA2. The main issue is the memory problem discussed here, and that the loading/unloading takes more time and that the output ain't that great sometimes. So I've mostly switched it off now.

@derfasthirnlosenick commented on GitHub (Aug 31, 2026): > "Prompt enhancement" doesn't work any good with rdna2 at all, I don't have any issues at all with the enhancement part. Sure, it's not fast (takes a few sec), but nothing is on RDNA2. The main issue is the memory problem discussed here, and that the loading/unloading takes more time and that the output ain't that great sometimes. So I've mostly switched it off now.

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 31, 2026):

To be honest, I don't think the Comfy devs will investigate this issue.

@0xDELUXA commented on GitHub (Aug 31, 2026): To be honest, I don't think the Comfy devs will investigate this issue.

bsakaguchi commented

Author

Owner

@MillanSiS commented on GitHub (Aug 31, 2026):

Least we could do is report this based on this explicit warning regarding --disable-dynamic-vram:

[WARNING] Dynamic vram disabled with argument. If you have any issues with dynamic vram enabled please give us a detailed reports as this argument will be removed soon. If you use gguf we recommend keeping dynamic vram enabled and using native ComfyUI model formats instead. ComfyUI native formats like fp8 will be faster even if they are larger than your memory.

@MillanSiS commented on GitHub (Aug 31, 2026): Least we could do is report this based on this explicit warning regarding --disable-dynamic-vram: `[WARNING] Dynamic vram disabled with argument. If you have any issues with dynamic vram enabled please give us a detailed reports as this argument will be removed soon. If you use gguf we recommend keeping dynamic vram enabled and using native ComfyUI model formats instead. ComfyUI native formats like fp8 will be faster even if they are larger than your memory.`

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 31, 2026):

ComfyUI native formats like fp8 will be faster even if they are larger than your memory.

github.com/Comfy-Org/ComfyUI@8a33128f2f changes this line, so please pull the latest updates.

@0xDELUXA commented on GitHub (Aug 31, 2026): > ComfyUI native formats like fp8 will be faster even if they are larger than your memory. https://github.com/Comfy-Org/ComfyUI/commit/8a33128f2f8c5585c57486c07de481241e70a39c changes this line, so please pull the latest updates.

bsakaguchi commented

Author

Owner

@MillanSiS commented on GitHub (Aug 31, 2026):

Comfy-Org/ComfyUI@8a33128 changes this line, so please pull the latest updates.

I like to stay on release tags. master has nothing that addresses this issue anyway, so the issue persists with latest master too

@MillanSiS commented on GitHub (Aug 31, 2026): > [Comfy-Org/ComfyUI@8a33128](https://github.com/Comfy-Org/ComfyUI/commit/8a33128f2f8c5585c57486c07de481241e70a39c) changes this line, so please pull the latest updates. I like to stay on release tags. master has nothing that addresses this issue anyway, so the issue persists with latest master too

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Aug 31, 2026):

And it will continue to have nothing.

@0xDELUXA commented on GitHub (Aug 31, 2026): And it will continue to have nothing.

bsakaguchi commented

Author

Owner

@crosson commented on GitHub (Oct 25, 2026):

I'm reproducing this on RDNA4 (r9700 pro) as well, so it isn't limited to RDNA2.

Setup: gfx1201 R9700 pro 32 GB, Linux, torch 2.11.0+rocm7.14, ComfyUI master, comfy-aimdo 0.5.6 (also reproduces
on 0.6.0).

Repro: Krea2 bf16 UNET + Qwen3-VL text encoder + VAE at 1024x1024, re-queued with a new seed each time. After 3–7 runs
the UNET output latent is entirely NaN (black image) and stays that way until models are unloaded.

I don't use the enhancer node at all. My WF is more basic than that. I have tested with ck and sage attention and triton backend on/off.

Corruptions: once outputs go NaN, I compared each layer's resident VBAR weight (_v_weight) against the
checkpoint tensor. Several layers no longer match, even though vbar_signature_compare still reports them as resident:

layer	elements differing
`tproj.1.weight`	67.0M / 226M (with NaNs)
`tmlp.2.weight`	37.7M / 37.7M
`blocks.9.mlp.up.weight`	8.4M / 100M
`txtmlp.3.weight`	4.2M / 37.7M

On 0.6.0 only tproj.1 was hit, at 33,554,342 elements, which is almost exactly 64 MiB of bf16. So whole pages look like
they get reused or remapped while their signature stays valid, and resident layers then read stale data on every later
run.

Workaround: --disable-smart-memory fixes it (0 failures in 9 runs, including 2400x2400) while keeping dynamic VRAM
and fast disk enabled.

@crosson commented on GitHub (Oct 25, 2026): I'm reproducing this on RDNA4 (r9700 pro) as well, so it isn't limited to RDNA2. **Setup:** gfx1201 R9700 pro 32 GB, Linux, torch 2.11.0+rocm7.14, ComfyUI master, comfy-aimdo 0.5.6 (also reproduces on 0.6.0). **Repro:** Krea2 bf16 UNET + Qwen3-VL text encoder + VAE at 1024x1024, re-queued with a new seed each time. After 3–7 runs the UNET output latent is entirely NaN (black image) and stays that way until models are unloaded. I don't use the enhancer node at all. My WF is more basic than that. I have tested with ck and sage attention and triton backend on/off. **Corruptions:** once outputs go NaN, I compared each layer's resident VBAR weight (`_v_weight`) against the checkpoint tensor. Several layers no longer match, even though `vbar_signature_compare` still reports them as resident: | layer | elements differing | |---|---| | `tproj.1.weight` | 67.0M / 226M (with NaNs) | | `tmlp.2.weight` | 37.7M / 37.7M | | `blocks.9.mlp.up.weight` | 8.4M / 100M | | `txtmlp.3.weight` | 4.2M / 37.7M | On 0.6.0 only `tproj.1` was hit, at 33,554,342 elements, which is almost exactly 64 MiB of bf16. So whole pages look like they get reused or remapped while their signature stays valid, and resident layers then read stale data on every later run. **Workaround:** `--disable-smart-memory` fixes it (0 failures in 9 runs, including 2400x2400) while keeping dynamic VRAM and fast disk enabled.

bsakaguchi commented

Author

Owner

@0xDELUXA commented on GitHub (Oct 25, 2026):

This looks Linux-specific. RDNA4 on Windows is fine for me.

The corrupted range on 0.6.0 is almost exactly two 32 MiB VBAR pages, with their signatures unchanged, so a page got overwritten while still mapped. One Linux-only difference on that path is unmap_workaround in src/plat.h, added for ROCm/ROCm#6021. ROCr has fixed that since (ROCm/rocm-systems#4363), so on 7.14 the range now gets remapped twice. Not proven harmful, but it's the first suspect.

The script below checks that signature-resident chunks keep their contents under eviction pressure, no models needed. On Windows (gfx1200, aimdo 0.6.0) it passes 20 rounds with 0 corrupt chunks. Could someone on Linux run it? If it fails, try again with unmap_workaround defined empty.

vbar_resident_integrity.py

# Two VBARs that together exceed VRAM are run alternately with activation
# pressure in between. Any chunk whose fault signature still matches the one
# saved when it was written must still hold the written bytes.
import gc
import os
import sys

os.environ.setdefault("PYTORCH_ALLOC_CONF", "backend:cudaMallocAsync")

import torch
import comfy_aimdo.control as aimdo

assert aimdo.init() and aimdo.init_device(0)
import comfy_aimdo.torch  # noqa: E402
from comfy_aimdo.model_vbar import ModelVBAR, vbar_fault, vbar_unpin, vbar_signature_compare  # noqa: E402

M = 1024 * 1024
CHUNK = 128 * M
RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 10
device = torch.device("cuda", 0)
torch.empty(1, device=device)

free, capacity = torch.cuda.mem_get_info(0)
model = int(capacity * 0.6) // CHUNK * CHUNK
print(f"{torch.cuda.get_device_name(0)} capacity={capacity // M}M model={model // M}M x2 runs={RUNS}")

base = torch.arange(CHUNK // 4, dtype=torch.int32, device=device)


class Model:
    def __init__(self, name, seed):
        self.name = name
        self.seed = seed
        self.vbar = ModelVBAR(model * 2, 0)
        self.allocs = [self.vbar.alloc(CHUNK) for _ in range(model // CHUNK)]
        self.sigs = [None] * len(self.allocs)

    def run(self):
        self.vbar.prioritize()
        stats = {"hit": 0, "bad": 0, "written": 0, "offloaded": 0}
        for i, alloc in enumerate(self.allocs):
            sig = vbar_fault(alloc)
            if sig is None:
                stats["offloaded"] += 1
                continue
            view = comfy_aimdo.torch.aimdo_to_tensor(alloc, device).view(torch.int32)
            expected = base + (self.seed * 100003 + i * 7919)
            if vbar_signature_compare(sig, self.sigs[i]):
                stats["hit"] += 1
                if not torch.equal(view, expected):
                    stats["bad"] += 1
                    diff = int((view != expected).sum())
                    print(f"  CORRUPT {self.name} chunk {i}: {diff * 4 // M}M of {CHUNK // M}M differ")
            else:
                view.copy_(expected)
                self.sigs[i] = sig
                stats["written"] += 1
            _ = view[: 1 << 20].float().sum()
            del view, expected
            vbar_unpin(alloc)
        torch.cuda.synchronize()
        return stats


te = Model("te", 1)
unet = Model("unet", 2)
total_bad = 0
for r in range(RUNS):
    for m in (te, unet):
        s = m.run()
        total_bad += s["bad"]
        print(f"run {r} {m.name}: {s} loaded={m.vbar.loaded_size() // M}M")
    act = torch.randn((int(capacity * 0.25) // 4,), device=device)
    act.mul_(2.0)
    torch.cuda.synchronize()
    del act
    torch.cuda.empty_cache()

print("FAIL" if total_bad else "PASS", f"corrupt resident chunks: {total_bad}")

del te, unet, base
gc.collect()
torch.cuda.synchronize()
aimdo.deinit()

Run with python vbar_resident_integrity.py 20.

@0xDELUXA commented on GitHub (Oct 25, 2026): This looks Linux-specific. RDNA4 on Windows is fine for me. The corrupted range on 0.6.0 is almost exactly two 32 MiB VBAR pages, with their signatures unchanged, so a page got overwritten while still mapped. One Linux-only difference on that path is `unmap_workaround` in `src/plat.h`, added for ROCm/ROCm#6021. ROCr has fixed that since (ROCm/rocm-systems#4363), so on 7.14 the range now gets remapped twice. Not proven harmful, but it's the first suspect. The script below checks that signature-resident chunks keep their contents under eviction pressure, no models needed. On Windows (gfx1200, aimdo 0.6.0) it passes 20 rounds with 0 corrupt chunks. Could someone on Linux run it? If it fails, try again with `unmap_workaround` defined empty. <details> <summary>vbar_resident_integrity.py</summary> ```python # Two VBARs that together exceed VRAM are run alternately with activation # pressure in between. Any chunk whose fault signature still matches the one # saved when it was written must still hold the written bytes. import gc import os import sys os.environ.setdefault("PYTORCH_ALLOC_CONF", "backend:cudaMallocAsync") import torch import comfy_aimdo.control as aimdo assert aimdo.init() and aimdo.init_device(0) import comfy_aimdo.torch # noqa: E402 from comfy_aimdo.model_vbar import ModelVBAR, vbar_fault, vbar_unpin, vbar_signature_compare # noqa: E402 M = 1024 * 1024 CHUNK = 128 * M RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 10 device = torch.device("cuda", 0) torch.empty(1, device=device) free, capacity = torch.cuda.mem_get_info(0) model = int(capacity * 0.6) // CHUNK * CHUNK print(f"{torch.cuda.get_device_name(0)} capacity={capacity // M}M model={model // M}M x2 runs={RUNS}") base = torch.arange(CHUNK // 4, dtype=torch.int32, device=device) class Model: def __init__(self, name, seed): self.name = name self.seed = seed self.vbar = ModelVBAR(model * 2, 0) self.allocs = [self.vbar.alloc(CHUNK) for _ in range(model // CHUNK)] self.sigs = [None] * len(self.allocs) def run(self): self.vbar.prioritize() stats = {"hit": 0, "bad": 0, "written": 0, "offloaded": 0} for i, alloc in enumerate(self.allocs): sig = vbar_fault(alloc) if sig is None: stats["offloaded"] += 1 continue view = comfy_aimdo.torch.aimdo_to_tensor(alloc, device).view(torch.int32) expected = base + (self.seed * 100003 + i * 7919) if vbar_signature_compare(sig, self.sigs[i]): stats["hit"] += 1 if not torch.equal(view, expected): stats["bad"] += 1 diff = int((view != expected).sum()) print(f" CORRUPT {self.name} chunk {i}: {diff * 4 // M}M of {CHUNK // M}M differ") else: view.copy_(expected) self.sigs[i] = sig stats["written"] += 1 _ = view[: 1 << 20].float().sum() del view, expected vbar_unpin(alloc) torch.cuda.synchronize() return stats te = Model("te", 1) unet = Model("unet", 2) total_bad = 0 for r in range(RUNS): for m in (te, unet): s = m.run() total_bad += s["bad"] print(f"run {r} {m.name}: {s} loaded={m.vbar.loaded_size() // M}M") act = torch.randn((int(capacity * 0.25) // 4,), device=device) act.mul_(2.0) torch.cuda.synchronize() del act torch.cuda.empty_cache() print("FAIL" if total_bad else "PASS", f"corrupt resident chunks: {total_bad}") del te, unet, base gc.collect() torch.cuda.synchronize() aimdo.deinit() ``` Run with `python vbar_resident_integrity.py 20`. </details>

bsakaguchi commented