[GH-ISSUE #99] [ROCm multi-GPU] account_free uses the current device instead of the allocation owner #35

New issue

Open

opened 2026-10-26 22:23:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:23:55 +00:00

Owner

Originally created by @rusantiman on GitHub (Aug 30, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/99

Summary

On Windows ROCm with multiple GPUs visible, comfy-aimdo 0.4.15 can fail to account a free when a pointer was allocated on one GPU but another GPU is current when the pointer is freed.

The warning is:

account_free: could not account free

This is reproducible without DynamicVRAM or VBAR.

Environment

Windows 11
AMD Radeon RX 9070 XT ×2
ROCm
comfy-aimdo 0.4.15
PyTorch with both GPUs visible

Minimal reproduction

I tested a small allocator probe without DynamicVRAM/VBAR.

Case 1: 2 GPUs visible, mismatched current device

Allocate a normal torch tensor on GPU 0.
Change the current device to GPU 1.
Free the GPU 0 allocation while GPU 1 is current.

Result:

account_free: could not account free

The probe produced 3 accounting misses.

The AIMDO log reported:

Total VRAM for VBARs: 0 MB

so this does not require DynamicVRAM/VBAR.

Case 2: 2 GPUs visible, current device matches allocation owner

Allocate and free on the same current device.

Result:

No account_free misses.

Case 3: 1 GPU visible

Run the same allocation/free probe with only one GPU visible.

Result:

No account_free misses.

Diagnosis

The failure appears to occur because the free/accounting path uses the current device's allocation table rather than locating the device that actually owns the tracked allocation.

For example:

allocation owner: GPU 0
current device at free time: GPU 1
accounting searches GPU 1 state
tracked GPU 0 allocation is not found
account_free: could not account free

DynamicVRAM/VBAR-heavy workflows make this easier to trigger, but VBAR is not required.

Impact

In my larger ComfyUI test workload with stock comfy-aimdo 0.4.15, this warning occurred 119 times.

The tested workflows still completed, so I am not claiming this accounting issue alone causes a crash. However, it leaves AIMDO's per-device allocation accounting inconsistent in multi-GPU use.

Local fix

I have tested a local fix that looks up the actual allocation owner and performs the free/accounting operation using that owner's device/context.

Originally created by @rusantiman on GitHub (Aug 30, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/99 ## Summary On Windows ROCm with multiple GPUs visible, comfy-aimdo 0.4.15 can fail to account a free when a pointer was allocated on one GPU but another GPU is current when the pointer is freed. The warning is: `account_free: could not account free` This is reproducible without DynamicVRAM or VBAR. ## Environment - Windows 11 - AMD Radeon RX 9070 XT ×2 - ROCm - comfy-aimdo 0.4.15 - PyTorch with both GPUs visible ## Minimal reproduction I tested a small allocator probe without DynamicVRAM/VBAR. ### Case 1: 2 GPUs visible, mismatched current device 1. Allocate a normal torch tensor on GPU 0. 2. Change the current device to GPU 1. 3. Free the GPU 0 allocation while GPU 1 is current. Result: `account_free: could not account free` The probe produced 3 accounting misses. The AIMDO log reported: `Total VRAM for VBARs: 0 MB` so this does not require DynamicVRAM/VBAR. ### Case 2: 2 GPUs visible, current device matches allocation owner Allocate and free on the same current device. Result: No `account_free` misses. ### Case 3: 1 GPU visible Run the same allocation/free probe with only one GPU visible. Result: No `account_free` misses. ## Diagnosis The failure appears to occur because the free/accounting path uses the current device's allocation table rather than locating the device that actually owns the tracked allocation. For example: - allocation owner: GPU 0 - current device at free time: GPU 1 - accounting searches GPU 1 state - tracked GPU 0 allocation is not found - `account_free: could not account free` DynamicVRAM/VBAR-heavy workflows make this easier to trigger, but VBAR is not required. ## Impact In my larger ComfyUI test workload with stock comfy-aimdo 0.4.15, this warning occurred 119 times. The tested workflows still completed, so I am not claiming this accounting issue alone causes a crash. However, it leaves AIMDO's per-device allocation accounting inconsistent in multi-GPU use. ## Local fix I have tested a local fix that looks up the actual allocation owner and performs the free/accounting operation using that owner's device/context.

bsakaguchi referenced this issue

2026-10-26 22:24:20 +00:00

[PR #35] [MERGED] AMD Support #74