Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #97] Segfault in `hostbuf_free()` during RAM-pressure cache eviction (`HostBuffer.del`) #34

New issue

Open

opened 2026-10-26 22:23:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:23:55 +00:00

Owner

Originally created by @Romic77 on GitHub (Aug 25, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/97

Summary

A ComfyUI instance running with DynamicVRAM enabled dies with SIGSEGV inside
aimdo.so's hostbuf_free(). The crash is reached from HostBuffer.__del__,
which is invoked when ComfyUI's RAM-pressure cache evicts a stale cache entry
holding the last reference to a ModelPatcherDynamic (and therefore to its
dynamic_pins HostBuffer objects).

The process is killed outright — faulthandler prints the Python-level stack,
then the interpreter is gone. It has happened twice on the same host, both times
about 2 seconds after a new prompt was submitted, i.e. at the first
post-node-execution eviction of the previous prompt's cached models.

comfy-aimdo 0.4.14 does not fix it: its aimdo.so is byte-identical to
0.4.13 (see Ruled out below).

Environment

Item	Value
comfy-aimdo	0.4.13 (`aimdo.so` md5 `8a58ab558fac0b291cd4dbe1730947b7`, Build ID `1771a215de85c1a92a27615f7336a42cf7932e6f`)
ComfyUI	0.33.0
Python	3.11.13 (conda)
torch	2.8.0+cu128
xformers	0.0.32.post2
numpy	1.26.3
psutil	6.1.1
OS	Ubuntu 24.04.1 LTS, kernel 6.11.0-29-generic
NVIDIA driver	560.28.03
GPU	2 × RTX 4090 24 GB (sm_89); one ComfyUI instance pinned per GPU via `CUDA_VISIBLE_DEVICES`
Host RAM	61 GiB total, 109 GiB swap

Launch arguments of the crashing instance (GPU 0):

python main.py --listen <ip> --port 8190 --disable-pinned-memory \
  --models-directory <path> --extra-model-paths-config <path> \
  --database-url sqlite:///<path>/comfyui_gpu0.db --temp-directory <path>/gpu0

Note there is no --cache-* flag, so the default RAM-pressure cache
(CacheType.RAM_PRESSURE) is in effect.

Crash output

Identical stack on both occurrences (2026-08-24 15:03:27 CST, PID 2115801;
2026-08-25 15:17:01 CST, PID 2415539):

[INFO] got prompt
[INFO] current client id: <redacted>
Fatal Python error: Segmentation fault

Stack (most recent call first):
  File ".../site-packages/comfy_aimdo/host_buffer.py", line 129 in __del__
  File ".../ComfyUI/comfy_execution/caching.py", line 593 in ram_release
  File ".../ComfyUI/execution.py", line 800 in execute_async
  File ".../python3.11/asyncio/events.py", line 84 in _run
  File ".../python3.11/asyncio/base_events.py", line 1936 in _run_once
  File ".../python3.11/asyncio/base_events.py", line 608 in run_forever
  File ".../python3.11/asyncio/base_events.py", line 641 in run_until_complete
  File ".../python3.11/asyncio/runners.py", line 118 in run
  File ".../python3.11/asyncio/runners.py", line 190 in run
  File ".../ComfyUI/execution.py", line 728 in execute
  File ".../ComfyUI/main.py", line 450 in prompt_worker
  File ".../python3.11/threading.py", line 982 in run
  File ".../python3.11/threading.py", line 1045 in _bootstrap_inner
  File ".../python3.11/threading.py", line 1002 in _bootstrap
Extension modules: <321 modules>

host_buffer.py:129 is the lib.hostbuf_free(ptr) call in HostBuffer.__del__:

def __del__(self):
    ptr = getattr(self, "_ptr", None)
    if ptr:
        if lib is not None:
            lib.hostbuf_free(ptr)   # <-- line 129, SIGSEGV here
        self._ptr = None

How the crash path is reached

Reading the ComfyUI 0.33.0 source along the stack:

execution.py:799-811 — after every node execution, when
cache_type == CacheType.RAM_PRESSURE, the executor calls
ram_release_callback(ram_inactive_headroom).
comfy_execution/caching.py:550 RAMPressureCache.ram_release() builds a
candidate list of cache entries whose used_generation != generation
(i.e. not touched by the current prompt) and assigns each an OOM score.
Entries holding a ModelPatcher from an older generation are given
oom_ram_usage = 1e30 with the comment "old ModelPatchers are the first to
go", so they sort to the top of the eviction list.
caching.py:591-597 — the eviction loop del self.cache[key] drops the last
reference to that ModelPatcherDynamic. Its
model.dynamic_pins[device] pin-state tuples (created at
comfy/model_patcher.py:1877-1880 and comfy/model_management.py:1453) each
hold a comfy_aimdo.host_buffer.HostBuffer, so Python refcounting runs
HostBuffer.__del__ → lib.hostbuf_free() → SIGSEGV.

Why this path runs constantly on this host

main.py:412-417 computes the two RAM-pressure thresholds from total RAM:

cache_ram          = min(10.0, max(2.0, total_ram_mb * 0.10 / 1024.0))  # ~6.2 GiB
cache_ram_inactive = min(128.0, total_ram_mb / 1024.0)                  # ~62.0 GiB

ram_release() returns early only when psutil.virtual_memory().available >= target.
With ram_inactive_headroom ≈ 62 GiB on a 61 GiB machine, that condition can
never be satisfied, so the eviction loop executes after every single node and
frees every inactive entry it can find. On this host available sits around
32 GiB, so the loop always runs to completion.

That matches the observed timing exactly: both crashes landed ~2 s after
got prompt, before any model-loading log line — i.e. at the first eviction
following the first node of a new prompt, when the previous prompt's
ModelPatcherDynamic objects became stale.

Ruled out

Not an OOM kill. No oom-kill / Killed process entries in the kernel log
or journal; at crash time the host had ~32 GiB RAM available and 103 GiB of
109 GiB swap free. The process died from SIGSEGV, and faulthandler printed a
Python stack, which an OOM kill would not produce.

Not fixed in 0.4.14. The only release newer than 0.4.13 ships the same
native library:

$ pip download --no-deps comfy-aimdo==0.4.14
$ md5sum {installed-0.4.13,downloaded-0.4.14}/comfy_aimdo/aimdo.so
8a58ab558fac0b291cd4dbe1730947b7  0.4.13/comfy_aimdo/aimdo.so
8a58ab558fac0b291cd4dbe1730947b7  0.4.14/comfy_aimdo/aimdo.so
$ diff -rq 0.4.13/comfy_aimdo 0.4.14/comfy_aimdo
Files ... _version.py differ          # only file that differs

host_buffer.py is identical between the two versions.

Not simply "freeing a degenerate buffer". Because the instance runs with
--disable-pinned-memory, MAX_PINNED_MEMORY stays at its initial -1
(comfy/model_management.py:1562), so

def pinned_hostbuf_size(size):        # model_management.py:1592
    return max(0, int(min(size, MAX_PINNED_MEMORY) * 2))   # -> 0

returns 0, and the pin-state buffers are constructed as
HostBuffer(0, 64 MiB, 0) / HostBuffer(0, 8 MiB, 0) — a non-zero prewarm
with max_mmap_size == 0. We tested that combination in isolation:

control.init()
b = HostBuffer(0, 64 * 1024**2, 0)   # allocates, returns non-null ptr
del b                                # -> "FREED_OK", no crash

All of (0,0,0), (0,8 MiB,0), (0,64 MiB,0), (0,8 MiB,2 GiB) and
(0,64 MiB,2 GiB) allocate and free cleanly in a fresh process. So the crash
needs additional state — a buffer that has actually been extend()ed and
register()ed, with live CUDA host registrations or in-flight transfers at the
moment of free. That state is not reachable from a standalone script, which is
why we cannot offer a self-contained reproducer.

Frequency and impact

Two crashes in two days on the GPU 0 instance. A second instance on the same
host with an identical command line (only port, GPU, DB path, temp dir and
--disable-metadata differ) has run since 2026-08-19 without crashing, so the
trigger appears workload-dependent rather than configuration-dependent — both
instances execute the same ram_release path on every node.

The workload preceding the 2026-08-25 crash was a Flux2 graph using dynamic VRAM
staging, e.g.:

Model Flux2TEModel_ prepared for dynamic VRAM loading.  8262MB Staged.
Model Flux2         prepared for dynamic VRAM loading. 17316MB Staged.
Model AutoencoderKL prepared for dynamic VRAM loading.   160MB Staged.
Prompt executed in 74.97 seconds

Impact is larger than a single failed prompt: the service is under
Restart=always, and reloading ~4260 custom nodes takes ~6 min 48 s before the
HTTP port is bound again, so each crash costs about 7 minutes of downtime.

Suspected cause

We cannot see inside aimdo.so, but the evidence points at
hostbuf_free() releasing host memory that is still referenced by the CUDA
driver — either a still-registered range (no cudaHostUnregister before the
munmap/free) or a buffer whose decommit is racing an async copy. ComfyUI's own
comment at execution.py:807 acknowledges asynchronous decommit behaviour:

# AIMDO MEM_DECOMMIT can outrun psutil.available catching up.
time.sleep(0.05)

A defensive hostbuf_free() that unregisters and synchronises before releasing —
or that tolerates being called on a buffer whose registrations were already torn
down — would likely stop the process from dying.

What we can provide

The host is still running and the fault is reproducible over a timescale of
days. On request we can supply:

comfy_aimdo.control.set_log_debug() output leading up to a crash
(ComfyUI exposes this via --verbose DEBUG).
A core dump with aimdo.so symbols, if a debug build or symbol file is
published — the shipped .so is stripped apart from its Build ID.
The exact workflow JSON, under NDA-free redaction if needed.

Secondary observation (ComfyUI side, not aimdo)

pinned_hostbuf_size() silently collapses to 0 when --disable-pinned-memory
is set, because MAX_PINNED_MEMORY remains -1 and min(size, -1) * 2 is
negative before the max(0, ...) clamp. The pin-state buffers are then created
with a non-zero prewarm but a zero max_mmap_size. That combination does not
crash on its own (tested above), but it is almost certainly not the intended
configuration and may be worth guarding on the ComfyUI side.

Originally created by @Romic77 on GitHub (Aug 25, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/97 ## Summary A ComfyUI instance running with DynamicVRAM enabled dies with `SIGSEGV` inside `aimdo.so`'s `hostbuf_free()`. The crash is reached from `HostBuffer.__del__`, which is invoked when ComfyUI's RAM-pressure cache evicts a stale cache entry holding the last reference to a `ModelPatcherDynamic` (and therefore to its `dynamic_pins` `HostBuffer` objects). The process is killed outright — `faulthandler` prints the Python-level stack, then the interpreter is gone. It has happened twice on the same host, both times about 2 seconds after a new prompt was submitted, i.e. at the first post-node-execution eviction of the previous prompt's cached models. `comfy-aimdo` 0.4.14 does **not** fix it: its `aimdo.so` is byte-identical to 0.4.13 (see *Ruled out* below). ## Environment | Item | Value | | ------------- | ------------------------------------------------------------ | | comfy-aimdo | 0.4.13 (`aimdo.so` md5 `8a58ab558fac0b291cd4dbe1730947b7`, Build ID `1771a215de85c1a92a27615f7336a42cf7932e6f`) | | ComfyUI | 0.33.0 | | Python | 3.11.13 (conda) | | torch | 2.8.0+cu128 | | xformers | 0.0.32.post2 | | numpy | 1.26.3 | | psutil | 6.1.1 | | OS | Ubuntu 24.04.1 LTS, kernel 6.11.0-29-generic | | NVIDIA driver | 560.28.03 | | GPU | 2 × RTX 4090 24 GB (sm_89); one ComfyUI instance pinned per GPU via `CUDA_VISIBLE_DEVICES` | | Host RAM | 61 GiB total, 109 GiB swap | Launch arguments of the crashing instance (GPU 0): ``` python main.py --listen <ip> --port 8190 --disable-pinned-memory \ --models-directory <path> --extra-model-paths-config <path> \ --database-url sqlite:///<path>/comfyui_gpu0.db --temp-directory <path>/gpu0 ``` Note there is **no** `--cache-*` flag, so the default RAM-pressure cache (`CacheType.RAM_PRESSURE`) is in effect. ## Crash output Identical stack on both occurrences (2026-08-24 15:03:27 CST, PID 2115801; 2026-08-25 15:17:01 CST, PID 2415539): ``` [INFO] got prompt [INFO] current client id: <redacted> Fatal Python error: Segmentation fault Stack (most recent call first): File ".../site-packages/comfy_aimdo/host_buffer.py", line 129 in __del__ File ".../ComfyUI/comfy_execution/caching.py", line 593 in ram_release File ".../ComfyUI/execution.py", line 800 in execute_async File ".../python3.11/asyncio/events.py", line 84 in _run File ".../python3.11/asyncio/base_events.py", line 1936 in _run_once File ".../python3.11/asyncio/base_events.py", line 608 in run_forever File ".../python3.11/asyncio/base_events.py", line 641 in run_until_complete File ".../python3.11/asyncio/runners.py", line 118 in run File ".../python3.11/asyncio/runners.py", line 190 in run File ".../ComfyUI/execution.py", line 728 in execute File ".../ComfyUI/main.py", line 450 in prompt_worker File ".../python3.11/threading.py", line 982 in run File ".../python3.11/threading.py", line 1045 in _bootstrap_inner File ".../python3.11/threading.py", line 1002 in _bootstrap Extension modules: <321 modules> ``` `host_buffer.py:129` is the `lib.hostbuf_free(ptr)` call in `HostBuffer.__del__`: ```python def __del__(self): ptr = getattr(self, "_ptr", None) if ptr: if lib is not None: lib.hostbuf_free(ptr) # <-- line 129, SIGSEGV here self._ptr = None ``` ## How the crash path is reached Reading the ComfyUI 0.33.0 source along the stack: 1. `execution.py:799-811` — after **every** node execution, when `cache_type == CacheType.RAM_PRESSURE`, the executor calls `ram_release_callback(ram_inactive_headroom)`. 2. `comfy_execution/caching.py:550` `RAMPressureCache.ram_release()` builds a candidate list of cache entries whose `used_generation != generation` (i.e. not touched by the current prompt) and assigns each an OOM score. Entries holding a `ModelPatcher` from an older generation are given `oom_ram_usage = 1e30` with the comment *"old ModelPatchers are the first to go"*, so they sort to the top of the eviction list. 3. `caching.py:591-597` — the eviction loop `del self.cache[key]` drops the last reference to that `ModelPatcherDynamic`. Its `model.dynamic_pins[device]` pin-state tuples (created at `comfy/model_patcher.py:1877-1880` and `comfy/model_management.py:1453`) each hold a `comfy_aimdo.host_buffer.HostBuffer`, so Python refcounting runs `HostBuffer.__del__` → `lib.hostbuf_free()` → SIGSEGV. ### Why this path runs constantly on this host `main.py:412-417` computes the two RAM-pressure thresholds from total RAM: ```python cache_ram = min(10.0, max(2.0, total_ram_mb * 0.10 / 1024.0)) # ~6.2 GiB cache_ram_inactive = min(128.0, total_ram_mb / 1024.0) # ~62.0 GiB ``` `ram_release()` returns early only when `psutil.virtual_memory().available >= target`. With `ram_inactive_headroom ≈ 62 GiB` on a 61 GiB machine, that condition can never be satisfied, so the eviction loop executes after *every single node* and frees every inactive entry it can find. On this host `available` sits around 32 GiB, so the loop always runs to completion. That matches the observed timing exactly: both crashes landed ~2 s after `got prompt`, before any model-loading log line — i.e. at the first eviction following the first node of a new prompt, when the *previous* prompt's `ModelPatcherDynamic` objects became stale. ## Ruled out **Not an OOM kill.** No `oom-kill` / `Killed process` entries in the kernel log or journal; at crash time the host had ~32 GiB RAM available and 103 GiB of 109 GiB swap free. The process died from `SIGSEGV`, and `faulthandler` printed a Python stack, which an OOM kill would not produce. **Not fixed in 0.4.14.** The only release newer than 0.4.13 ships the same native library: ``` $ pip download --no-deps comfy-aimdo==0.4.14 $ md5sum {installed-0.4.13,downloaded-0.4.14}/comfy_aimdo/aimdo.so 8a58ab558fac0b291cd4dbe1730947b7 0.4.13/comfy_aimdo/aimdo.so 8a58ab558fac0b291cd4dbe1730947b7 0.4.14/comfy_aimdo/aimdo.so $ diff -rq 0.4.13/comfy_aimdo 0.4.14/comfy_aimdo Files ... _version.py differ # only file that differs ``` `host_buffer.py` is identical between the two versions. **Not simply "freeing a degenerate buffer".** Because the instance runs with `--disable-pinned-memory`, `MAX_PINNED_MEMORY` stays at its initial `-1` (`comfy/model_management.py:1562`), so ```python def pinned_hostbuf_size(size): # model_management.py:1592 return max(0, int(min(size, MAX_PINNED_MEMORY) * 2)) # -> 0 ``` returns `0`, and the pin-state buffers are constructed as `HostBuffer(0, 64 MiB, 0)` / `HostBuffer(0, 8 MiB, 0)` — a non-zero `prewarm` with `max_mmap_size == 0`. We tested that combination in isolation: ```python control.init() b = HostBuffer(0, 64 * 1024**2, 0) # allocates, returns non-null ptr del b # -> "FREED_OK", no crash ``` All of `(0,0,0)`, `(0,8 MiB,0)`, `(0,64 MiB,0)`, `(0,8 MiB,2 GiB)` and `(0,64 MiB,2 GiB)` allocate and free cleanly in a fresh process. So the crash needs additional state — a buffer that has actually been `extend()`ed and `register()`ed, with live CUDA host registrations or in-flight transfers at the moment of free. That state is not reachable from a standalone script, which is why we cannot offer a self-contained reproducer. ## Frequency and impact Two crashes in two days on the GPU 0 instance. A second instance on the same host with an identical command line (only port, GPU, DB path, temp dir and `--disable-metadata` differ) has run since 2026-08-19 without crashing, so the trigger appears workload-dependent rather than configuration-dependent — both instances execute the same `ram_release` path on every node. The workload preceding the 2026-08-25 crash was a Flux2 graph using dynamic VRAM staging, e.g.: ``` Model Flux2TEModel_ prepared for dynamic VRAM loading. 8262MB Staged. Model Flux2 prepared for dynamic VRAM loading. 17316MB Staged. Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. Prompt executed in 74.97 seconds ``` Impact is larger than a single failed prompt: the service is under `Restart=always`, and reloading ~4260 custom nodes takes ~6 min 48 s before the HTTP port is bound again, so each crash costs about 7 minutes of downtime. ## Suspected cause We cannot see inside `aimdo.so`, but the evidence points at `hostbuf_free()` releasing host memory that is still referenced by the CUDA driver — either a still-registered range (no `cudaHostUnregister` before the munmap/free) or a buffer whose decommit is racing an async copy. ComfyUI's own comment at `execution.py:807` acknowledges asynchronous decommit behaviour: ```python # AIMDO MEM_DECOMMIT can outrun psutil.available catching up. time.sleep(0.05) ``` A defensive `hostbuf_free()` that unregisters and synchronises before releasing — or that tolerates being called on a buffer whose registrations were already torn down — would likely stop the process from dying. ## What we can provide The host is still running and the fault is reproducible over a timescale of days. On request we can supply: - `comfy_aimdo.control.set_log_debug()` output leading up to a crash (ComfyUI exposes this via `--verbose DEBUG`). - A core dump with `aimdo.so` symbols, if a debug build or symbol file is published — the shipped `.so` is stripped apart from its Build ID. - The exact workflow JSON, under NDA-free redaction if needed. ## Secondary observation (ComfyUI side, not aimdo) `pinned_hostbuf_size()` silently collapses to `0` when `--disable-pinned-memory` is set, because `MAX_PINNED_MEMORY` remains `-1` and `min(size, -1) * 2` is negative before the `max(0, ...)` clamp. The pin-state buffers are then created with a non-zero `prewarm` but a zero `max_mmap_size`. That combination does not crash on its own (tested above), but it is almost certainly not the intended configuration and may be worth guarding on the ComfyUI side.

bsakaguchi referenced this issue

2026-10-26 22:24:19 +00:00

[PR #34] [MERGED] Dev/multi gpu 2 (CORE-21) #72

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#34

No description provided.

Rows
Columns

[GH-ISSUE #97] Segfault in hostbuf_free() during RAM-pressure cache eviction (HostBuffer.__del__) #34