[PR #77] [CLOSED] Synchronize CUDA before VBAR fault eviction #104

New issue

Closed

opened 2026-10-26 22:24:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:24:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-aimdo/pull/77
Author: @I-spyne-I
Created: 7/26/2026
Status: ❌ Closed

Base: master ← Head: agent/fix-vbar-fault-sync

📝 Commits (1)

825b7a5 Synchronize CUDA before VBAR fault eviction

📊 Changes

1 file changed (+7 additions, -0 deletions)

View changed files

📝 src/model-vbar.c (+7 -0)

📄 Description

What changed

Add a CUDA context synchronization boundary at the start of vbar_fault(), after selecting the AIMDO device context and before any budget-driven residency changes can occur.

Root cause

vbar_fault() can run while a kernel launched for the preceding layer is still consuming VBAR-backed pages. Its budget enforcement path may then evict and unmap those pages. That creates a use-after-unmap race in the CUDA virtual-memory layer, which can manifest as nondeterministic inference corruption, a stuck first sampler step, or a Windows nvlddmkm TDR/reset rather than a recoverable Python exception.

This is the remaining unsynchronized residency boundary described in #65; the other major unmap paths already synchronize around their changes.

Impact

The synchronization trades some asynchronous overlap at model page-fault boundaries for deterministic page-lifetime safety. It does not alter exported APIs, page accounting, or wheel compatibility.

Validation

Branch is based on the exact v0.4.10 release commit (ace72ab).
Diff is limited to seven added lines in src/model-vbar.c.
GitHub Actions validation is pending on this draft PR.

Addresses #65.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-aimdo/pull/77 **Author:** [@I-spyne-I](https://github.com/I-spyne-I) **Created:** 7/26/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `agent/fix-vbar-fault-sync` --- ### 📝 Commits (1) - [`825b7a5`](https://github.com/Comfy-Org/comfy-aimdo/commit/825b7a587caa4253f2510682a3e9ee474772f197) Synchronize CUDA before VBAR fault eviction ### 📊 Changes **1 file changed** (+7 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `src/model-vbar.c` (+7 -0) </details> ### 📄 Description ## What changed Add a CUDA context synchronization boundary at the start of `vbar_fault()`, after selecting the AIMDO device context and before any budget-driven residency changes can occur. ## Root cause `vbar_fault()` can run while a kernel launched for the preceding layer is still consuming VBAR-backed pages. Its budget enforcement path may then evict and unmap those pages. That creates a use-after-unmap race in the CUDA virtual-memory layer, which can manifest as nondeterministic inference corruption, a stuck first sampler step, or a Windows `nvlddmkm` TDR/reset rather than a recoverable Python exception. This is the remaining unsynchronized residency boundary described in #65; the other major unmap paths already synchronize around their changes. ## Impact The synchronization trades some asynchronous overlap at model page-fault boundaries for deterministic page-lifetime safety. It does not alter exported APIs, page accounting, or wheel compatibility. ## Validation - Branch is based on the exact `v0.4.10` release commit (`ace72ab`). - Diff is limited to seven added lines in `src/model-vbar.c`. - GitHub Actions validation is pending on this draft PR. Addresses #65. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:24:35 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#104

No description provided.

Rows
Columns