[PR #38] Reduce memory pressure and fix DLPack lifetime in RTX VSR #42

New issue

Open

opened 2026-10-26 22:28:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:28:20 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/ComfyUI_Nvidia_VFX_Nodes/pull/38
Author: @xmarre
Created: 8/22/2026
Status: 🔄 Open

Base: main ← Head: upstream/reduce-rtx-vsr-memory-pressure

📝 Commits (1)

46ffadb Reduce RTX VSR memory pressure and fix DLPack lifetime

📊 Changes

2 files changed (+144 additions, -20 deletions)

View changed files

📝 README.md (+14 -2)
📝 __init__.py (+130 -18)

📄 Description

Summary

This reduces memory pressure in long/high-resolution RTX Video Super Resolution workflows and fixes the NVVFX DLPack output lifetime without adding per-frame CUDA allocator flushes.

The change is intentionally limited to the existing RTX VSR node and its README documentation.

Problems addressed

The current implementation has three separate memory/lifetime costs:

The returned IMAGE tensor is allocated on images.device. In normal CPU-backed ComfyUI image chains, that materializes the entire upscaled video in system RAM even when the next consumer can accept a CUDA IMAGE tensor.
Frames are grouped by pixel count and the whole group is converted to float32 CUDA before inference, so temporary CUDA memory scales with the selected batch rather than one frame.
VideoSuperRes.run() returns DLPack-backed storage owned by NVVFX. The frame must establish PyTorch-owned storage before that effect-owned buffer can be reused or torn down. This is the lifetime issue tracked in #35 and targeted by #36.

For video chains that already perform a latent spatial upscale before VSR, the final pixel tensor can become very large. A 2x latent spatial upscale followed by 2x VSR produces 16x the original pixel count, so unnecessarily retaining another complete float32 RGB video in host memory can mean tens of GiB.

Changes

Process VSR exactly one frame at a time instead of pre-staging a multi-frame float32 CUDA batch.
Build each NVVFX input as one owned contiguous float32 CUDA frame rather than a .to(...).contiguous() chain that can require two CUDA temporaries.
Clone each NVVFX DLPack result immediately, keep the NVVFX result alive through the copy, and release the per-frame references before the next run().
Copy the owned output frame directly into the preallocated result tensor with copy_, including device/dtype conversion, avoiding another full-frame output_frame.to(...) temporary.
Do not call torch.cuda.empty_cache() per frame. Released PyTorch blocks remain reusable by the caching allocator without forcing a synchronization/allocation cycle every frame. This incorporates the outstanding review feedback on #36.
Add an append-only output_device control:
- auto (default): keep the full result on CUDA only when current free VRAM can hold the output, peak per-frame scratch, and conservative headroom; otherwise use CPU.
- cuda: explicitly keep the complete result in VRAM.
- cpu: explicitly keep the complete result in system RAM.
Resolve auto only after VideoSuperRes.load() so NVVFX resident allocations are already reflected in the free-VRAM measurement.
Account for the actual explicit CUDA peak in auto: one float32 input frame plus the simultaneously-live NVVFX float32 output and owned float32 clone, in addition to the complete returned tensor and conservative headroom.
Perform the fit check even when the input is already CUDA, since the larger VSR output may not fit just because its input does.
Preserve the input dtype in the returned IMAGE tensor.
Validate the [frames, height, width, channels] contract and preserve empty inputs.
Document the memory/output-device behavior.

Compatibility

nvvfx.VideoSuperRes remains the inference backend.
Quality mapping is unchanged.
Scale-by and target-dimensions behavior is unchanged, including 8-pixel output alignment.
The node still returns one ComfyUI IMAGE tensor.
output_device is appended to the schema and execute() supplies a default, so existing workflows that do not serialize the new input continue to have a valid call signature.
Explicit CPU output remains available for downstream nodes that require host tensors.

VideoHelperSuite's current standard FFmpeg path iterates the IMAGE tensor frame by frame and converts each frame to CPU before writing it to FFmpeg. That path therefore does not require a second complete floating-point host tensor when this node returns CUDA output. Formats that require a pre-pass may retain encoded frame bytes separately.

Validation

Based directly on current upstream main (892515e3, v0.1.3); the PR branch is exactly one commit ahead with no upstream drift.
Manually exercised in the motivating long/high-resolution H3 -> RTX VSR workflow; the CUDA-output/DLPack-lifetime path is working successfully.
Review follow-up validated the peak-allocation accounting and the PyTorch copy_/contiguous-copy behavior used by the final implementation.
No GPU CI is available in this repository to exercise NVVFX inference automatically.

This includes the core owned-DLPack fix from #36 while also addressing the independent host-memory and batch-staging pressure. It deliberately omits #36's per-frame torch.cuda.empty_cache() because that call was also flagged in review as unnecessary allocator churn.

Closes #35.
Addresses #16.
Related: #36.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/ComfyUI_Nvidia_VFX_Nodes/pull/38 **Author:** [@xmarre](https://github.com/xmarre) **Created:** 8/22/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `upstream/reduce-rtx-vsr-memory-pressure` --- ### 📝 Commits (1) - [`46ffadb`](https://github.com/Comfy-Org/ComfyUI_Nvidia_VFX_Nodes/commit/46ffadbdb5b5a9f58342bbfaafdf22aa79084606) Reduce RTX VSR memory pressure and fix DLPack lifetime ### 📊 Changes **2 files changed** (+144 additions, -20 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+14 -2) 📝 `__init__.py` (+130 -18) </details> ### 📄 Description ## Summary This reduces memory pressure in long/high-resolution `RTX Video Super Resolution` workflows and fixes the NVVFX DLPack output lifetime without adding per-frame CUDA allocator flushes. The change is intentionally limited to the existing RTX VSR node and its README documentation. ## Problems addressed The current implementation has three separate memory/lifetime costs: 1. The returned `IMAGE` tensor is allocated on `images.device`. In normal CPU-backed ComfyUI image chains, that materializes the entire upscaled video in system RAM even when the next consumer can accept a CUDA `IMAGE` tensor. 2. Frames are grouped by pixel count and the whole group is converted to float32 CUDA before inference, so temporary CUDA memory scales with the selected batch rather than one frame. 3. `VideoSuperRes.run()` returns DLPack-backed storage owned by NVVFX. The frame must establish PyTorch-owned storage before that effect-owned buffer can be reused or torn down. This is the lifetime issue tracked in #35 and targeted by #36. For video chains that already perform a latent spatial upscale before VSR, the final pixel tensor can become very large. A 2x latent spatial upscale followed by 2x VSR produces 16x the original pixel count, so unnecessarily retaining another complete float32 RGB video in host memory can mean tens of GiB. ## Changes - Process VSR exactly one frame at a time instead of pre-staging a multi-frame float32 CUDA batch. - Build each NVVFX input as one owned contiguous float32 CUDA frame rather than a `.to(...).contiguous()` chain that can require two CUDA temporaries. - Clone each NVVFX DLPack result immediately, keep the NVVFX result alive through the copy, and release the per-frame references before the next `run()`. - Copy the owned output frame directly into the preallocated result tensor with `copy_`, including device/dtype conversion, avoiding another full-frame `output_frame.to(...)` temporary. - Do **not** call `torch.cuda.empty_cache()` per frame. Released PyTorch blocks remain reusable by the caching allocator without forcing a synchronization/allocation cycle every frame. This incorporates the outstanding review feedback on #36. - Add an append-only `output_device` control: - `auto` (default): keep the full result on CUDA only when current free VRAM can hold the output, peak per-frame scratch, and conservative headroom; otherwise use CPU. - `cuda`: explicitly keep the complete result in VRAM. - `cpu`: explicitly keep the complete result in system RAM. - Resolve `auto` only after `VideoSuperRes.load()` so NVVFX resident allocations are already reflected in the free-VRAM measurement. - Account for the actual explicit CUDA peak in `auto`: one float32 input frame plus the simultaneously-live NVVFX float32 output and owned float32 clone, in addition to the complete returned tensor and conservative headroom. - Perform the fit check even when the input is already CUDA, since the larger VSR output may not fit just because its input does. - Preserve the input dtype in the returned `IMAGE` tensor. - Validate the `[frames, height, width, channels]` contract and preserve empty inputs. - Document the memory/output-device behavior. ## Compatibility - `nvvfx.VideoSuperRes` remains the inference backend. - Quality mapping is unchanged. - Scale-by and target-dimensions behavior is unchanged, including 8-pixel output alignment. - The node still returns one ComfyUI `IMAGE` tensor. - `output_device` is appended to the schema and `execute()` supplies a default, so existing workflows that do not serialize the new input continue to have a valid call signature. - Explicit CPU output remains available for downstream nodes that require host tensors. VideoHelperSuite's current standard FFmpeg path iterates the `IMAGE` tensor frame by frame and converts each frame to CPU before writing it to FFmpeg. That path therefore does not require a second complete floating-point host tensor when this node returns CUDA output. Formats that require a pre-pass may retain encoded frame bytes separately. ## Validation - Based directly on current upstream `main` (`892515e3`, v0.1.3); the PR branch is exactly one commit ahead with no upstream drift. - Manually exercised in the motivating long/high-resolution H3 -> RTX VSR workflow; the CUDA-output/DLPack-lifetime path is working successfully. - Review follow-up validated the peak-allocation accounting and the PyTorch `copy_`/contiguous-copy behavior used by the final implementation. - No GPU CI is available in this repository to exercise NVVFX inference automatically. This includes the core owned-DLPack fix from #36 while also addressing the independent host-memory and batch-staging pressure. It deliberately omits #36's per-frame `torch.cuda.empty_cache()` because that call was also flagged in review as unnecessary allocator churn. Closes #35. Addresses #16. Related: #36. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:28:20 +00:00

bsakaguchi referenced this issue

2026-10-26 22:28:20 +00:00

[PR #42] [MERGED] Update super resolution, add frame generation and true HDR #43