[PR #424] [CLOSED] fix(nvidia): zero-copy via DMABuf + glupload on Blackwell (sm_120+) #457

New issue

Closed

opened 2026-10-15 13:00:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 13:00:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/wolf/pull/424
Author: @JBailes
Created: 6/8/2026
Status: ❌ Closed

Base: stable ← Head: fix/nvidia-blackwell-zero-copy

📝 Commits (1)

2f22ef9 fix(nvidia): zero-copy via DMABuf + glupload on Blackwell (sm_120+)

📊 Changes

1 file changed (+54 additions, -1 deletions)

View changed files

📝 src/moonlight-server/state/configTOML.cpp (+54 -1)

📄 Description

Problem

On NVIDIA Blackwell GPUs (RTX 50-series, compute capability 12.x / sm_120) the zero-copy CUDA encode pipeline fails to negotiate. The virtual compositor hands GStreamer a video/x-raw(memory:CUDAMemory) buffer in BGRA, and cudaconvertscale cannot transform it on this architecture:

cudaconvertscale ... could not transform video/x-raw(memory:CUDAMemory), format=BGRA ... not negotiated

The result is a black screen that drops the session. The same pipeline works on older NVIDIA GPUs (Ampere/Ada), so it is architecture-specific. Reported in #417 (also seen on RTX 5070 Ti / RTX PRO 4000).

Fix

Keep zero-copy on Blackwell, but stop relying on the broken direct CUDAMemory interop. Instead, take the frame out of the compositor as DMABuf and import it entirely on the GPU in the encoder:

glupload      (DMABuf  -> GLMemory)
cudaupload    (GLMemory -> CUDAMemory)
cudaconvertscale ...

cudaupload can't import DMABuf directly, which is why glupload sits in front of it. The GL->CUDA step uses CUDA-GL interop and negotiates correctly on Blackwell (it does emit a harmless OpenGL context could not support PBO download warning, then falls back to the working interop path).

Implementation (state/configTOML.cpp):

Detect the GPU compute capability via nvidia-smi. On >= 12 (Blackwell+), set the producer buffer caps to video/x-raw(memory:DMABuf) and prepend glupload to the zero-copy encoder params.
Every other NVIDIA GPU keeps the existing direct video/x-raw(memory:CUDAMemory) zero-copy path unchanged. No effect on AMD/Intel.

No frames touch system memory, so this stays a true zero-copy path on both old and new NVIDIA.

Testing

RTX 5080 (driver 610.43.02, CUDA 13, sm_120): without the patch the stream black-screens with the not negotiated error above; with the patch the log shows Using zero copy pipeline on Nvidia, the encoder runs glupload -> cudaupload -> cudaconvertscale, and games stream correctly.
The GL->CUDA chain was verified standalone on the same GPU (gltestsrc ! glupload ! cudaupload ! cudaconvertscale ! nvh264enc reaches EOS).
Pre-Blackwell NVIDIA is unaffected (compute capability < 12 keeps the direct CUDAMemory path).

Closes #417

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/wolf/pull/424 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 6/8/2026 **Status:** ❌ Closed **Base:** `stable` ← **Head:** `fix/nvidia-blackwell-zero-copy` --- ### 📝 Commits (1) - [`2f22ef9`](https://github.com/games-on-whales/wolf/commit/2f22ef9040514df4b03a8e78f1a116474f09c267) fix(nvidia): zero-copy via DMABuf + glupload on Blackwell (sm_120+) ### 📊 Changes **1 file changed** (+54 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `src/moonlight-server/state/configTOML.cpp` (+54 -1) </details> ### 📄 Description ## Problem On NVIDIA **Blackwell** GPUs (RTX 50-series, compute capability 12.x / sm_120) the zero-copy CUDA encode pipeline fails to negotiate. The virtual compositor hands GStreamer a `video/x-raw(memory:CUDAMemory)` buffer in **BGRA**, and `cudaconvertscale` cannot transform it on this architecture: ``` cudaconvertscale ... could not transform video/x-raw(memory:CUDAMemory), format=BGRA ... not negotiated ``` The result is a black screen that drops the session. The same pipeline works on older NVIDIA GPUs (Ampere/Ada), so it is architecture-specific. Reported in #417 (also seen on RTX 5070 Ti / RTX PRO 4000). ## Fix Keep **zero-copy** on Blackwell, but stop relying on the broken *direct* CUDAMemory interop. Instead, take the frame out of the compositor as **DMABuf** and import it entirely on the GPU in the encoder: ``` glupload (DMABuf -> GLMemory) cudaupload (GLMemory -> CUDAMemory) cudaconvertscale ... ``` `cudaupload` can't import DMABuf directly, which is why `glupload` sits in front of it. The GL->CUDA step uses CUDA-GL interop and negotiates correctly on Blackwell (it does emit a harmless `OpenGL context could not support PBO download` warning, then falls back to the working interop path). Implementation (`state/configTOML.cpp`): - Detect the GPU compute capability via `nvidia-smi`. On `>= 12` (Blackwell+), set the producer buffer caps to `video/x-raw(memory:DMABuf)` and prepend `glupload` to the zero-copy encoder params. - Every other NVIDIA GPU keeps the existing direct `video/x-raw(memory:CUDAMemory)` zero-copy path unchanged. No effect on AMD/Intel. No frames touch system memory, so this stays a true zero-copy path on both old and new NVIDIA. ## Testing - **RTX 5080** (driver 610.43.02, CUDA 13, sm_120): without the patch the stream black-screens with the `not negotiated` error above; with the patch the log shows `Using zero copy pipeline on Nvidia`, the encoder runs `glupload -> cudaupload -> cudaconvertscale`, and games stream correctly. - The GL->CUDA chain was verified standalone on the same GPU (`gltestsrc ! glupload ! cudaupload ! cudaconvertscale ! nvh264enc` reaches EOS). - Pre-Blackwell NVIDIA is unaffected (compute capability `< 12` keeps the direct CUDAMemory path). Closes #417 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 13:00:43 +00:00