games-on-whales/wolf

Fork 0

mirror of https://github.com/games-on-whales/wolf.git synced 2026-10-15 21:12:08 +00:00

[GH-ISSUE #501] Zero-copy NVIDIA pipeline crashes (GsCUDABuf/apply_video_info) on 2nd+ app launch — WOLF_USE_ZERO_COPY=FALSE works around it #305

New issue

Open

opened 2026-10-15 12:59:30 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 12:59:30 +00:00

Owner

Originally created by @jdonaldson on GitHub (Sep 14, 2026).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/501

Summary

Under the zero-copy NVIDIA pipeline (default, WOLF_USE_ZERO_COPY unset), the first app launched in a Wolf process streams fine, but every subsequent app launch in the same process crashes with a panic in GsCUDABuf/apply_video_info, and the crash is severe enough that it sometimes segfaults the whole wolf process (exit status 11), auto-restarted by supervisord.

Workaround confirmed: setting WOLF_USE_ZERO_COPY=FALSE avoids the crash entirely — verified stable with two app containers (Wolf-UI and a custom Steam app) running concurrently, where the zero-copy path had crashed on the second launch every time it was tried.

Filing this because AGENTS.md documents that zero-copy is supposed to "auto-fall back to legacy when an encoder can't support it" — that fallback doesn't appear to be triggering here even though the zero-copy path is reliably broken in this environment, so I think the auto-fallback logic (rather than just the crash itself) is the actual bug.

Environment

GPU: RTX 4090
Driver: 550.163.01 (proprietary, not open kernel modules)
Host: Debian 13 (trixie), standard linux-image-amd64 kernel (6.12.107+deb13-amd64) — not the Debian cloud kernel flavor, which lacks uinput and working nvidia_drm modeset entirely (separate issue, not filing that one, just noting for reproduction)
Wolf image: ghcr.io/games-on-whales/wolf:stable
Installation method: the documented "NVIDIA manual method" (gow/nvidia-driver volume + NVIDIA_DRIVER_VOLUME_NAME), not nvidia-container-toolkit/runtime: nvidia — switched to this specifically because the container-toolkit method hit the same GsCUDABuf crash on the very first launch (matches #379), and the manual method fixed that specific case
Docker 29.8.0, network_mode: host, container run privileged: true

wolf:stable's image bundles a stale /usr/local/nvidia/lib/libnvrtc.so.11.0 dated Aug 2020 — neither the container-toolkit method nor the manual driver-volume method touches this path (NVRTC is a CUDA Toolkit component, not shipped by either the driver .run installer or the container toolkit's device injection). Every boot logs:

Failed to load 'nvrtcGetCUBINSize', 'nvrtcGetCUBINSize': /usr/local/nvidia/lib/libnvrtc.so: undefined symbol: nvrtcGetCUBINSize

(nvrtcGetCUBINSize was added in CUDA 11.2 — the bundled file is CUDA 11.0.) nvcodec still gets selected as the encoder despite this warning, so it's not obviously fatal on its own, but I wonder if it's what's actually causing the zero-copy path's CUDA context/buffer allocation to be unreliable, which would explain why the auto-fallback should be kicking in but isn't. I tried bind-mounting a real CUDA 12 libnvrtc.so over the stale file as a workaround — that made things worse (broke cudaconvertscale entirely, forcing full software encoding), presumably an ABI mismatch with sibling libraries in that same bundled set, so that's not a viable fix on its own.

Repro steps

Provision Wolf via the manual NVIDIA driver-volume method (quickstart docs), WOLF_USE_ZERO_COPY unset (default/on).
Pair a Moonlight client, launch any app (e.g. the built-in "Wolf UI") — streams successfully.
Without fully tearing down that session, launch a second app (tried: a custom Steam app entry, and separately the built-in Firefox app — both fail identically).
Second app's container starts then immediately stops (~0.5–0.6s later). Wolf's log shows the crash below at the moment of the second launch.

Note: the first app's underlying container/session does not appear to fully tear down when the Moonlight client just navigates back to the app grid — Stopped container: /Wolf-UI_... in the log lands ~20-30s after the second app's launch already failed, suggesting overlapping GPU/CUDA context lifetimes between the two sessions. Fully disconnecting the Moonlight client (not just backing out to the app grid) before the second launch did not avoid the crash either.

Crash (zero-copy path, default)

thread '<unnamed>' (238) panicked at wayland-display-core/src/comp/mod.rs:395:18:
Failed to create GsCUDABuf

and, in another occurrence of the same underlying crash:

(wolf:240): GStreamer-Cuda-CRITICAL **: gboolean gst_cuda_context_push(GstCudaContext*): assertion 'GST_IS_CUDA_CONTEXT (ctx)' failed
(wolf:240): GStreamer-Cuda-CRITICAL **: GstBufferPool* gst_cuda_buffer_pool_new(GstCudaContext*): assertion 'GST_IS_CUDA_CONTEXT (context)' failed

thread '<unnamed>' (306) panicked at wayland-display-core/src/utils/allocator/mod.rs:182:26:
Failed to create CUDA image from EGLImage: "Failed to push CUDA context"

stack backtrace:
  11:  waylanddisplaycore::utils::allocator::GsCUDABuf::new
  12:  waylanddisplaycore::comp::apply_video_info
  ...
  17:  waylanddisplaycore::comp::init

Both times, immediately followed by:

0:xx:xx.xxxxxxxxx WARN waylanddisplaysrc: message = Failed to recv buffer ack.; err = RecvError;
0:xx:xx.xxxxxxxxx WARN basesrc: error: Internal data stream error.
0:xx:xx.xxxxxxxxx WARN basesrc: error: streaming stopped, reason error (-5)
ERROR | [GSTREAMER] Pipeline error: Internal data stream error.

On at least one occurrence, this escalated to a full wolf process crash, auto-restarted by supervisord:

2026-09-14 15:52:38,667 WARN exited: wolf (exit status 11; not expected)
2026-09-14 15:52:39,669 INFO spawned: 'wolf' with pid 205

...and a separate shutdown-time stack trace directly implicated /usr/nvidia/lib/libcuda.so.1 (not just Wolf's own code) in a SIGTERM-triggered crash, though I don't have the full trace saved for that one.

I also saw, on a different attempt, a non-panicking but still fatal caps-negotiation failure at this same point, reproduced identically at three different resolutions (1280×720, 1920×1080, and 3456×2160 — ruling out resolution as a variable):

WARN basetransform: transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
WARN basesrc: error: streaming stopped, reason not-negotiated (-4)

Confirmed working with `WOLF_USE_ZERO_COPY=FALSE`

Same environment, same two-app sequence, WOLF_USE_ZERO_COPY=FALSE set:

INFO | Using legacy pipeline on Nvidia (/dev/dri/renderD128)

Both Wolf-UI and the Steam app containers ran concurrently (docker ps showing both Up) with no crash. There was a similar-looking transient warning during the second app's launch (transform could not transform video/x-raw, format=(string)RGBx... warning: not negotiated), but unlike the zero-copy path, the pipeline recovered and the container stayed up rather than exiting.

Ask

Is the zero-copy CUDA path known to be fragile with concurrent/sequential sessions on consumer GPUs (RTX 4090) with driver 550.163.01, or is this specific to something in my setup?
Should the documented auto-fallback from zero-copy to legacy be triggering here? If the stale bundled libnvrtc.so.11.0 is the actual root cause (rather than something inherent to zero-copy + concurrent sessions), a fix there might be higher-leverage than fixing the fallback logic alone.

Happy to gather more diagnostics (full logs, WOLF_LOG_LEVEL=DEBUG output, etc.) — this environment is still up and I can reproduce reliably.

Diagnosed with help from Claude Code (Anthropic) — I've verified the log excerpts and repro steps above against my own session, but flagging the AI assistance for transparency.

Originally created by @jdonaldson on GitHub (Sep 14, 2026). Original GitHub issue: https://github.com/games-on-whales/wolf/issues/501 ## Summary Under the zero-copy NVIDIA pipeline (default, `WOLF_USE_ZERO_COPY` unset), the first app launched in a Wolf process streams fine, but **every subsequent app launch in the same process crashes** with a panic in `GsCUDABuf`/`apply_video_info`, and the crash is severe enough that it sometimes segfaults the whole `wolf` process (exit status 11), auto-restarted by supervisord. **Workaround confirmed**: setting `WOLF_USE_ZERO_COPY=FALSE` avoids the crash entirely — verified stable with two app containers (`Wolf-UI` and a custom `Steam` app) running concurrently, where the zero-copy path had crashed on the second launch every time it was tried. Filing this because `AGENTS.md` documents that zero-copy is supposed to "auto-fall back to legacy when an encoder can't support it" — that fallback doesn't appear to be triggering here even though the zero-copy path is reliably broken in this environment, so I think the auto-fallback logic (rather than just the crash itself) is the actual bug. ## Environment - GPU: RTX 4090 - Driver: 550.163.01 (proprietary, not open kernel modules) - Host: Debian 13 (trixie), standard `linux-image-amd64` kernel (6.12.107+deb13-amd64) — **not** the Debian cloud kernel flavor, which lacks `uinput` and working `nvidia_drm` modeset entirely (separate issue, not filing that one, just noting for reproduction) - Wolf image: `ghcr.io/games-on-whales/wolf:stable` - Installation method: the documented "NVIDIA manual method" (`gow/nvidia-driver` volume + `NVIDIA_DRIVER_VOLUME_NAME`), not `nvidia-container-toolkit`/`runtime: nvidia` — switched to this specifically because the container-toolkit method hit the same `GsCUDABuf` crash on the very first launch (matches #379), and the manual method fixed *that* specific case - Docker 29.8.0, `network_mode: host`, container run `privileged: true` ## Secondary finding, possibly related `wolf:stable`'s image bundles a stale `/usr/local/nvidia/lib/libnvrtc.so.11.0` dated **Aug 2020** — neither the container-toolkit method nor the manual driver-volume method touches this path (NVRTC is a CUDA Toolkit component, not shipped by either the driver `.run` installer or the container toolkit's device injection). Every boot logs: ``` Failed to load 'nvrtcGetCUBINSize', 'nvrtcGetCUBINSize': /usr/local/nvidia/lib/libnvrtc.so: undefined symbol: nvrtcGetCUBINSize ``` (`nvrtcGetCUBINSize` was added in CUDA 11.2 — the bundled file is CUDA 11.0.) `nvcodec` still gets selected as the encoder despite this warning, so it's not obviously fatal on its own, but I wonder if it's what's actually causing the zero-copy path's CUDA context/buffer allocation to be unreliable, which would explain why the auto-fallback should be kicking in but isn't. I tried bind-mounting a real CUDA 12 `libnvrtc.so` over the stale file as a workaround — that made things *worse* (broke `cudaconvertscale` entirely, forcing full software encoding), presumably an ABI mismatch with sibling libraries in that same bundled set, so that's not a viable fix on its own. ## Repro steps 1. Provision Wolf via the manual NVIDIA driver-volume method (quickstart docs), `WOLF_USE_ZERO_COPY` unset (default/on). 2. Pair a Moonlight client, launch any app (e.g. the built-in "Wolf UI") — streams successfully. 3. Without fully tearing down that session, launch a second app (tried: a custom Steam app entry, and separately the built-in Firefox app — both fail identically). 4. Second app's container starts then immediately stops (~0.5–0.6s later). Wolf's log shows the crash below at the moment of the second launch. Note: the first app's underlying container/session does **not** appear to fully tear down when the Moonlight client just navigates back to the app grid — `Stopped container: /Wolf-UI_...` in the log lands ~20-30s *after* the second app's launch already failed, suggesting overlapping GPU/CUDA context lifetimes between the two sessions. Fully disconnecting the Moonlight client (not just backing out to the app grid) before the second launch did **not** avoid the crash either. ## Crash (zero-copy path, default) ``` thread '<unnamed>' (238) panicked at wayland-display-core/src/comp/mod.rs:395:18: Failed to create GsCUDABuf ``` and, in another occurrence of the same underlying crash: ``` (wolf:240): GStreamer-Cuda-CRITICAL **: gboolean gst_cuda_context_push(GstCudaContext*): assertion 'GST_IS_CUDA_CONTEXT (ctx)' failed (wolf:240): GStreamer-Cuda-CRITICAL **: GstBufferPool* gst_cuda_buffer_pool_new(GstCudaContext*): assertion 'GST_IS_CUDA_CONTEXT (context)' failed thread '<unnamed>' (306) panicked at wayland-display-core/src/utils/allocator/mod.rs:182:26: Failed to create CUDA image from EGLImage: "Failed to push CUDA context" stack backtrace: 11: waylanddisplaycore::utils::allocator::GsCUDABuf::new 12: waylanddisplaycore::comp::apply_video_info ... 17: waylanddisplaycore::comp::init ``` Both times, immediately followed by: ``` 0:xx:xx.xxxxxxxxx WARN waylanddisplaysrc: message = Failed to recv buffer ack.; err = RecvError; 0:xx:xx.xxxxxxxxx WARN basesrc: error: Internal data stream error. 0:xx:xx.xxxxxxxxx WARN basesrc: error: streaming stopped, reason error (-5) ERROR | [GSTREAMER] Pipeline error: Internal data stream error. ``` On at least one occurrence, this escalated to a full `wolf` process crash, auto-restarted by supervisord: ``` 2026-09-14 15:52:38,667 WARN exited: wolf (exit status 11; not expected) 2026-09-14 15:52:39,669 INFO spawned: 'wolf' with pid 205 ``` ...and a separate shutdown-time stack trace directly implicated `/usr/nvidia/lib/libcuda.so.1` (not just Wolf's own code) in a SIGTERM-triggered crash, though I don't have the full trace saved for that one. I also saw, on a different attempt, a *non-panicking* but still fatal caps-negotiation failure at this same point, reproduced identically at **three different resolutions** (1280×720, 1920×1080, and 3456×2160 — ruling out resolution as a variable): ``` WARN basetransform: transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support WARN basesrc: error: streaming stopped, reason not-negotiated (-4) ``` ## Confirmed working with `WOLF_USE_ZERO_COPY=FALSE` Same environment, same two-app sequence, `WOLF_USE_ZERO_COPY=FALSE` set: ``` INFO | Using legacy pipeline on Nvidia (/dev/dri/renderD128) ``` Both `Wolf-UI` and the Steam app containers ran concurrently (`docker ps` showing both `Up`) with no crash. There was a similar-looking transient warning during the second app's launch (`transform could not transform video/x-raw, format=(string)RGBx... warning: not negotiated`), but unlike the zero-copy path, the pipeline recovered and the container stayed up rather than exiting. ## Ask - Is the zero-copy CUDA path known to be fragile with concurrent/sequential sessions on consumer GPUs (RTX 4090) with driver 550.163.01, or is this specific to something in my setup? - Should the documented auto-fallback from zero-copy to legacy be triggering here? If the stale bundled `libnvrtc.so.11.0` is the actual root cause (rather than something inherent to zero-copy + concurrent sessions), a fix there might be higher-leverage than fixing the fallback logic alone. Happy to gather more diagnostics (full logs, `WOLF_LOG_LEVEL=DEBUG` output, etc.) — this environment is still up and I can reproduce reliably. --- Diagnosed with help from Claude Code (Anthropic) — I've verified the log excerpts and repro steps above against my own session, but flagging the AI assistance for transparency.

bsakaguchi commented

2026-10-15 12:59:31 +00:00

Author

Owner

@r0b0tb4c0n commented on GitHub (Sep 18, 2026):

i think i ran into this same issue yesterday/today and didn't see your issue. i made a pr but if you'd be willing to test to see if it fixes your issue as well, i'd love to have another set of eyes on it. the issue i opened is #504 let me know if you need help building the image. I'm using nvidia-open 615.71.09-1 but our errors seem to be the same

@r0b0tb4c0n commented on GitHub (Sep 18, 2026): i think i ran into this same issue yesterday/today and didn't see your issue. i made a pr but if you'd be willing to test to see if it fixes your issue as well, i'd love to have another set of eyes on it. the issue i opened is #504 let me know if you need help building the image. I'm using nvidia-open 615.71.09-1 but our errors seem to be the same

bsakaguchi commented

2026-10-15 12:59:31 +00:00

Author

Owner

@jdonaldson commented on GitHub (Sep 21, 2026):

Thanks for the offer, @r0b0tb4c0n — much appreciated, and sorry for the slow reply.

I built both images to A/B your #505 (stable @ 2273b2e, and the same commit plus your patch, so they differ only by the fix). Before running it I went back through the resume path, and I now think the root cause sits a layer above the CUDA context.

The trigger in both our reports looks like re-entering a session that is still running — not app-switching, which is how I originally described it in this issue. That also explains the asymmetry you spotted: closing with the ✕ leaves the session up, so the next connect is a resume, while ctrl-alt-shift-q ends it, so the next connect is a clean launch.

Two separate things in resume() (rest/endpoints.hpp):

It double-responds on success — the trailing server_error() is at function scope, so a successful resume emits 200 + resume XML followed by 400 + error XML. Filed as #509, fix in #510. Cosmetic next to the crash, but it means a successful resume and a failed one look identical on the wire, which makes this path miserable to debug.
It never stops the old pipeline. It swaps the session record and moves the wayland_display and input devices onto a new session id, but fires neither PauseStreamEvent nor StopStreamEvent — those only come from the control channel and the API. Meanwhile the PauseStreamEvent handler in streaming/streaming.cpp states the requirement outright: "The only solution is to kill the pipeline and re-create it again when a resume happens." So the old pipeline is left running on the same Wayland display, and its session id is already out of running_sessions, so nothing can stop it by id afterwards.

Observed on my box: a Wolf-UI stream ended at 01:16:14 and Stopped container: /Wolf-UI_... did not appear until 11:39:20 — the session lingered about ten hours. Any reconnect inside that window is a re-entry into a live session.

If that reading is right, #505 helps on one side of a race — cached context already destroyed, cache still pointing at it — but when the old pipeline is still alive and owns the context, the validity probe passes, the context is reused for a second concurrent pipeline, and you get the original collision. Worth noting too that your driver is nvidia-open 615.71.09 and mine is 550.163.01 proprietary, so we may not even be landing on the same side of that race.

So my plan is to try correcting the underlying lifetime issue first, then treat the symptom, and separately give the user better ground truth about what happened — as it stands, a re-entry neither resumes cleanly nor tells you the session is still up. I'll still report back with the #505 A/B either way; the two aren't mutually exclusive and your guard looks reasonable regardless of what sits underneath it.

_{Analysis assisted by Claude Code; code references and timestamps above were checked against the repository and my own logs before posting.}

@jdonaldson commented on GitHub (Sep 21, 2026): Thanks for the offer, @r0b0tb4c0n — much appreciated, and sorry for the slow reply. I built both images to A/B your #505 (stable @ `2273b2e`, and the same commit plus your patch, so they differ only by the fix). Before running it I went back through the resume path, and I now think the root cause sits a layer above the CUDA context. The trigger in both our reports looks like **re-entering a session that is still running** — not app-switching, which is how I originally described it in this issue. That also explains the asymmetry you spotted: closing with the ✕ leaves the session up, so the next connect is a *resume*, while `ctrl-alt-shift-q` ends it, so the next connect is a clean *launch*. Two separate things in `resume()` (`rest/endpoints.hpp`): 1. **It double-responds on success** — the trailing `server_error()` is at function scope, so a successful resume emits `200` + resume XML followed by `400` + error XML. Filed as #509, fix in #510. Cosmetic next to the crash, but it means a successful resume and a failed one look identical on the wire, which makes this path miserable to debug. 2. **It never stops the old pipeline.** It swaps the session record and moves the `wayland_display` and input devices onto a new session id, but fires neither `PauseStreamEvent` nor `StopStreamEvent` — those only come from the control channel and the API. Meanwhile the `PauseStreamEvent` handler in `streaming/streaming.cpp` states the requirement outright: *"The only solution is to kill the pipeline and re-create it again when a resume happens."* So the old pipeline is left running on the same Wayland display, and its session id is already out of `running_sessions`, so nothing can stop it by id afterwards. Observed on my box: a Wolf-UI stream ended at `01:16:14` and `Stopped container: /Wolf-UI_...` did not appear until `11:39:20` — the session lingered about ten hours. Any reconnect inside that window is a re-entry into a live session. If that reading is right, #505 helps on one side of a race — cached context already destroyed, cache still pointing at it — but when the old pipeline is still alive and owns the context, the validity probe passes, the context is reused for a second concurrent pipeline, and you get the original collision. Worth noting too that your driver is nvidia-open 615.71.09 and mine is 550.163.01 proprietary, so we may not even be landing on the same side of that race. So my plan is to try correcting the underlying lifetime issue first, then treat the symptom, and separately give the user better ground truth about what happened — as it stands, a re-entry neither resumes cleanly nor tells you the session is still up. I'll still report back with the #505 A/B either way; the two aren't mutually exclusive and your guard looks reasonable regardless of what sits underneath it. <sub>Analysis assisted by Claude Code; code references and timestamps above were checked against the repository and my own logs before posting.</sub>

bsakaguchi referenced this issue

2026-10-15 13:00:20 +00:00

[PR #305] Pause/Resume containers #407