games-on-whales/wolf

Fork 0

mirror of https://github.com/games-on-whales/wolf.git synced 2026-10-15 13:02:03 +00:00

[GH-ISSUE #400] NVidia P100 GStreamer h264 Errors - transform could not transform #264

New issue

Closed

opened 2026-10-15 12:59:00 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented

2026-10-15 12:59:00 +00:00

Owner

Originally created by @tjacoby2006 on GitHub (Apr 24, 2026).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/400

I am having an issue with Wolf streaming using an NVidia P100. The Test Ball application works fine. It only occurs with h264 when running Wolf, HEVC works fine. I would stick with HEVC, but I have clients that will need h264.

wolf-1  | 0:09:56.539064374     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:56.551689486     1 0x71f718001700 WARN                 basesrc gstbasesrc.c:4255:gst_base_src_push_segment:<interpipesrc_14663631860551416512_video> segment format mismatched, ignore
wolf-1  | 0:09:56.551809702     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:56.551916262     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:56.551929086     1 0x71f718001700 WARN                 basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_14663631860551416512_video> error: Internal data stream error.
wolf-1  | 0:09:56.551933474     1 0x71f718001700 WARN                 basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_14663631860551416512_video> error: streaming stopped, reason not-negotiated (-4)
wolf-1  | 13:26:02.107304273 ERROR | [GSTREAMER] Pipeline error: Internal data stream error.
wolf-1  | 0:09:56.552040776     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:56.552130193     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:56.552222697     1 0x71f718001700 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
wolf-1  | 0:09:57.009520512     1 0x71f718001260 WARN                 basesrc gstbasesrc.c:4255:gst_base_src_push_segment:<interpipesrc_14663631860551416512_audio> segment format mismatched, ignore
wolf-1  | 13:26:12.815054932 INFO  | [GSTREAMER] Pipeline reached End Of Stream

services:
  wolf:
    image: ghcr.io/games-on-whales/wolf:stable
    environment:
      - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol
      - TZ=America/Detroit
      - WOLF_RENDER_NODE=/dev/dri/renderD128
    volumes:
      - ./data:/etc/wolf:rw
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
      - nvidia-driver-vol:/usr/nvidia:rw
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
      - /dev/nvidiactl
      - /dev/nvidia0
      - /dev/nvidia-modeset
    device_cgroup_rules:
      - 'c 13:* rmw'
    network_mode: host
    restart: unless-stopped

volumes:
  nvidia-driver-vol:
    external: true

[gstreamer.video]
default_sink = '''queue ! rtpmoonlightpay_video name=moonlight_pay payload_size={payload_size} fec_percentage={fec_percentage} min_required_fec_packets={min_required_fec_packets} !
appsink sync=false max-buffers=1 drop=true name=wolf_udp_sink
'''
default_source = 'interpipesrc name=interpipesrc_{}_video listen-to={session_id}_video is-live=true stream-sync=restart-ts max-bytes=0 max-buffers=1 leaky-type=downstream'

    [gstreamer.video.defaults.nvcodec]
    video_params = '''cudaupload !
cudaconvertscale add-borders=true !
video/x-raw(memory:CUDAMemory), width={width}, height={height}, chroma-site={color_range}, format=NV12, colorimetry={color_space}, pixel-aspect-ratio=1/1'''
    video_params_zero_copy = '''cudaupload !
cudaconvertscale add-borders=true !
video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1
'''

Originally created by @tjacoby2006 on GitHub (Apr 24, 2026). Original GitHub issue: https://github.com/games-on-whales/wolf/issues/400 I am having an issue with Wolf streaming using an NVidia P100. The Test Ball application works fine. It only occurs with h264 when running Wolf, HEVC works fine. I would stick with HEVC, but I have clients that will need h264. ```bash wolf-1 | 0:09:56.539064374 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:56.551689486 1 0x71f718001700 WARN basesrc gstbasesrc.c:4255:gst_base_src_push_segment:<interpipesrc_14663631860551416512_video> segment format mismatched, ignore wolf-1 | 0:09:56.551809702 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:56.551916262 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:56.551929086 1 0x71f718001700 WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_14663631860551416512_video> error: Internal data stream error. wolf-1 | 0:09:56.551933474 1 0x71f718001700 WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_14663631860551416512_video> error: streaming stopped, reason not-negotiated (-4) wolf-1 | 13:26:02.107304273 ERROR | [GSTREAMER] Pipeline error: Internal data stream error. wolf-1 | 0:09:56.552040776 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:56.552130193 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:56.552222697 1 0x71f718001700 WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale6> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support wolf-1 | 0:09:57.009520512 1 0x71f718001260 WARN basesrc gstbasesrc.c:4255:gst_base_src_push_segment:<interpipesrc_14663631860551416512_audio> segment format mismatched, ignore wolf-1 | 13:26:12.815054932 INFO | [GSTREAMER] Pipeline reached End Of Stream ``` ```yaml services: wolf: image: ghcr.io/games-on-whales/wolf:stable environment: - NVIDIA_DRIVER_VOLUME_NAME=nvidia-driver-vol - TZ=America/Detroit - WOLF_RENDER_NODE=/dev/dri/renderD128 volumes: - ./data:/etc/wolf:rw - /var/run/docker.sock:/var/run/docker.sock:rw - /dev/:/dev/:rw - /run/udev:/run/udev:rw - nvidia-driver-vol:/usr/nvidia:rw devices: - /dev/dri - /dev/uinput - /dev/uhid - /dev/nvidia-uvm - /dev/nvidia-uvm-tools - /dev/nvidia-caps/nvidia-cap1 - /dev/nvidia-caps/nvidia-cap2 - /dev/nvidiactl - /dev/nvidia0 - /dev/nvidia-modeset device_cgroup_rules: - 'c 13:* rmw' network_mode: host restart: unless-stopped volumes: nvidia-driver-vol: external: true ``` ```toml [gstreamer.video] default_sink = '''queue ! rtpmoonlightpay_video name=moonlight_pay payload_size={payload_size} fec_percentage={fec_percentage} min_required_fec_packets={min_required_fec_packets} ! appsink sync=false max-buffers=1 drop=true name=wolf_udp_sink ''' default_source = 'interpipesrc name=interpipesrc_{}_video listen-to={session_id}_video is-live=true stream-sync=restart-ts max-bytes=0 max-buffers=1 leaky-type=downstream' [gstreamer.video.defaults.nvcodec] video_params = '''cudaupload ! cudaconvertscale add-borders=true ! video/x-raw(memory:CUDAMemory), width={width}, height={height}, chroma-site={color_range}, format=NV12, colorimetry={color_space}, pixel-aspect-ratio=1/1''' video_params_zero_copy = '''cudaupload ! cudaconvertscale add-borders=true ! video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1 ''' ```

bsakaguchi

2026-10-15 12:59:00 +00:00

closed this issue
added the
fixed?

nvidia
labels

bsakaguchi commented

2026-10-15 12:59:00 +00:00

Author

Owner

@ABeltramo commented on GitHub (Apr 24, 2026):

Can you please post the full Wolf logs when the stream breaks with the added env variable WOLF_LOG_LEVEL=DEBUG?

@ABeltramo commented on GitHub (Apr 24, 2026): Can you please post the full Wolf logs when the stream breaks with the added env variable `WOLF_LOG_LEVEL=DEBUG`?

bsakaguchi commented

2026-10-15 12:59:00 +00:00

Author

Owner

@tjacoby2006 commented on GitHub (Apr 24, 2026):

Ok looks like reproducing is a little more nuanced as to when it fails and when it doesn't. It seems to work a bit more often when the debug log level is enabled (race condition somewhere?). Basically, I connect, start the test ball, then go to Wolf and it usually fails. Usually going even directly to Wolf fails as well, but it does succeed on occasion.

logs.txt

@tjacoby2006 commented on GitHub (Apr 24, 2026): Ok looks like reproducing is a little more nuanced as to when it fails and when it doesn't. It seems to work a bit more often when the debug log level is enabled (race condition somewhere?). Basically, I connect, start the test ball, then go to Wolf and it _usually_ fails. Usually going even directly to Wolf fails as well, but it does succeed on occasion. [logs.txt](https://github.com/user-attachments/files/27054491/logs.txt)

bsakaguchi commented

2026-10-15 12:59:00 +00:00

Author

Owner

@ABeltramo commented on GitHub (Apr 24, 2026):

These logs don't capture any issue, sure there are warnings, but the stream was working for ~10 seconds until the client disconnected. Are you sure it's not a network issue between the client and the server? Moonlight clients have an automatic 10 second timeout if they don't receive any video packet..

@ABeltramo commented on GitHub (Apr 24, 2026): These logs don't capture any issue, sure there are warnings, but the stream was working for ~10 seconds until the client disconnected. Are you sure it's not a network issue between the client and the server? Moonlight clients have an automatic 10 second timeout if they don't receive any video packet..

bsakaguchi commented

2026-10-15 12:59:00 +00:00

Author

Owner

@tjacoby2006 commented on GitHub (Apr 24, 2026):

Line 216?

wolf-1 | 13:50:02.927375123 ERROR | [GSTREAMER] Pipeline error: Internal data stream error.

@tjacoby2006 commented on GitHub (Apr 24, 2026): Line 216? `wolf-1 | 13:50:02.927375123 ERROR | [GSTREAMER] Pipeline error: Internal data stream error.`

bsakaguchi commented

2026-10-15 12:59:01 +00:00

Author

Owner

@ABeltramo commented on GitHub (Apr 24, 2026):

Right, missed that. I guess your card doesn't support some formats; you could try playing with the config.toml line

[gstreamer.video.defaults.nvcodec]
    video_params_zero_copy = '''cudaupload !
cudaconvertscale add-borders=true !
video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1
'''

A couple of simple tests that you can do:

completely remove the cudaconvertscale ... line and basically make it `video_params_zero_copy="cudaupload"
if that fails, put it back and just change the format from NV12 to BGRA

Remember to restart the Wolf container after you edit config.toml

@ABeltramo commented on GitHub (Apr 24, 2026): Right, missed that. I guess your card doesn't support some formats; you could try playing with the config.toml line ```toml [gstreamer.video.defaults.nvcodec] video_params_zero_copy = '''cudaupload ! cudaconvertscale add-borders=true ! video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1 ''' ``` A couple of simple tests that you can do: - completely remove the `cudaconvertscale ...` line and basically make it `video_params_zero_copy="cudaupload" - if that fails, put it back and just change the format from `NV12` to `BGRA` Remember to restart the Wolf container after you edit `config.toml`

bsakaguchi commented

2026-10-15 12:59:01 +00:00

Author

Owner

@tjacoby2006 commented on GitHub (Apr 24, 2026):

I was not able to figure out a combination to get it working, and unfortunately I just don't understand the GStreamer pipelines very well, so I ended up just settling on doing a CPU conversion to get the format correct. Not ideal, but any added latency should not be the end of the world.

    [[gstreamer.video.h264_encoders]]
    check_elements = [ 'nvh264enc', 'cudaupload' ] # Removed cudaconvertscale from checks
    encoder_pipeline = '''
queue !
videoconvert !
cudaupload !
queue !
video/x-raw(memory:CUDAMemory), format=NV12, width={width}, height={height}, \
chroma-site={color_range}, colorimetry={color_space}, pixel-aspect-ratio=1/1 !
nvh264enc preset=low-latency-hq zerolatency=true gop-size=0 rc-mode=cbr-ld-hq bitrate={bitrate} vbv-buffer-size={vbv_buffer_size} aud=false !
h264parse !
video/x-h264, profile=main, stream-format=byte-stream'''
    plugin_name = 'nvcodec'

@tjacoby2006 commented on GitHub (Apr 24, 2026): I was not able to figure out a combination to get it working, and unfortunately I just don't understand the GStreamer pipelines very well, so I ended up just settling on doing a CPU conversion to get the format correct. Not ideal, but any added latency should not be the end of the world. ``` [[gstreamer.video.h264_encoders]] check_elements = [ 'nvh264enc', 'cudaupload' ] # Removed cudaconvertscale from checks encoder_pipeline = ''' queue ! videoconvert ! cudaupload ! queue ! video/x-raw(memory:CUDAMemory), format=NV12, width={width}, height={height}, \ chroma-site={color_range}, colorimetry={color_space}, pixel-aspect-ratio=1/1 ! nvh264enc preset=low-latency-hq zerolatency=true gop-size=0 rc-mode=cbr-ld-hq bitrate={bitrate} vbv-buffer-size={vbv_buffer_size} aud=false ! h264parse ! video/x-h264, profile=main, stream-format=byte-stream''' plugin_name = 'nvcodec' ```

bsakaguchi commented

2026-10-15 12:59:01 +00:00

Author

Owner

@ABeltramo commented on GitHub (Jun 14, 2026):

This might be fixed by https://github.com/games-on-whales/wolf/pull/425 would you be able to pull the latest

docker pull ghcr.io/games-on-whales/wolf:stable

revert the changes in config.toml and test it out again?

@ABeltramo commented on GitHub (Jun 14, 2026): This might be fixed by https://github.com/games-on-whales/wolf/pull/425 would you be able to pull the latest ``` docker pull ghcr.io/games-on-whales/wolf:stable ``` revert the changes in `config.toml` and test it out again?

bsakaguchi commented

2026-10-15 12:59:01 +00:00

Author

Owner

@piotrminkina commented on GitHub (Sep 13, 2026):

Same failure signature here, on Blackwell rather than a P100. After a lot of testing I can reduce it to something more general than a codec problem: the shared CUDA context degrades with every session teardown. An H.264 pipeline fails after one teardown, an H.265 pipeline survives one and fails after two. H.264 is not special, only more sensitive — which is why it looks like an H.264 bug at first.

Minimal reproduction

One client, one codec, nothing else involved.

With H.264:

Restart Wolf.
Connect. It streams.
Close the session, connect again → Pipeline error: Internal data stream error.

With H.265 — the same thing, one step later:

Restart Wolf.
Connect, close. Connect, close. Both stream fine.
Connect a third time → fails.

Every further attempt also fails, and Wolf often dies outright after a few of them. Only restarting Wolf clears it.

Environment


Wolf	`ghcr.io/games-on-whales/wolf:stable`, digest `sha256:edb348d7…`, built from wolf `2273b2e` (2026-09-07)
GPU	NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
Driver	595.91.07
GStreamer	1.26.7-0
Encoders	`Using h264 encoder: nvcodec`, `h265: nvcodec`, `av1: nvcodec`; zero-copy pipeline on `/dev/dri/renderD128`
Clients	NVIDIA Shield (Android Moonlight) and a Windows 10 laptop (Moonlight 6.1.0)

Blackwell + driver 595 is the hardware family of #417 and #405, both closed.

The failure has two stages

One Wolf process, one client, H.264 only — every pipeline's codec read from caps on its own capsfilter element, so there is no attribution ambiguity and no H.265 anywhere in the log:

attempt	element	signature	result
1	`capsfilter3`	none	streams fine
2	`capsfilter7`	`could not transform` x4	fails
3	`capsfilter9`	`could not transform` x4	fails
4	`capsfilter13`	context assertions x5	fails
5	`capsfilter17`	context assertions x5	fails
6	`capsfilter21`	context assertions x5	fails

Stage 1 — cudaconvertscale cannot configure a conversion, while the CUDA context is still a valid object:

WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale4>
     transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA,
     width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support
WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_..._video>
     error: streaming stopped, reason not-negotiated (-4)
ERROR | [GSTREAMER] Pipeline error: Internal data stream error.

Confirmed identical at 1920x1080 and 3840x2160.

Stage 2 — once two sessions have been torn down, the shared GstCudaContext stops being a valid GObject at all. This does not require any failed attempt first: in the H.265-only run below it arrives immediately after two sessions that both streamed perfectly.

GLib-GObject-CRITICAL **: g_object_get: assertion 'G_IS_OBJECT (object)' failed
GLib-GObject-CRITICAL **: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
GStreamer-Cuda-CRITICAL **: GstCudaStream* gst_cuda_stream_new(GstCudaContext*): assertion failed
GStreamer-Cuda-CRITICAL **: GstBufferPool* gst_cuda_buffer_pool_new(GstCudaContext*): assertion failed
GStreamer-Cuda-CRITICAL **: gboolean gst_cuda_context_push(GstCudaContext*): assertion 'ctx' failed

These fire before the new session's Wayland display ready line — the context is already dead when setup begins — and the could not transform warnings disappear, because the pipeline no longer gets far enough to negotiate.

H.265 is never observed in stage 1. It tolerates the one-teardown damage level entirely, so by the time it fails the context is already dead and it goes straight to stage 2 — which is exactly what the tolerance table below predicts.

The client-side symptom tracks the stage: in stage 1 Moonlight reports a terminated stream but still offers to resume; in stage 2 it shows the generic "something went wrong on the host PC while starting the stream" dialog.

In one run this ended with Wolf dying (exited: wolf (exit status 11; not expected)), with a stack that lands in Wolf's own context-sharing path:

#10 WaylandDisplaySrc::set_context                      libgstwaylanddisplaysrc.so
#12 gst_video_context::set_context                      /wolf/src/gst-video-context/gst-video-context.cpp:150
#13 need_context_handler                                /wolf/src/moonlight-server/streaming/streaming.cpp:61
#14 bus_sync_handler                                    /wolf/src/moonlight-server/streaming/streaming.cpp:69
...
#31 gst_pad_set_active
GStreamer-CRITICAL **: gst_object_unref: assertion 'object != NULL' failed

I read the two stages as one progressive corruption driven by teardowns: closing the first session leaves the shared context damaged but still alive, so a stage-1 pipeline passes the GObject assertions yet finds no usable CUDA resources; closing a second one finishes the object off. Teardowns of sessions that actually streamed appear to cost more than teardowns of ones that already failed — two of the former are enough, whereas the all-H.264 run needed one successful plus two failed. That is an interpretation, not a measurement.

What actually predicts the failure: teardown count

Counting teardowns that precede the first failure, across archived runs:

run	codec of the failing pipeline	teardowns before it
Shield, H.265 only	H.265	2
Shield, H.264 only	H.264	1
mixed scenario	H.265	2
laptop + Shield	H.264	1

The H.265-only run reproduced twice inside one log, in two consecutive Wolf processes, with identical caps counts per pipeline (130 / 103 / 48 / 21) and no H.264 anywhere in the file: two sessions stream, the third fails with the stage-2 context assertions, then Wolf exits 11.

This also resolves a case that looked contradictory earlier: a run where a third pipeline on H.265 worked fine. There the first client's session was never closed, so only one teardown had happened — within H.265's tolerance.

So the pipeline's position in the process is a proxy. What the failure tracks is how many sessions have been torn down since Wolf started:

codec	teardowns tolerated
H.264	0
H.265	1

"First pipeline" means first in the process lifetime, not first socket

Wolf reuses Wayland socket names, but a reconnecting client always gets new elements:

21:29:29  wayland-1  ->  cudaconvertscale3
21:30:02  wayland-2  ->  cudaconvertscale4
21:33:32  wayland-2  ->  cudaconvertscale5   <- same socket name, new instance
--------- Wolf restart ---------
21:45:14  wayland-1  ->  cudaconvertscale3   <- counter resets

The element counter is per process and resets only on restart, which is the only action that clears the bug. Reconnecting onto the socket name that just worked does not help. In every capture the first pipeline of a process is cudaconvertscale3 and succeeds. Position itself is not the cause — the teardown count is — but the element index is a convenient greppable marker for which pipeline in a process you are looking at.

Ruled out by experiment

Each tested with that variable isolated:

Client identity — reproduced independently on both clients, and on each of them with either codec. Both clients show the same tolerance thresholds, so nothing here is specific to a device or to a Moonlight build.
Resolution — same root-cause signature at 1920x1080 and 3840x2160.
A second client, or any codec mixing — both halves of the minimal reproduction above use a single client and a single codec throughout; the H.265 run contains no H.264 in the log at all, and vice versa.
Socket name reuse — failures occur on freshly numbered wayland-N and on reused ones alike.
Time between sessions — 6 s and 72 s behave identically.
Chroma / 4:4:4 — the pipeline hardcodes format=NV12; both codecs negotiate profile=main and chroma-format=4:2:0.
Deprecated NVENC properties — the shipped h264 nvcodec pipeline uses preset=low-latency-hq + rc-mode=cbr-ld-hq, both flagged deprecated by gst-inspect, while h265 and av1 already use preset=p1 + tune + multi-pass. Replacing the h264 line with the modern equivalent did not fix it (the new parameters encode fine standalone). Unrelated to this bug, the inconsistency may still be worth tidying.
A second nvh264enc in the process as such — two of them in one process via tee work fine.

CUDAMemory BGRA also reaches pipelines that work

Which element emits the transform warning matters. In a Lutris session — the third pipeline created in that process — cudaconvertscale never complains, but a capsfilter does, exactly once:

WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<capsfilter8>
     transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA,
     width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support

That pipeline streams with no error at all — and note it was the third pipeline in its process, with zero teardowns before it, because the two earlier sessions were still running. Position alone costs nothing. (I did not measure its own codec — the caps lines in its time window belong to the still-running first session — but the point here does not depend on it.) So a rejected candidate during negotiation is routine, and CUDAMemory BGRA reaches pipelines that succeed as well as ones that fail. Only the <cudaconvertscale*> variant is fatal.

This is not a stale image

My image is the current stable (registry digest matches) and contains every relevant fix merged so far: #425, and gst-wayland-display 016b4fc pinned by 1173e1a5, which includes the July/August CUDA lifetime fixes — ce6535d5 (pool config pinning GstCudaContext), 01027bad (clearing a pool the query still owns), PR #42 (CUDA context refcount), ce354c19 (reference the gst_cuda helpers' short-circuit never transferred), PR #36 (DMABuf→CUDA context teardown). The stage-2 assertions above happen with all of those applied.

The deeper Blackwell fix mentioned in #417 ("I'm working on a fix for blackwell and non-blackwell architectures") does not appear to have landed — I found no merged or open PR for it.

Separately, gst-cuda still logs Failed to load 'nvrtcGetCUBINSize' … libnvrtc.so: undefined symbol here. games-on-whales/gow#331 addresses that, but by its own test plan it is a no-op for driver-volume setups like mine, and its author notes the missing symbols only disable CUBIN while PTX still works — mentioned for completeness rather than as a suspect.

The open question

Both codecs share the same video_params_zero_copy, so cudaconvertscale is asked for the same output caps either way:

[gstreamer.video.defaults.nvcodec]
video_params_zero_copy = '''cudaupload !
cudaconvertscale add-borders=true !
video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1'''

Same requested output either way — yet H.264 gives up after one teardown and H.265 after two. Since both eventually fail, this is a difference in sensitivity to a degraded context rather than a categorical one: something the H.264 pipeline needs from the context is unavailable a step sooner. My guess is that the two encoders advertise different sink template caps, so negotiation resolves to different requirements upstream, but I have not verified it.

The more useful question for whoever picks this up is probably the other one: what does tearing a session down do to the shared GstCudaContext such that the next pipeline gets a damaged one, and the one after that gets an object that fails G_IS_OBJECT?

Workaround

There is no real workaround, only a way to postpone it. Switching a client to HEVC buys exactly one more session per Wolf lifetime, at the cost of software decoding on clients without hardware HEVC decode; the third session fails anyway. The only reliable answer is restarting Wolf, which is not practical in normal use.

Logs for every run above are small (60–900 lines) and I can attach any of them. Given the Blackwell hardware, happy to move this to its own issue and link back if you prefer.

@piotrminkina commented on GitHub (Sep 13, 2026): Same failure signature here, on Blackwell rather than a P100. After a lot of testing I can reduce it to something more general than a codec problem: **the shared CUDA context degrades with every session teardown.** An H.264 pipeline fails after one teardown, an H.265 pipeline survives one and fails after two. H.264 is not special, only more sensitive — which is why it looks like an H.264 bug at first. ### Minimal reproduction One client, one codec, nothing else involved. **With H.264:** 1. Restart Wolf. 2. Connect. It streams. 3. Close the session, connect again → `Pipeline error: Internal data stream error.` **With H.265** — the same thing, one step later: 1. Restart Wolf. 2. Connect, close. Connect, close. Both stream fine. 3. Connect a third time → fails. Every further attempt also fails, and Wolf often dies outright after a few of them. Only restarting Wolf clears it. ### Environment | | | |---|---| | Wolf | `ghcr.io/games-on-whales/wolf:stable`, digest `sha256:edb348d7…`, built from wolf `2273b2e` (2026-09-07) | | GPU | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | | Driver | 595.91.07 | | GStreamer | 1.26.7-0 | | Encoders | `Using h264 encoder: nvcodec`, `h265: nvcodec`, `av1: nvcodec`; zero-copy pipeline on `/dev/dri/renderD128` | | Clients | NVIDIA Shield (Android Moonlight) and a Windows 10 laptop (Moonlight 6.1.0) | Blackwell + driver 595 is the hardware family of #417 and #405, both closed. ### The failure has two stages One Wolf process, one client, H.264 only — every pipeline's codec read from caps on its own `capsfilter` element, so there is no attribution ambiguity and no H.265 anywhere in the log: | attempt | element | signature | result | |---|---|---|---| | 1 | `capsfilter3` | none | **streams fine** | | 2 | `capsfilter7` | `could not transform` x4 | fails | | 3 | `capsfilter9` | `could not transform` x4 | fails | | 4 | `capsfilter13` | context assertions x5 | fails | | 5 | `capsfilter17` | context assertions x5 | fails | | 6 | `capsfilter21` | context assertions x5 | fails | **Stage 1** — `cudaconvertscale` cannot configure a conversion, while the CUDA context is still a valid object: ``` WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<cudaconvertscale4> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support WARN basesrc gstbasesrc.c:3187:gst_base_src_loop:<interpipesrc_..._video> error: streaming stopped, reason not-negotiated (-4) ERROR | [GSTREAMER] Pipeline error: Internal data stream error. ``` Confirmed identical at 1920x1080 and 3840x2160. **Stage 2** — once two sessions have been torn down, the shared `GstCudaContext` stops being a valid GObject at all. This does not require any failed attempt first: in the H.265-only run below it arrives immediately after two sessions that both streamed perfectly. ``` GLib-GObject-CRITICAL **: g_object_get: assertion 'G_IS_OBJECT (object)' failed GLib-GObject-CRITICAL **: g_object_ref: assertion 'G_IS_OBJECT (object)' failed GStreamer-Cuda-CRITICAL **: GstCudaStream* gst_cuda_stream_new(GstCudaContext*): assertion failed GStreamer-Cuda-CRITICAL **: GstBufferPool* gst_cuda_buffer_pool_new(GstCudaContext*): assertion failed GStreamer-Cuda-CRITICAL **: gboolean gst_cuda_context_push(GstCudaContext*): assertion 'ctx' failed ``` These fire **before** the new session's `Wayland display ready` line — the context is already dead when setup begins — and the `could not transform` warnings disappear, because the pipeline no longer gets far enough to negotiate. H.265 is never observed in stage 1. It tolerates the one-teardown damage level entirely, so by the time it fails the context is already dead and it goes straight to stage 2 — which is exactly what the tolerance table below predicts. The client-side symptom tracks the stage: in stage 1 Moonlight reports a terminated stream but still offers to resume; in stage 2 it shows the generic "something went wrong on the host PC while starting the stream" dialog. In one run this ended with Wolf dying (`exited: wolf (exit status 11; not expected)`), with a stack that lands in Wolf's own context-sharing path: ``` #10 WaylandDisplaySrc::set_context libgstwaylanddisplaysrc.so #12 gst_video_context::set_context /wolf/src/gst-video-context/gst-video-context.cpp:150 #13 need_context_handler /wolf/src/moonlight-server/streaming/streaming.cpp:61 #14 bus_sync_handler /wolf/src/moonlight-server/streaming/streaming.cpp:69 ... #31 gst_pad_set_active GStreamer-CRITICAL **: gst_object_unref: assertion 'object != NULL' failed ``` I read the two stages as one progressive corruption driven by teardowns: closing the first session leaves the shared context damaged but still alive, so a stage-1 pipeline passes the GObject assertions yet finds no usable CUDA resources; closing a second one finishes the object off. Teardowns of sessions that actually streamed appear to cost more than teardowns of ones that already failed — two of the former are enough, whereas the all-H.264 run needed one successful plus two failed. That is an interpretation, not a measurement. ### What actually predicts the failure: teardown count Counting teardowns that precede the first failure, across archived runs: | run | codec of the failing pipeline | teardowns before it | |---|---|---| | Shield, H.265 only | H.265 | **2** | | Shield, H.264 only | H.264 | **1** | | mixed scenario | H.265 | **2** | | laptop + Shield | H.264 | **1** | The H.265-only run reproduced twice inside one log, in two consecutive Wolf processes, with identical caps counts per pipeline (130 / 103 / 48 / 21) and **no H.264 anywhere in the file**: two sessions stream, the third fails with the stage-2 context assertions, then Wolf exits 11. This also resolves a case that looked contradictory earlier: a run where a third pipeline on H.265 worked fine. There the first client's session was never closed, so only one teardown had happened — within H.265's tolerance. So the pipeline's position in the process is a proxy. What the failure tracks is how many sessions have been torn down since Wolf started: | codec | teardowns tolerated | |---|---| | H.264 | 0 | | H.265 | 1 | ### "First pipeline" means first in the process lifetime, not first socket Wolf reuses Wayland socket names, but a reconnecting client always gets new elements: ``` 21:29:29 wayland-1 -> cudaconvertscale3 21:30:02 wayland-2 -> cudaconvertscale4 21:33:32 wayland-2 -> cudaconvertscale5 <- same socket name, new instance --------- Wolf restart --------- 21:45:14 wayland-1 -> cudaconvertscale3 <- counter resets ``` The element counter is per process and resets only on restart, which is the only action that clears the bug. Reconnecting onto the socket name that just worked does not help. In every capture the first pipeline of a process is `cudaconvertscale3` and succeeds. Position itself is not the cause — the teardown count is — but the element index is a convenient greppable marker for which pipeline in a process you are looking at. ### Ruled out by experiment Each tested with that variable isolated: - **Client identity** — reproduced independently on both clients, and on each of them with either codec. Both clients show the same tolerance thresholds, so nothing here is specific to a device or to a Moonlight build. - **Resolution** — same root-cause signature at 1920x1080 and 3840x2160. - **A second client, or any codec mixing** — both halves of the minimal reproduction above use a single client and a single codec throughout; the H.265 run contains no H.264 in the log at all, and vice versa. - **Socket name reuse** — failures occur on freshly numbered `wayland-N` and on reused ones alike. - **Time between sessions** — 6 s and 72 s behave identically. - **Chroma / 4:4:4** — the pipeline hardcodes `format=NV12`; both codecs negotiate `profile=main` and `chroma-format=4:2:0`. - **Deprecated NVENC properties** — the shipped h264 nvcodec pipeline uses `preset=low-latency-hq` + `rc-mode=cbr-ld-hq`, both flagged deprecated by `gst-inspect`, while h265 and av1 already use `preset=p1` + `tune` + `multi-pass`. Replacing the h264 line with the modern equivalent did **not** fix it (the new parameters encode fine standalone). Unrelated to this bug, the inconsistency may still be worth tidying. - **A second `nvh264enc` in the process as such** — two of them in one process via `tee` work fine. ### CUDAMemory BGRA also reaches pipelines that work Which element emits the transform warning matters. In a Lutris session — the third pipeline created in that process — `cudaconvertscale` never complains, but a `capsfilter` does, exactly once: ``` WARN basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<capsfilter8> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)BGRA, width=(int)1920, height=(int)1080, framerate=(fraction)60/1 in anything we support ``` That pipeline streams with no error at all — and note it was the third pipeline in its process, with **zero** teardowns before it, because the two earlier sessions were still running. Position alone costs nothing. (I did not measure its own codec — the caps lines in its time window belong to the still-running first session — but the point here does not depend on it.) So a rejected candidate during negotiation is routine, and CUDAMemory BGRA reaches pipelines that succeed as well as ones that fail. Only the `<cudaconvertscale*>` variant is fatal. ### This is not a stale image My image is the current `stable` (registry digest matches) and contains every relevant fix merged so far: #425, and gst-wayland-display `016b4fc` pinned by `1173e1a5`, which includes the July/August CUDA lifetime fixes — `ce6535d5` (pool config pinning `GstCudaContext`), `01027bad` (clearing a pool the query still owns), PR #42 (CUDA context refcount), `ce354c19` (reference the gst_cuda helpers' short-circuit never transferred), PR #36 (DMABuf→CUDA context teardown). The stage-2 assertions above happen *with* all of those applied. The deeper Blackwell fix mentioned in #417 ("I'm working on a fix for blackwell and non-blackwell architectures") does not appear to have landed — I found no merged or open PR for it. Separately, `gst-cuda` still logs `Failed to load 'nvrtcGetCUBINSize' … libnvrtc.so: undefined symbol` here. games-on-whales/gow#331 addresses that, but by its own test plan it is a no-op for driver-volume setups like mine, and its author notes the missing symbols only disable CUBIN while PTX still works — mentioned for completeness rather than as a suspect. ### The open question Both codecs share the same `video_params_zero_copy`, so `cudaconvertscale` is asked for the same output caps either way: ```toml [gstreamer.video.defaults.nvcodec] video_params_zero_copy = '''cudaupload ! cudaconvertscale add-borders=true ! video/x-raw(memory:CUDAMemory),format=NV12, width={width}, height={height}, pixel-aspect-ratio=1/1''' ``` Same requested output either way — yet H.264 gives up after one teardown and H.265 after two. Since both eventually fail, this is a difference in sensitivity to a degraded context rather than a categorical one: something the H.264 pipeline needs from the context is unavailable a step sooner. My guess is that the two encoders advertise different sink template caps, so negotiation resolves to different requirements upstream, but I have not verified it. The more useful question for whoever picks this up is probably the other one: what does tearing a session down do to the shared `GstCudaContext` such that the next pipeline gets a damaged one, and the one after that gets an object that fails `G_IS_OBJECT`? ### Workaround There is no real workaround, only a way to postpone it. Switching a client to HEVC buys exactly one more session per Wolf lifetime, at the cost of software decoding on clients without hardware HEVC decode; the third session fails anyway. The only reliable answer is restarting Wolf, which is not practical in normal use. Logs for every run above are small (60–900 lines) and I can attach any of them. Given the Blackwell hardware, happy to move this to its own issue and link back if you prefer.

bsakaguchi commented

2026-10-15 12:59:01 +00:00

Author

Owner

@tjacoby2006 commented on GitHub (Sep 15, 2026):

This might be fixed by #425 would you be able to pull the latest
docker pull ghcr.io/games-on-whales/wolf:stable
revert the changes in config.toml and test it out again?

The latest is working for me on a fresh setup. Thanks!

@tjacoby2006 commented on GitHub (Sep 15, 2026): > This might be fixed by [#425](https://github.com/games-on-whales/wolf/pull/425) would you be able to pull the latest > > ``` > docker pull ghcr.io/games-on-whales/wolf:stable > ``` > > revert the changes in `config.toml` and test it out again? The latest is working for me on a fresh setup. Thanks!