[GH-ISSUE #25] [Nvidia] Messed up image with custom resolutions #9

Closed
opened 2026-10-15 12:47:21 +00:00 by bsakaguchi · 7 comments
Owner

Originally created by @ABeltramo on GitHub (Nov 21, 2025).
Original GitHub issue: https://github.com/games-on-whales/gst-wayland-display/issues/25

This seems to happen only on Nvidia, some custom resolutions like 1366x768 results in the following:

Image

but if I run the same on an AMD GPU it seems to work

Image

At first I thought it was something to do with the Nvidia encoders or Moonlight but I can reproduce the issue even locally with the following simple pipeline:

gst-launch-1.0 waylanddisplaysrc ! 'video/x-raw(memory:CUDAMemory),width=1366,height=768,framerate=60/1' ! cudaconvert ! cudadownload ! autovideosink

if instead I use the non-zero copy pipeline it seems to display just fine:

gst-launch-1.0 waylanddisplaysrc ! 'video/x-raw,width=1366,height=768,framerate=60/1' ! videoconvert ! autovideosink
Originally created by @ABeltramo on GitHub (Nov 21, 2025). Original GitHub issue: https://github.com/games-on-whales/gst-wayland-display/issues/25 This seems to happen only on Nvidia, some custom resolutions like `1366x768` results in the following: <img width="1708" height="960" alt="Image" src="https://github.com/user-attachments/assets/92370700-9d83-439c-905f-8194d30bc08d" /> but if I run the same on an AMD GPU it seems to work <img width="1708" height="960" alt="Image" src="https://github.com/user-attachments/assets/d0003dd7-0ca6-4941-85b6-41e7212f7831" /> At first I thought it was something to do with the Nvidia encoders or Moonlight but I can reproduce the issue even locally with the following simple pipeline: ``` gst-launch-1.0 waylanddisplaysrc ! 'video/x-raw(memory:CUDAMemory),width=1366,height=768,framerate=60/1' ! cudaconvert ! cudadownload ! autovideosink ``` if instead I use the non-zero copy pipeline it seems to display just fine: ``` gst-launch-1.0 waylanddisplaysrc ! 'video/x-raw,width=1366,height=768,framerate=60/1' ! videoconvert ! autovideosink ```
Author
Owner

@Greaper88 commented on GitHub (Nov 21, 2025):

Is this related to https://github.com/games-on-whales/wolf/issues/41 ?

<!-- gh-comment-id:3562115860 --> @Greaper88 commented on GitHub (Nov 21, 2025): Is this related to https://github.com/games-on-whales/wolf/issues/41 ?
Author
Owner

@ABeltramo commented on GitHub (Nov 21, 2025):

Unlikely, since this isn't related to the encoders or Gstreamer. Plus I can't reproduce it on AMD.
It's most likely something to do with our new EGL->Cuda code to get a high performance pipeline on Nvidia.

Thanks for looking that up though!

<!-- gh-comment-id:3562140588 --> @ABeltramo commented on GitHub (Nov 21, 2025): Unlikely, since this isn't related to the encoders or Gstreamer. Plus I can't reproduce it on AMD. It's most likely something to do with our new `EGL->Cuda` code to get a high performance pipeline on Nvidia. Thanks for looking that up though!
Author
Owner
<!-- gh-comment-id:3567792603 --> @ABeltramo commented on GitHub (Nov 23, 2025): Note to self: I wonder if it's this: https://github.com/games-on-whales/gst-wayland-display/blob/stable/wayland-display-core%2Fsrc%2Futils%2Fallocator%2Fcuda%2Fmod.rs#L501-L508
Author
Owner

@Pizmovc commented on GitHub (Nov 23, 2025):

Adding another data point (from Discord):

having a custom resolution (1512x950), messes up the display for wolf-ui. Test ball is working fine.
Here is the messed up display:
Image
1080p and 1440p (from the dropdown) work fine.

<!-- gh-comment-id:3568310492 --> @Pizmovc commented on GitHub (Nov 23, 2025): Adding another data point (from [Discord](https://discord.com/channels/856434175455133727/856436453860704266/1440049223813824624)): > having a custom resolution (1512x950), messes up the display for wolf-ui. Test ball is working fine. > Here is the messed up display: > <img width="3024" height="1964" alt="Image" src="https://github.com/user-attachments/assets/ad344f4a-8f4c-416c-b6a4-d0bdd1c4c1f4" /> > 1080p and 1440p (from the dropdown) work fine.
Author
Owner

@DatCaptainHorse commented on GitHub (Nov 24, 2025):

Some encoders don't work well with resolutions that aren't to power of 2 or so, for example OBS-Studio replaces invalid resolutions that user may input with closest valid power of 2 one. (Could be something else than power of 2, but I can't remember exactly).

<!-- gh-comment-id:3570702057 --> @DatCaptainHorse commented on GitHub (Nov 24, 2025): Some encoders don't work well with resolutions that aren't to power of 2 or so, for example OBS-Studio replaces invalid resolutions that user may input with closest valid power of 2 one. (Could be something else than power of 2, but I can't remember exactly).
Author
Owner

@ABeltramo commented on GitHub (Nov 24, 2025):

Some encoders don't work well with resolutions that aren't to power of 2

Unfortunately I can replicate this even when we aren't using any encoder, so it's not that. By the look of it it's most likely some wrong stride/offset param passed between our DMA buffer -> EGL -> CUDA. I'm trying to pin it down properly, but these things are always painful to debug.. I'll get there, just need some time!

<!-- gh-comment-id:3571539527 --> @ABeltramo commented on GitHub (Nov 24, 2025): > Some encoders don't work well with resolutions that aren't to power of 2 Unfortunately I can replicate this even when we aren't using any encoder, so it's not that. By the look of it it's most likely some wrong stride/offset param passed between our DMA buffer -> EGL -> CUDA. I'm trying to pin it down properly, but these things are always painful to debug.. I'll get there, just need some time!
Author
Owner

@ABeltramo commented on GitHub (Nov 25, 2025):

Just pushed a fix for this, wolf:stable is currently building, give it a few minutes. Feel free to re-open this if the issue is still present.

<!-- gh-comment-id:3575520192 --> @ABeltramo commented on GitHub (Nov 25, 2025): Just pushed a fix for this, `wolf:stable` is currently building, give it a few minutes. Feel free to re-open this if the issue is still present.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
games-on-whales/gst-wayland-display#9
No description provided.