[PR #40] [CLOSED] feat(nv12): negotiate the encoder's NV12 modifier #41

New issue

Closed

opened 2026-10-15 12:47:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:50 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/40
Author: @JBailes
Created: 6/23/2026
Status: ❌ Closed

Base: feat/vulkan-encode ← Head: feat/nv12-modifier-negotiate

📝 Commits (1)

b8a6340 feat(nv12): negotiate the encoder's NV12 modifier

📊 Changes

4 files changed (+215 additions, -48 deletions)

View changed files

📝 gst-plugin-wayland-display/src/waylandsrc/imp.rs (+114 -24)
📝 wayland-display-core/src/utils/mod.rs (+1 -0)
➕ wayland-display-core/src/utils/va_query.rs (+97 -0)
📝 wayland-display-core/src/utils/vulkan_nv12.rs (+3 -24)

📄 Description

Stacks on #37. Answers the modifier question raised there.

#37 advertised every NV12 modifier and let negotiation pick. Behind interpipe nothing negotiates, so it picked a tiled modifier the consumer reads as green. Same modifier guess as wolf:stable.

This pins it instead of guessing:

nv12=true advertises LINEAR only. No encoder in the pipeline to negotiate with, so pin the one modifier every VA importer takes. Kills the green.
Direct ! vah265enc orders the encoder's own modifier first, queried from the driver via gst_va_dmabuf_get_modifier_for_format. Negotiation lands where the encoder wants, not on merge order.
Fail loud on a modifier we can't export, instead of garbled frames.
Drops the env LINEAR hack.

New va_query.rs loads the same libgstva-1.0 as va_share and calls two public symbols. Missing lib is a runtime warning, not a link error.

Tested AMD 7900 XTX (renderD128):

driver reports NV12 import modifier LINEAR
direct waylanddisplaysrc ! vah265enc negotiates LINEAR, reaches EOS
nv12=true ! vapostproc ! VAMemory NV12 ! vah265enc ! vah265dec decodes SMPTE clean, no green

Note: the 9070 case (encoder accepts only DCC) stays blocked by radv/radeonsi DCC interop. The 7900 XTX imports LINEAR directly, so direct encode works there. For Wolf, keep vapostproc on the consumer and set nv12=true on the producer (draft wolf PR to follow).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/40 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 6/23/2026 **Status:** ❌ Closed **Base:** `feat/vulkan-encode` ← **Head:** `feat/nv12-modifier-negotiate` --- ### 📝 Commits (1) - [`b8a6340`](https://github.com/games-on-whales/gst-wayland-display/commit/b8a6340976d5bd2415a25d45ade116be60cf954e) feat(nv12): negotiate the encoder's NV12 modifier ### 📊 Changes **4 files changed** (+215 additions, -48 deletions) <details> <summary>View changed files</summary> 📝 `gst-plugin-wayland-display/src/waylandsrc/imp.rs` (+114 -24) 📝 `wayland-display-core/src/utils/mod.rs` (+1 -0) ➕ `wayland-display-core/src/utils/va_query.rs` (+97 -0) 📝 `wayland-display-core/src/utils/vulkan_nv12.rs` (+3 -24) </details> ### 📄 Description Stacks on #37. Answers the modifier question raised there. #37 advertised every NV12 modifier and let negotiation pick. Behind interpipe nothing negotiates, so it picked a tiled modifier the consumer reads as green. Same modifier guess as wolf:stable. This pins it instead of guessing: - `nv12=true` advertises LINEAR only. No encoder in the pipeline to negotiate with, so pin the one modifier every VA importer takes. Kills the green. - Direct `! vah265enc` orders the encoder's own modifier first, queried from the driver via `gst_va_dmabuf_get_modifier_for_format`. Negotiation lands where the encoder wants, not on merge order. - Fail loud on a modifier we can't export, instead of garbled frames. - Drops the env LINEAR hack. New `va_query.rs` loads the same libgstva-1.0 as `va_share` and calls two public symbols. Missing lib is a runtime warning, not a link error. Tested AMD 7900 XTX (renderD128): - driver reports NV12 import modifier LINEAR - direct `waylanddisplaysrc ! vah265enc` negotiates LINEAR, reaches EOS - `nv12=true ! vapostproc ! VAMemory NV12 ! vah265enc ! vah265dec` decodes SMPTE clean, no green Note: the 9070 case (encoder accepts only DCC) stays blocked by radv/radeonsi DCC interop. The 7900 XTX imports LINEAR directly, so direct encode works there. For Wolf, keep vapostproc on the consumer and set `nv12=true` on the producer (draft wolf PR to follow). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:47:50 +00:00

bsakaguchi referenced this issue

2026-10-15 12:47:50 +00:00

[PR #41] [MERGED] docs: document meson build recipe for the vkh264enc DPB patch #43

bsakaguchi closed this issue

2026-10-15 12:47:50 +00:00

bsakaguchi referenced this issue

2026-10-15 12:48:00 +00:00

[PR #57] [CLOSED] Chore/drop dead vulkan share #58