[PR #62] patches: Vulkan AV1 encoder + encode-family fixes #62

New issue

Open

opened 2026-10-15 12:48:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:48:03 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/62
Author: @salty2011
Created: 8/22/2026
Status: 🔄 Open

Base: feat/vulkan-encode ← Head: patches/vulkan-av1-encode

📝 Commits (5)

5597fc4 patches: vulkanav1enc + encode-family fixes
f1e67af docker: apply the full patch chain in the vulkan image
8bc3630 patches: address review on AV1 caps and live-retarget threading
6b4dd3b ci: build the patched GStreamer with tests and run the Vulkan tests
e9e1237 patches: narrow vulkanh265enc to the profiles it can encode

📊 Changes

11 files changed (+6673 additions, -60 deletions)

View changed files

➕ .github/workflows/gst-patches.yml (+58 -0)
📝 .gitignore (+1 -0)
📝 ci/README.md (+14 -0)
➕ ci/gst-patches.sh (+83 -0)
📝 docker/vulkan.Dockerfile (+9 -6)
📝 patches/README.md (+19 -3)
➕ patches/gstreamer-vulkan-rc-retarget-no-reset.patch (+137 -0)
➕ patches/series (+7 -0)
➕ patches/vkh264enc-rc-fix.patch (+286 -0)
➕ patches/vulkanav1enc.patch (+5714 -0)
📝 patches/vulkanh265enc.patch (+345 -51)

📄 Description

Patch files for this branch's patches/ set. I run all of these in my streaming stack on an RTX 5090 (driver 610.57.04, gst 1.28.4).

Patch	What it does
`vulkanav1enc.patch` (new)	Vulkan Video AV1 encoder: base class + element + registration. Also adds `av1_encode_caps()` to `gstvkvideocaps.c`. Stock 1.28.4 returns FALSE for AV1 encode there, so nothing registers without it. Includes a GPU-free test suite for the reference model.
`gstreamer-vulkan-rc-retarget-no-reset.patch` (new)	A live bitrate retarget raises only `ENCODE_RATE_CONTROL_BIT`, not `RESET_BIT`. Before this, every retarget reset the session and forced a keyframe, on all three codecs. Also fixes DPB slot 0 never being released.
`vkh264enc-rc-fix.patch` (new here)	vulkanh264enc rate control is broken as shipped in 1.28 (silently CQP). The retarget patch needs it.
`vulkanh265enc.patch` (updated)	Same file plus NVIDIA conformance fixes: adopt driver parameter sets, emit driver SPS/PPS bytes verbatim, colorimetry on output caps. Without these, NVIDIA h265 fails `ffmpeg -err_detect +explode` on 116/120 frames.

Apply order: dpb-pool (unchanged), vulkanh265enc, rc-fix, rc-retarget, vulkanav1enc.

AV1 evidence:

120/120 and 600/600 frames strict-decoded on vulkanav1dec
Live WebRTC into Chrome at 1080p60 and 1440p120
CBR tracks within a few percent. Upstream libs_vkvideoencodeav1 conformance stays green
Khronos validation layer clean

Scope: 8-bit 4:2:0 Main, intra + forward inter, single tile, CBR/VBR/CQP. No B-frames, no show_existing_frame, no 10-bit. All noted as TODOs in the source.

Caveat: NVIDIA only. I have no RADV encode box. Like the existing files these are vendored patches, regenerated per GStreamer bump. I keep them current.

Second commit: docker/vulkan.Dockerfile now applies the full chain, so this image build compiles the AV1 element too.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/62 **Author:** [@salty2011](https://github.com/salty2011) **Created:** 8/22/2026 **Status:** 🔄 Open **Base:** `feat/vulkan-encode` ← **Head:** `patches/vulkan-av1-encode` --- ### 📝 Commits (5) - [`5597fc4`](https://github.com/games-on-whales/gst-wayland-display/commit/5597fc44e647b6eb8656b168c060f2ad281ed1fa) patches: vulkanav1enc + encode-family fixes - [`f1e67af`](https://github.com/games-on-whales/gst-wayland-display/commit/f1e67afd7607bb7d78e52e16868deb7c18893d7f) docker: apply the full patch chain in the vulkan image - [`8bc3630`](https://github.com/games-on-whales/gst-wayland-display/commit/8bc36302d325b1bc9d1f4e2a6075f2bcb780f283) patches: address review on AV1 caps and live-retarget threading - [`6b4dd3b`](https://github.com/games-on-whales/gst-wayland-display/commit/6b4dd3b7cfbf76b181e5139fac26c66fb5380dad) ci: build the patched GStreamer with tests and run the Vulkan tests - [`e9e1237`](https://github.com/games-on-whales/gst-wayland-display/commit/e9e12377208f214af41183da8c7cf7ead5a204f8) patches: narrow vulkanh265enc to the profiles it can encode ### 📊 Changes **11 files changed** (+6673 additions, -60 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/gst-patches.yml` (+58 -0) 📝 `.gitignore` (+1 -0) 📝 `ci/README.md` (+14 -0) ➕ `ci/gst-patches.sh` (+83 -0) 📝 `docker/vulkan.Dockerfile` (+9 -6) 📝 `patches/README.md` (+19 -3) ➕ `patches/gstreamer-vulkan-rc-retarget-no-reset.patch` (+137 -0) ➕ `patches/series` (+7 -0) ➕ `patches/vkh264enc-rc-fix.patch` (+286 -0) ➕ `patches/vulkanav1enc.patch` (+5714 -0) 📝 `patches/vulkanh265enc.patch` (+345 -51) </details> ### 📄 Description Patch files for this branch's `patches/` set. I run all of these in my streaming stack on an RTX 5090 (driver 610.57.04, gst 1.28.4). | Patch | What it does | |---|---| | `vulkanav1enc.patch` (new) | Vulkan Video AV1 encoder: base class + element + registration. Also adds `av1_encode_caps()` to `gstvkvideocaps.c`. Stock 1.28.4 returns FALSE for AV1 encode there, so nothing registers without it. Includes a GPU-free test suite for the reference model. | | `gstreamer-vulkan-rc-retarget-no-reset.patch` (new) | A live bitrate retarget raises only `ENCODE_RATE_CONTROL_BIT`, not `RESET_BIT`. Before this, every retarget reset the session and forced a keyframe, on all three codecs. Also fixes DPB slot 0 never being released. | | `vkh264enc-rc-fix.patch` (new here) | vulkanh264enc rate control is broken as shipped in 1.28 (silently CQP). The retarget patch needs it. | | `vulkanh265enc.patch` (updated) | Same file plus NVIDIA conformance fixes: adopt driver parameter sets, emit driver SPS/PPS bytes verbatim, colorimetry on output caps. Without these, NVIDIA h265 fails `ffmpeg -err_detect +explode` on 116/120 frames. | Apply order: dpb-pool (unchanged), vulkanh265enc, rc-fix, rc-retarget, vulkanav1enc. AV1 evidence: - 120/120 and 600/600 frames strict-decoded on vulkanav1dec - Live WebRTC into Chrome at 1080p60 and 1440p120 - CBR tracks within a few percent. Upstream `libs_vkvideoencodeav1` conformance stays green - Khronos validation layer clean Scope: 8-bit 4:2:0 Main, intra + forward inter, single tile, CBR/VBR/CQP. No B-frames, no `show_existing_frame`, no 10-bit. All noted as TODOs in the source. Caveat: NVIDIA only. I have no RADV encode box. Like the existing files these are vendored patches, regenerated per GStreamer bump. I keep them current. Second commit: `docker/vulkan.Dockerfile` now applies the full chain, so this image build compiles the AV1 element too. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:48:03 +00:00

bsakaguchi referenced this issue

2026-10-15 12:48:04 +00:00

[PR #66] patches: fix vulkanh265enc reference sets and IDR output flag #65