games-on-whales/gst-wayland-display

Fork 0

mirror of https://github.com/games-on-whales/gst-wayland-display.git synced 2026-10-15 12:51:57 +00:00

[PR #66] patches: fix vulkanh265enc reference sets and IDR output flag #65

New issue

Open

opened 2026-10-15 12:48:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:48:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/66
Author: @salty2011
Created: 9/25/2026
Status: 🔄 Open

Base: feat/vulkan-encode ← Head: fix/vulkanh265enc-rps

📝 Commits (6)

5597fc4 patches: vulkanav1enc + encode-family fixes
f1e67af docker: apply the full patch chain in the vulkan image
8bc3630 patches: address review on AV1 caps and live-retarget threading
6b4dd3b ci: build the patched GStreamer with tests and run the Vulkan tests
e9e1237 patches: narrow vulkanh265enc to the profiles it can encode
f81f3f0 patches: fix vulkanh265enc reference sets and IDR output flag

📊 Changes

11 files changed (+7115 additions, -97 deletions)

View changed files

➕ .github/workflows/gst-patches.yml (+58 -0)
📝 .gitignore (+1 -0)
📝 ci/README.md (+14 -0)
➕ ci/gst-patches.sh (+84 -0)
📝 docker/vulkan.Dockerfile (+9 -6)
📝 patches/README.md (+31 -7)
➕ patches/gstreamer-vulkan-rc-retarget-no-reset.patch (+137 -0)
➕ patches/series (+7 -0)
➕ patches/vkh264enc-rc-fix.patch (+286 -0)
➕ patches/vulkanav1enc.patch (+6026 -0)
📝 patches/vulkanh265enc.patch (+462 -84)

📄 Description

Stacked on #62. Review only the last commit, f81f3f0. The rest is #62 and drops out of this diff once it merges.

Summary

vulkanh265enc wrote H.265 streams that decoders reconstruct from the wrong pictures whenever a picture had more than one reference, which is the default (num-ref-frames=3). With B frames, decoders also dropped a frame at the end of every GOP.

 gst_h265_encoder_encode_frame_with_ref_lists
   build list0 / list1 from the reference queue
   slicehdr_init
-    num_ref_idx_l0_active = len(list0)        (P only)
-    no_output_of_prior_pics_flag = 1 on IDR   (drops pictures still waiting for output)
+    num_ref_idx_l0/l1_active = len(list0/list1)   (P and B)
+    no_output_of_prior_pics_flag = 0
+  _fill_short_term_rps
+    every held reference, nearest first
+    UsedByCurrPic = in list0 / list1
   vkh265enc encode_frame
-    RPS: num_negative_pics = len(list0), but only entry 0 filled
-         (delta_poc_s0_minus1[1..] = 0 -> POC n-3, n-5 ... never existed)
-         used_by_curr_pic_s0_flag = 0b1
+    _setup_short_term_rps: per-step POC delta + used bit for every entry

POCs advance by 2 per frame, so each entry left at 0 named an odd POC that doesn't exist. Meanwhile the driver was handed len(list0) distinct pictures as active references, while a decoder resolved them all to the one picture flagged as used.

New test, elements_vkh265enc (added to ci/gst-patches.sh). It parses the encoder's output with GstH265Parser for P and B GOPs, and checks:

every RPS entry is a picture still held for reference
the active reference counts equal the pictures flagged as used
IDRs don't set no_output_of_prior_pics_flag
one access unit per input frame

Evidence

RTX 5090, driver 610.57.04, GStreamer 1.28.4 with patches/series. Decoding uses ffmpeg -err_detect +explode and vulkanh265dec, which gave identical results in every run. PSNR is the decoded Y plane against the lossless source, compared raw to raw.

Before: 1080p60, 600 frames, CBR 8 Mbps, defaults:

ffmpeg logs 580 × Could not find ref with POC <odd>
16.7 dB average, 213/600 frames below 30 dB
b-frames=2: 300 access units, 296 decoded

The new test against the old code:

test_h265_bitstream_p_frames: picture 3 (POC 4): RPS names POC 1, which isn't held for reference
test_h265_bitstream_b_frames: picture 5 (POC 12): RPS names POC 5, which isn't held for reference
(with only the RPS fixed: IDR picture 1 sets no_output_of_prior_pics_flag)

After: same encode: 0 decode errors, 92.6 dB, 0/600 below 30 dB.
- ci/gst-patches.sh: 6/6 OK. 25 serial repeats: 150/150.
- Every configuration decodes all frames with 0 errors: 1, 3 and 8 references, b-frames=2, b-frames=3 b-pyramid=true, idr-period=240, and Main 10 from P010.
- Moving SMPTE bars at 40 Mbps: 53.1 dB P, 53.3 dB B, 0 frames below 30 dB.
- waylanddisplaysrc vulkan=true ! vulkanh265enc: 240/240 frames, 0 errors.
- Vulkan validation layer: clean for P and b-frames=2.

Not fixed here. All of these are the same on #62 without this commit:

b-pyramid=true validation errors: VUID-vkCmdEncodeVideoKHR-flags-08347/08348. It puts B pictures in the reference lists without checking B_FRAME_IN_L0/L1_LIST, and stock vkh264enc does the same. The output still decodes cleanly.
Zero-copy B-frame hang: waylanddisplaysrc vulkan=true ! vulkanh265enc b-frames=2 hangs; vulkanh264enc b-frames=2 doesn't.
Harness gpu phase crash: test_vulkan_encode_pipeline now gets past H.265 start, and then hits a crash in stock vulkancolorconvert: RGBA→NV12 in calculate_reorder_indexes, which crashes vulkanh264enc too.

Merge Danger

Door: two-way

Patch-only change to a vendored encoder. A revert restores the old bytes exactly.

Blast Radius: H.265-only

It changes the H.265 slice headers every vulkanh265enc stream carries, P frames included. The new headers are what a conforming decoder already expects; the old ones only decoded correctly when every picture had a single reference. H.264 and AV1 are untouched.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/66 **Author:** [@salty2011](https://github.com/salty2011) **Created:** 9/25/2026 **Status:** 🔄 Open **Base:** `feat/vulkan-encode` ← **Head:** `fix/vulkanh265enc-rps` --- ### 📝 Commits (6) - [`5597fc4`](https://github.com/games-on-whales/gst-wayland-display/commit/5597fc44e647b6eb8656b168c060f2ad281ed1fa) patches: vulkanav1enc + encode-family fixes - [`f1e67af`](https://github.com/games-on-whales/gst-wayland-display/commit/f1e67afd7607bb7d78e52e16868deb7c18893d7f) docker: apply the full patch chain in the vulkan image - [`8bc3630`](https://github.com/games-on-whales/gst-wayland-display/commit/8bc36302d325b1bc9d1f4e2a6075f2bcb780f283) patches: address review on AV1 caps and live-retarget threading - [`6b4dd3b`](https://github.com/games-on-whales/gst-wayland-display/commit/6b4dd3b7cfbf76b181e5139fac26c66fb5380dad) ci: build the patched GStreamer with tests and run the Vulkan tests - [`e9e1237`](https://github.com/games-on-whales/gst-wayland-display/commit/e9e12377208f214af41183da8c7cf7ead5a204f8) patches: narrow vulkanh265enc to the profiles it can encode - [`f81f3f0`](https://github.com/games-on-whales/gst-wayland-display/commit/f81f3f0a6913b1f5b1aa52a7702b47dd3f981c04) patches: fix vulkanh265enc reference sets and IDR output flag ### 📊 Changes **11 files changed** (+7115 additions, -97 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/gst-patches.yml` (+58 -0) 📝 `.gitignore` (+1 -0) 📝 `ci/README.md` (+14 -0) ➕ `ci/gst-patches.sh` (+84 -0) 📝 `docker/vulkan.Dockerfile` (+9 -6) 📝 `patches/README.md` (+31 -7) ➕ `patches/gstreamer-vulkan-rc-retarget-no-reset.patch` (+137 -0) ➕ `patches/series` (+7 -0) ➕ `patches/vkh264enc-rc-fix.patch` (+286 -0) ➕ `patches/vulkanav1enc.patch` (+6026 -0) 📝 `patches/vulkanh265enc.patch` (+462 -84) </details> ### 📄 Description Stacked on #62. Review only the last commit, f81f3f0. The rest is #62 and drops out of this diff once it merges. ## Summary `vulkanh265enc` wrote H.265 streams that decoders reconstruct from the wrong pictures whenever a picture had more than one reference, which is the default (`num-ref-frames=3`). With B frames, decoders also dropped a frame at the end of every GOP. ```diff gst_h265_encoder_encode_frame_with_ref_lists build list0 / list1 from the reference queue slicehdr_init - num_ref_idx_l0_active = len(list0) (P only) - no_output_of_prior_pics_flag = 1 on IDR (drops pictures still waiting for output) + num_ref_idx_l0/l1_active = len(list0/list1) (P and B) + no_output_of_prior_pics_flag = 0 + _fill_short_term_rps + every held reference, nearest first + UsedByCurrPic = in list0 / list1 vkh265enc encode_frame - RPS: num_negative_pics = len(list0), but only entry 0 filled - (delta_poc_s0_minus1[1..] = 0 -> POC n-3, n-5 ... never existed) - used_by_curr_pic_s0_flag = 0b1 + _setup_short_term_rps: per-step POC delta + used bit for every entry ``` POCs advance by 2 per frame, so each entry left at 0 named an odd POC that doesn't exist. Meanwhile the driver was handed `len(list0)` distinct pictures as active references, while a decoder resolved them all to the one picture flagged as used. New test, `elements_vkh265enc` (added to `ci/gst-patches.sh`). It parses the encoder's output with `GstH265Parser` for P and B GOPs, and checks: - every RPS entry is a picture still held for reference - the active reference counts equal the pictures flagged as used - IDRs don't set `no_output_of_prior_pics_flag` - one access unit per input frame ## Evidence RTX 5090, driver 610.57.04, GStreamer 1.28.4 with `patches/series`. Decoding uses ffmpeg `-err_detect +explode` and `vulkanh265dec`, which gave identical results in every run. PSNR is the decoded Y plane against the lossless source, compared raw to raw. - **Before:** 1080p60, 600 frames, CBR 8 Mbps, defaults: - ffmpeg logs 580 × `Could not find ref with POC <odd>` - 16.7 dB average, 213/600 frames below 30 dB - `b-frames=2`: 300 access units, 296 decoded The new test against the old code: ``` test_h265_bitstream_p_frames: picture 3 (POC 4): RPS names POC 1, which isn't held for reference test_h265_bitstream_b_frames: picture 5 (POC 12): RPS names POC 5, which isn't held for reference (with only the RPS fixed: IDR picture 1 sets no_output_of_prior_pics_flag) ``` - **After:** same encode: 0 decode errors, 92.6 dB, 0/600 below 30 dB. - `ci/gst-patches.sh`: 6/6 OK. 25 serial repeats: 150/150. - Every configuration decodes all frames with 0 errors: 1, 3 and 8 references, `b-frames=2`, `b-frames=3 b-pyramid=true`, `idr-period=240`, and Main 10 from P010. - Moving SMPTE bars at 40 Mbps: 53.1 dB P, 53.3 dB B, 0 frames below 30 dB. - `waylanddisplaysrc vulkan=true ! vulkanh265enc`: 240/240 frames, 0 errors. - Vulkan validation layer: clean for P and `b-frames=2`. Not fixed here. All of these are the same on #62 without this commit: - **`b-pyramid=true` validation errors:** `VUID-vkCmdEncodeVideoKHR-flags-08347/08348`. It puts B pictures in the reference lists without checking `B_FRAME_IN_L0/L1_LIST`, and stock `vkh264enc` does the same. The output still decodes cleanly. - **Zero-copy B-frame hang:** `waylanddisplaysrc vulkan=true ! vulkanh265enc b-frames=2` hangs; `vulkanh264enc b-frames=2` doesn't. - **Harness `gpu` phase crash:** `test_vulkan_encode_pipeline` now gets past H.265 start, and then hits a crash in stock `vulkancolorconvert`: RGBA→NV12 in `calculate_reorder_indexes`, which crashes `vulkanh264enc` too. ## Merge Danger **Door:** two-way Patch-only change to a vendored encoder. A revert restores the old bytes exactly. **Blast Radius:** H.265-only It changes the H.265 slice headers every `vulkanh265enc` stream carries, P frames included. The new headers are what a conforming decoder already expects; the old ones only decoded correctly when every picture had a single reference. H.264 and AV1 are untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:48:04 +00:00

No labels

enhancement

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

games-on-whales/gst-wayland-display#65

No description provided.

Rows
Columns