[PR #46] [MERGED] fix(vulkan): actually fall back to tiled when a LINEAR encode-src alloc fails #46

New issue

Closed

opened 2026-10-15 12:47:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/46
Author: @salty2011
Created: 7/26/2026
Status: ✅ Merged
Merged: 8/16/2026
Merged by: @JBailes

Base: feat/vulkan-encode ← Head: fix/linear-encsrc-fallback

📝 Commits (2)

ffd7b36 fix(vulkan): actually fall back to tiled when a LINEAR encode-src alloc fails
c764925 review: log effective encode-src tiling post-alloc; scope fallback comment to allocation failure

📊 Changes

1 file changed (+27 additions, -6 deletions)

View changed files

📝 wayland-display-core/src/utils/vulkan_share.rs (+27 -6)

📄 Description

WOLF_VULKAN_LINEAR_ENCSRC's comment promises the knob is safe because it "falls back to tiled if the linear alloc fails", but the code does not do that: when gst_vulkan_image_memory_alloc_with_image_info returns null for the LINEAR image, alloc_encode_src_buffer simply returns None. That fails VulkanNv12::new_on_shared ("encode-src image allocation failed") and the session's entire Vulkan output fails to be created.

Verified live on an RTX 5090 (2026-07-24): NVIDIA's Vulkan-Video encoder rejects a LINEAR encode-src image, so setting the knob on an NVIDIA host hard-failed every session instead of degrading gracefully.

This implements the documented behaviour: if the LINEAR allocation returns null while WOLF_VULKAN_LINEAR_ENCSRC is set, warn and retry the identical allocation with vk::ImageTiling::OPTIMAL. A genuine allocation failure still returns None after the retry, and the non-knob path is byte-identical because the retry branch is gated on linear_encsrc.

Scope of the safety claim: the fallback only covers a driver that rejects the LINEAR image at allocation time. A driver that accepts the LINEAR image and then corrupts or fails at vkCmdEncodeVideo is not covered. So on a mixed-GPU fleet the knob degrades gracefully for allocation-time rejection (the NVIDIA case above), nothing more.

Build-verified on this branch's head (43d4c25).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/46 **Author:** [@salty2011](https://github.com/salty2011) **Created:** 7/26/2026 **Status:** ✅ Merged **Merged:** 8/16/2026 **Merged by:** [@JBailes](https://github.com/JBailes) **Base:** `feat/vulkan-encode` ← **Head:** `fix/linear-encsrc-fallback` --- ### 📝 Commits (2) - [`ffd7b36`](https://github.com/games-on-whales/gst-wayland-display/commit/ffd7b36d77b4352e2bce085f1e225ee5c136188a) fix(vulkan): actually fall back to tiled when a LINEAR encode-src alloc fails - [`c764925`](https://github.com/games-on-whales/gst-wayland-display/commit/c76492598e19b668f0e194e448f3c716c19ee372) review: log effective encode-src tiling post-alloc; scope fallback comment to allocation failure ### 📊 Changes **1 file changed** (+27 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `wayland-display-core/src/utils/vulkan_share.rs` (+27 -6) </details> ### 📄 Description `WOLF_VULKAN_LINEAR_ENCSRC`'s comment promises the knob is safe because it "falls back to tiled if the linear alloc fails", but the code does not do that: when `gst_vulkan_image_memory_alloc_with_image_info` returns null for the LINEAR image, `alloc_encode_src_buffer` simply returns `None`. That fails `VulkanNv12::new_on_shared` ("encode-src image allocation failed") and the session's entire Vulkan output fails to be created. Verified live on an RTX 5090 (2026-07-24): NVIDIA's Vulkan-Video encoder **rejects** a LINEAR encode-src image, so setting the knob on an NVIDIA host hard-failed every session instead of degrading gracefully. This implements the documented behaviour: if the LINEAR allocation returns null while `WOLF_VULKAN_LINEAR_ENCSRC` is set, warn and retry the identical allocation with `vk::ImageTiling::OPTIMAL`. A genuine allocation failure still returns `None` after the retry, and the non-knob path is byte-identical because the retry branch is gated on `linear_encsrc`. Scope of the safety claim: the fallback only covers a driver that rejects the LINEAR image at allocation time. A driver that accepts the LINEAR image and then corrupts or fails at `vkCmdEncodeVideo` is not covered. So on a mixed-GPU fleet the knob degrades gracefully for allocation-time rejection (the NVIDIA case above), nothing more. Build-verified on this branch's head (`43d4c25`). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:47:54 +00:00