[PR #52] feat(vulkan): write directly into compatible encode images #53

New issue

Open

opened 2026-10-15 12:47:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:57 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/52
Author: @elibosley
Created: 7/30/2026
Status: 🔄 Open

Base: feat/vulkan-encode ← Head: agent/direct-encode-storage

📝 Commits (3)

fd1c97c fix(vulkan): preserve allocator image create flags
9d60844 feat(vulkan): write directly into compatible encode images
e8d7e98 docs(vulkan): document direct encode prerequisites

📊 Changes

5 files changed (+332 additions, -107 deletions)

View changed files

📝 docker/vulkan.Dockerfile (+6 -3)
📝 patches/README.md (+12 -0)
➕ patches/vulkan-image-create-flags-1.28.patch (+13 -0)
📝 wayland-display-core/src/utils/vulkan_nv12.rs (+71 -44)
📝 wayland-display-core/src/utils/vulkan_share.rs (+230 -60)

📄 Description

Summary

The shared Vulkan encoder path can now write NV12 directly into a compatible encode image, eliminating the scratch image and copy on drivers that support the required storage and video-encode combination.

Why This Exists

The shared encoder image was already created with VIDEO_ENCODE_SRC, but compute conversion still wrote to a separate storage image and copied the result into that encoder image. This added an allocation, synchronization, and full-frame transfer to every compatible frame.

Resolution

Probe STORAGE | VIDEO_ENCODE_SRC support with the active video profile, image-format list, tiling, and creation flags. When supported, allocate the encoder image with storage usage and create plane views directly on it. When unsupported or allocation fails, retain the existing scratch-image-and-copy path.

Reviewer Considerations

The direct capability probe deliberately includes the same video profile, format list, usage, and create flags as allocation so it does not assume storage support from the video-encode capability alone.
The bundled GStreamer 1.28.4 patch is required because its allocator validated image format properties with zero flags even when the actual image used non-zero flags. It covers only the allocation-with-image-info path used here; the legacy wrapped-image helper remains unchanged.
Hardware coverage is important: drivers may reject the requested flag/usage/profile combination, in which case the code must stay on the existing fallback path.

Behavior Changes

Supported Vulkan video-encode configurations convert directly into the encode source image.
Unsupported configurations and allocation failures keep the portable scratch-and-copy behavior.

Implementation Summary

Add a GStreamer 1.28.4 allocator backport that preserves image creation flags during format validation.
Query and allocate direct storage-capable encode images with MUTABLE_FORMAT and EXTENDED_USAGE.
Use plane storage views of the encode image for direct conversion, with the existing scratch image retained as fallback.
Add static usage/flag tests, document the GStreamer patch, and force a fresh Docker plugin build.

Verification

cargo fmt --check — passed.
git diff --check origin/feat/vulkan-encode...HEAD — passed.
cargo test -p wayland-display-core direct_encode --lib — blocked before compilation in this macOS checkout because wayland-server.pc is unavailable; the container image runs cargo test --release -p wayland-display-core direct_encode.

Risk

Moderate, hardware-specific: a driver can decline the combined image capabilities. The capability probe and allocation-failure fallback preserve the prior scratch-and-copy path; NVIDIA and RADV encode smoke tests are still needed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/52 **Author:** [@elibosley](https://github.com/elibosley) **Created:** 7/30/2026 **Status:** 🔄 Open **Base:** `feat/vulkan-encode` ← **Head:** `agent/direct-encode-storage` --- ### 📝 Commits (3) - [`fd1c97c`](https://github.com/games-on-whales/gst-wayland-display/commit/fd1c97c504011ba28438a228e859d9747c0abac7) fix(vulkan): preserve allocator image create flags - [`9d60844`](https://github.com/games-on-whales/gst-wayland-display/commit/9d608440c96a499508bdb283dd43b8dd5d5bcc22) feat(vulkan): write directly into compatible encode images - [`e8d7e98`](https://github.com/games-on-whales/gst-wayland-display/commit/e8d7e98ba3228aee54f538e9f0c3ebc8b962b622) docs(vulkan): document direct encode prerequisites ### 📊 Changes **5 files changed** (+332 additions, -107 deletions) <details> <summary>View changed files</summary> 📝 `docker/vulkan.Dockerfile` (+6 -3) 📝 `patches/README.md` (+12 -0) ➕ `patches/vulkan-image-create-flags-1.28.patch` (+13 -0) 📝 `wayland-display-core/src/utils/vulkan_nv12.rs` (+71 -44) 📝 `wayland-display-core/src/utils/vulkan_share.rs` (+230 -60) </details> ### 📄 Description ## Summary The shared Vulkan encoder path can now write NV12 directly into a compatible encode image, eliminating the scratch image and copy on drivers that support the required storage and video-encode combination. ## Why This Exists The shared encoder image was already created with `VIDEO_ENCODE_SRC`, but compute conversion still wrote to a separate storage image and copied the result into that encoder image. This added an allocation, synchronization, and full-frame transfer to every compatible frame. ## Resolution Probe `STORAGE | VIDEO_ENCODE_SRC` support with the active video profile, image-format list, tiling, and creation flags. When supported, allocate the encoder image with storage usage and create plane views directly on it. When unsupported or allocation fails, retain the existing scratch-image-and-copy path. ## Reviewer Considerations - The direct capability probe deliberately includes the same video profile, format list, usage, and create flags as allocation so it does not assume storage support from the video-encode capability alone. - The bundled GStreamer 1.28.4 patch is required because its allocator validated image format properties with zero flags even when the actual image used non-zero flags. It covers only the allocation-with-image-info path used here; the legacy wrapped-image helper remains unchanged. - Hardware coverage is important: drivers may reject the requested flag/usage/profile combination, in which case the code must stay on the existing fallback path. ## Behavior Changes - Supported Vulkan video-encode configurations convert directly into the encode source image. - Unsupported configurations and allocation failures keep the portable scratch-and-copy behavior. ## Implementation Summary - Add a GStreamer 1.28.4 allocator backport that preserves image creation flags during format validation. - Query and allocate direct storage-capable encode images with `MUTABLE_FORMAT` and `EXTENDED_USAGE`. - Use plane storage views of the encode image for direct conversion, with the existing scratch image retained as fallback. - Add static usage/flag tests, document the GStreamer patch, and force a fresh Docker plugin build. ## Verification - `cargo fmt --check` — passed. - `git diff --check origin/feat/vulkan-encode...HEAD` — passed. - `cargo test -p wayland-display-core direct_encode --lib` — blocked before compilation in this macOS checkout because `wayland-server.pc` is unavailable; the container image runs `cargo test --release -p wayland-display-core direct_encode`. ## Risk Moderate, hardware-specific: a driver can decline the combined image capabilities. The capability probe and allocation-failure fallback preserve the prior scratch-and-copy path; NVIDIA and RADV encode smoke tests are still needed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:47:57 +00:00

bsakaguchi referenced this issue

2026-10-15 12:47:57 +00:00

[PR #53] [MERGED] fix(comp): subtract the cursor hotspot when rendering client cursor surfaces #54