[PR #419] [CLOSED] 10-bit SDR streaming + DMA-BUF zero-copy via gst-interpipe context sharing #452

New issue

Closed

opened 2026-10-15 13:00:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 13:00:40 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/wolf/pull/419
Author: @JBailes
Created: 6/1/2026
Status: ❌ Closed

Base: stable ← Head: fix/amd-dma-buf-negotiation

📝 Commits (9)

ee4fc25 fix: intersect waylanddisplaysrc and vapostproc DMA-BUF caps
3b9b10c fix: query waylanddisplaysrc DMA formats at runtime to intersect with vapostproc
46ba1e9 style: wrap over-length log line to satisfy clang-format
56a5b9e docker: pin gst-interpipe to context-query-forwarding fix
8f42689 feat: 10-bit SDR streaming (HEVC/AV1 Main10)
8b12bca test: update AV1 ServerCodecModeSupport to protocol-correct value
d246afb feat: bump config to v8 and migrate so existing installs get 10-bit encoders
d9090d1 clean: removing gst-video-context module and CUDA linking
eb7e58b fix: negotiate the video producer format jointly with the encoder

📊 Changes

29 files changed (+858 additions, -895 deletions)

View changed files

📝 .github/workflows/linux-build-test.yml (+2 -1)
📝 .gitignore (+1 -0)
📝 CMakeLists.txt (+0 -1)
📝 docker/wolf.Dockerfile (+5 -1)
📝 src/core/src/core/gstreamer.hpp (+1 -47)
➖ src/gst-video-context/CMakeLists.txt (+0 -26)
➖ src/gst-video-context/gst-video-context/gst-video-context.cpp (+0 -191)
➖ src/gst-video-context/gst-video-context/gst-video-context.hpp (+0 -28)
➖ src/gst-video-context/gst-video-context/no-cuda-context.cpp (+0 -19)
➖ src/gst-video-context/gst-video-context/stub/cuda.h (+0 -473)
➖ src/gst-video-context/gst-video-context/stub/cudaGL.h (+0 -15)
📝 src/moonlight-protocol/moonlight.cpp (+16 -4)
📝 src/moonlight-protocol/moonlight/protocol.hpp (+3 -1)
📝 src/moonlight-server/CMakeLists.txt (+1 -2)
📝 src/moonlight-server/events/events.hpp (+5 -0)
📝 src/moonlight-server/rest/endpoints.hpp (+3 -1)
📝 src/moonlight-server/rtsp/commands.hpp (+10 -4)
📝 src/moonlight-server/sessions/lobbies.cpp (+1 -2)
📝 src/moonlight-server/sessions/moonlight.cpp (+3 -6)
📝 src/moonlight-server/state/configTOML.cpp (+58 -22)

...and 9 more files

📄 Description

Summary

Started as the AMD DMA-BUF zero-copy work split out of #397, and grew into the proper fix Abe asked for there: stop hand-pinning DMA-BUF formats, fix negotiation at the source, and build 10-bit SDR streaming on top.

1. Fix negotiation in gst-interpipe (Abe's option 1)

The root cause of the original caps trouble is that gst-interpipe forwards buffers and events between decoupled pipelines but not queries, so VA/DMABuf elements on either side of the interpipe can't share a context. Fixed upstream in games-on-whales/gst-interpipe#2 (forward GST_QUERY_CONTEXT across the boundary; tested, 15/15). Pinned in docker/gstreamer.Dockerfile until that merges.

2. Drop the manual DMA-BUF intersection

With negotiation working, the config-time get_dma_caps intersection is unnecessary. The producer now advertises a generic video/x-raw(memory:DMABuf) buffer and the compositor + vapostproc negotiate the concrete drm-format at runtime (verified on Intel VA with real waylanddisplaysrc: lands on a 10-bit format and propagates across the interpipe). Removes the now-dead get_dma_caps / get_dma_caps_runtime / parse_dma_drm_formats helpers.

3. 10-bit SDR streaming (HEVC/AV1 Main10)

10-bit encoding reduces compression banding even for 8-bit source. It's decoupled from HDR: the client's dynamicRangeMode=1 only selects the 10-bit decode profile; HDR display is a separate control-stream message Wolf never sends, so the client decodes 10-bit while staying SDR (BT.709).

Advertise H265_MAIN10 / AV1_MAIN10 in ServerCodecModeSupport, gated on a Main10 pipeline existing. Also fixes a real bug: the AV1 format constants were 0x10000/0x20000 but the protocol (Limelight.h) defines them as 0x1000/0x2000.
Negotiate: read x-nv-video[0].dynamicRangeMode, select a Main10 pipeline for HEVC/AV1.
Encode: P010 video_params + profile=main-10 pipelines for nvcodec / va / va-LP / qsv (HEVC and AV1). The compositor renders 10-bit; vapostproc converts to NV12 (8-bit sessions) or P010 (Main10).

Verification

gst-interpipe#2: built + 15/15 tests, plus an Intel-VA DMABuf-across-interpipe negotiation test.
Real waylanddisplaysrc on Intel VA: generic DMABuf negotiates 10-bit end-to-end across the interpipe.
Wolf CI: Docker build, Linux build + tests, clang-format all green.
Pending: end-to-end stream to a Moonlight client (decode/visual check) across vendors.

Notes for reviewers

Depends on games-on-whales/gst-interpipe#2 (Dockerfile pinned to that commit; unpin once merged).
Config migration: bumps the config to v8. Existing installs (v<=7) are migrated automatically: the gstreamer section is regenerated from the new default (with the 10-bit pipelines) while hostname/uuid/paired_clients/profiles are preserved, and the old config is backed up as config.toml.v7.old. New config.v8.toml is the make_includable source.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/wolf/pull/419 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 6/1/2026 **Status:** ❌ Closed **Base:** `stable` ← **Head:** `fix/amd-dma-buf-negotiation` --- ### 📝 Commits (9) - [`ee4fc25`](https://github.com/games-on-whales/wolf/commit/ee4fc250cc59d28ba2d5b5c459eaf0986406183e) fix: intersect waylanddisplaysrc and vapostproc DMA-BUF caps - [`3b9b10c`](https://github.com/games-on-whales/wolf/commit/3b9b10c05b3ee3b45e9b4bf979a6b73eadb016d4) fix: query waylanddisplaysrc DMA formats at runtime to intersect with vapostproc - [`46ba1e9`](https://github.com/games-on-whales/wolf/commit/46ba1e9ba7c005832cb1825940cc0921ec90b161) style: wrap over-length log line to satisfy clang-format - [`56a5b9e`](https://github.com/games-on-whales/wolf/commit/56a5b9eb119879417ec1d7301cad1c56292ad2f7) docker: pin gst-interpipe to context-query-forwarding fix - [`8f42689`](https://github.com/games-on-whales/wolf/commit/8f426892d779aa447eb116abe0f00026f6168070) feat: 10-bit SDR streaming (HEVC/AV1 Main10) - [`8b12bca`](https://github.com/games-on-whales/wolf/commit/8b12bcade10c4876163a0e3cbe6fcf8ba3f20909) test: update AV1 ServerCodecModeSupport to protocol-correct value - [`d246afb`](https://github.com/games-on-whales/wolf/commit/d246afbe22a9921b22769030466d43b3fe411895) feat: bump config to v8 and migrate so existing installs get 10-bit encoders - [`d9090d1`](https://github.com/games-on-whales/wolf/commit/d9090d142f721392937a417fd84fe5713ea6fc5e) clean: removing `gst-video-context` module and CUDA linking - [`eb7e58b`](https://github.com/games-on-whales/wolf/commit/eb7e58bf99b2c3aa6a0b64754212151959297ad6) fix: negotiate the video producer format jointly with the encoder ### 📊 Changes **29 files changed** (+858 additions, -895 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/linux-build-test.yml` (+2 -1) 📝 `.gitignore` (+1 -0) 📝 `CMakeLists.txt` (+0 -1) 📝 `docker/wolf.Dockerfile` (+5 -1) 📝 `src/core/src/core/gstreamer.hpp` (+1 -47) ➖ `src/gst-video-context/CMakeLists.txt` (+0 -26) ➖ `src/gst-video-context/gst-video-context/gst-video-context.cpp` (+0 -191) ➖ `src/gst-video-context/gst-video-context/gst-video-context.hpp` (+0 -28) ➖ `src/gst-video-context/gst-video-context/no-cuda-context.cpp` (+0 -19) ➖ `src/gst-video-context/gst-video-context/stub/cuda.h` (+0 -473) ➖ `src/gst-video-context/gst-video-context/stub/cudaGL.h` (+0 -15) 📝 `src/moonlight-protocol/moonlight.cpp` (+16 -4) 📝 `src/moonlight-protocol/moonlight/protocol.hpp` (+3 -1) 📝 `src/moonlight-server/CMakeLists.txt` (+1 -2) 📝 `src/moonlight-server/events/events.hpp` (+5 -0) 📝 `src/moonlight-server/rest/endpoints.hpp` (+3 -1) 📝 `src/moonlight-server/rtsp/commands.hpp` (+10 -4) 📝 `src/moonlight-server/sessions/lobbies.cpp` (+1 -2) 📝 `src/moonlight-server/sessions/moonlight.cpp` (+3 -6) 📝 `src/moonlight-server/state/configTOML.cpp` (+58 -22) _...and 9 more files_ </details> ### 📄 Description ## Summary Started as the AMD DMA-BUF zero-copy work split out of #397, and grew into the proper fix Abe asked for there: stop hand-pinning DMA-BUF formats, fix negotiation at the source, and build 10-bit SDR streaming on top. ### 1. Fix negotiation in gst-interpipe (Abe's option 1) The root cause of the original caps trouble is that `gst-interpipe` forwards buffers and events between decoupled pipelines but **not queries**, so VA/DMABuf elements on either side of the interpipe can't share a context. Fixed upstream in **games-on-whales/gst-interpipe#2** (forward `GST_QUERY_CONTEXT` across the boundary; tested, 15/15). Pinned in `docker/gstreamer.Dockerfile` until that merges. ### 2. Drop the manual DMA-BUF intersection With negotiation working, the config-time `get_dma_caps` intersection is unnecessary. The producer now advertises a generic `video/x-raw(memory:DMABuf)` buffer and the compositor + `vapostproc` negotiate the concrete drm-format at runtime (verified on Intel VA with real `waylanddisplaysrc`: lands on a 10-bit format and propagates across the interpipe). Removes the now-dead `get_dma_caps` / `get_dma_caps_runtime` / `parse_dma_drm_formats` helpers. ### 3. 10-bit SDR streaming (HEVC/AV1 Main10) 10-bit encoding reduces compression banding even for 8-bit source. It's decoupled from HDR: the client's `dynamicRangeMode=1` only selects the 10-bit *decode* profile; HDR *display* is a separate control-stream message Wolf never sends, so the client decodes 10-bit while staying SDR (BT.709). - **Advertise** `H265_MAIN10` / `AV1_MAIN10` in `ServerCodecModeSupport`, gated on a Main10 pipeline existing. Also fixes a real bug: the AV1 format constants were `0x10000`/`0x20000` but the protocol (Limelight.h) defines them as `0x1000`/`0x2000`. - **Negotiate**: read `x-nv-video[0].dynamicRangeMode`, select a Main10 pipeline for HEVC/AV1. - **Encode**: P010 `video_params` + `profile=main-10` pipelines for nvcodec / va / va-LP / qsv (HEVC and AV1). The compositor renders 10-bit; `vapostproc` converts to NV12 (8-bit sessions) or P010 (Main10). ## Verification - gst-interpipe#2: built + 15/15 tests, plus an Intel-VA DMABuf-across-interpipe negotiation test. - Real `waylanddisplaysrc` on Intel VA: generic DMABuf negotiates 10-bit end-to-end across the interpipe. - Wolf CI: Docker build, Linux build + tests, clang-format all green. - **Pending**: end-to-end stream to a Moonlight client (decode/visual check) across vendors. ## Notes for reviewers - Depends on **games-on-whales/gst-interpipe#2** (Dockerfile pinned to that commit; unpin once merged). - **Config migration**: bumps the config to v8. Existing installs (v<=7) are migrated automatically: the gstreamer section is regenerated from the new default (with the 10-bit pipelines) while hostname/uuid/paired_clients/profiles are preserved, and the old config is backed up as `config.toml.v7.old`. New `config.v8.toml` is the make_includable source. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 13:00:40 +00:00