[PR #20] [MERGED] [WIP] Producing CUDA buffers #23

New issue

Closed

opened 2026-10-15 12:47:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/20
Author: @ABeltramo
Created: 10/2/2025
Status: ✅ Merged
Merged: 10/22/2025
Merged by: @ABeltramo

Base: stable ← Head: cuda-interop

📝 Commits (10+)

6593619 feat: added basic cuda and gst-cuda FFI bindings
88ee6c7 feat: DMA Buffer -> EGL -> CUDA
6ba6efe feat: CUDA -> GstCudaStream -> GstBuffer
75ec420 fix: manually pass EGLDisplay from Smithay, removed dead code
ef02a94 lint: small refactor and cleanup
76726c8 feat: basic memory:CUDAMemory caps negotiation
b2fdce2 feat: CUDA cap exchanges and plugged together new GsCUDABuffer
e868786 feat: load EGL extensions only once
75e345c feat: share the CUDA Context with the Gstreamer pipeline
a3ee9ac lint: cleanup and minor refactor

📊 Changes

16 files changed (+1899 additions, -294 deletions)

View changed files

📝 Cargo.lock (+30 -22)
📝 README.md (+32 -5)
📝 gst-plugin-wayland-display/Cargo.toml (+2 -0)
📝 gst-plugin-wayland-display/src/lib.rs (+9 -0)
📝 gst-plugin-wayland-display/src/waylandsrc/imp.rs (+277 -18)
📝 wayland-display-core/Cargo.toml (+8 -0)
➕ wayland-display-core/build.rs (+25 -0)
📝 wayland-display-core/src/comp/mod.rs (+25 -2)
📝 wayland-display-core/src/comp/rendering.rs (+9 -4)
📝 wayland-display-core/src/lib.rs (+5 -2)
📝 wayland-display-core/src/tests/device.rs (+26 -19)
➕ wayland-display-core/src/utils/allocator/cuda/ffi.rs (+513 -0)
➕ wayland-display-core/src/utils/allocator/cuda/mod.rs (+562 -0)
📝 wayland-display-core/src/utils/allocator/mod.rs (+253 -170)
📝 wayland-display-core/src/utils/device/gpu.rs (+99 -52)
📝 wayland-display-core/src/utils/video_info.rs (+24 -0)

📄 Description

The idea here is that we'll directly produce and push GSTCudaMemory buffers, so that we can achieve something like

waylanddisplaysrc ! 'video/x-raw(memory:CUDAMemory),width=1920,height=1080,framerate=120/1' ! nvh265enc ! nvh265dec ! autovideosink

Basically embedding the whole glupload ! cudaupload that we need in order to import our DMA Buffers into CUDA. Ultimately, this should properly fix https://github.com/games-on-whales/wolf/pull/179#issuecomment-3058215348

Current status

It works!

The code is a mess and performance is terrible, but I've been iterating over different things for so long that I just need to push something that actually works end to end at displaying a proper image.

We currently incur only in a single GPU->GPU copy when going from CUDA->Gstreamer. I'm not sure if that can be avoided, it feels that we are ultimately doing the same dance that Gstreamer does when importing GL Buffers into CUDA but I would like to dive a bit deeper there..

This change will also help with multi-GPU setups because we'll be in full control over which Nvidia GPU will be picked and used in the full pipeline.

TODO

Take care of the performance issues and clean up the code
Make the requirement to build and use CUDA optional by properly using Cargo features
See if we can also avoid the GPU->GPU copy
Derive the CUDA device ID from the render_node param

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/20 **Author:** [@ABeltramo](https://github.com/ABeltramo) **Created:** 10/2/2025 **Status:** ✅ Merged **Merged:** 10/22/2025 **Merged by:** [@ABeltramo](https://github.com/ABeltramo) **Base:** `stable` ← **Head:** `cuda-interop` --- ### 📝 Commits (10+) - [`6593619`](https://github.com/games-on-whales/gst-wayland-display/commit/659361973fe14a49c7db6c6d59d359491fd87900) feat: added basic cuda and gst-cuda FFI bindings - [`88ee6c7`](https://github.com/games-on-whales/gst-wayland-display/commit/88ee6c74c87f9e275938955ba17ce3e7cb8f62b2) feat: DMA Buffer -> EGL -> CUDA - [`6ba6efe`](https://github.com/games-on-whales/gst-wayland-display/commit/6ba6efe128d89bc78f73a604ada575de8de8e997) feat: CUDA -> GstCudaStream -> GstBuffer - [`75ec420`](https://github.com/games-on-whales/gst-wayland-display/commit/75ec420a48e68d7a8d857a1f1e6008e6c3b76fb8) fix: manually pass EGLDisplay from Smithay, removed dead code - [`ef02a94`](https://github.com/games-on-whales/gst-wayland-display/commit/ef02a945184fac5211b5127a47e03ce606b494f7) lint: small refactor and cleanup - [`76726c8`](https://github.com/games-on-whales/gst-wayland-display/commit/76726c82ce694b00e39616ae3c7b891ebd2ac08a) feat: basic `memory:CUDAMemory` caps negotiation - [`b2fdce2`](https://github.com/games-on-whales/gst-wayland-display/commit/b2fdce2fe1999115a76f8fc28559eee38d98f38c) feat: CUDA cap exchanges and plugged together new GsCUDABuffer - [`e868786`](https://github.com/games-on-whales/gst-wayland-display/commit/e868786e44d98697c8338158843aebffcc4f38ab) feat: load EGL extensions only once - [`75e345c`](https://github.com/games-on-whales/gst-wayland-display/commit/75e345cf555e30cf68926fe2672bc6503576926c) feat: share the CUDA Context with the Gstreamer pipeline - [`a3ee9ac`](https://github.com/games-on-whales/gst-wayland-display/commit/a3ee9ac05e01c2409165a1dcfa5c74504140e218) lint: cleanup and minor refactor ### 📊 Changes **16 files changed** (+1899 additions, -294 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+30 -22) 📝 `README.md` (+32 -5) 📝 `gst-plugin-wayland-display/Cargo.toml` (+2 -0) 📝 `gst-plugin-wayland-display/src/lib.rs` (+9 -0) 📝 `gst-plugin-wayland-display/src/waylandsrc/imp.rs` (+277 -18) 📝 `wayland-display-core/Cargo.toml` (+8 -0) ➕ `wayland-display-core/build.rs` (+25 -0) 📝 `wayland-display-core/src/comp/mod.rs` (+25 -2) 📝 `wayland-display-core/src/comp/rendering.rs` (+9 -4) 📝 `wayland-display-core/src/lib.rs` (+5 -2) 📝 `wayland-display-core/src/tests/device.rs` (+26 -19) ➕ `wayland-display-core/src/utils/allocator/cuda/ffi.rs` (+513 -0) ➕ `wayland-display-core/src/utils/allocator/cuda/mod.rs` (+562 -0) 📝 `wayland-display-core/src/utils/allocator/mod.rs` (+253 -170) 📝 `wayland-display-core/src/utils/device/gpu.rs` (+99 -52) 📝 `wayland-display-core/src/utils/video_info.rs` (+24 -0) </details> ### 📄 Description The idea here is that we'll directly produce and push `GSTCudaMemory` buffers, so that we can achieve something like ``` waylanddisplaysrc ! 'video/x-raw(memory:CUDAMemory),width=1920,height=1080,framerate=120/1' ! nvh265enc ! nvh265dec ! autovideosink ``` Basically embedding the whole `glupload ! cudaupload` that we need in order to import our DMA Buffers into CUDA. Ultimately, this should properly fix https://github.com/games-on-whales/wolf/pull/179#issuecomment-3058215348 ## Current status **It works!** The code is a mess and performance is terrible, but I've been iterating over different things for so long that I just need to push something that actually works end to end at displaying a proper image. We currently incur only in a single GPU->GPU copy when going from CUDA->Gstreamer. I'm not sure if that can be avoided, it feels that we are ultimately doing the same dance that Gstreamer does when importing GL Buffers into CUDA but I would like to dive a bit deeper there.. This change will also **help with multi-GPU setups** because we'll be in full control over which Nvidia GPU will be picked and used in the full pipeline. ## TODO - [x] Take care of the performance issues and clean up the code - [x] Make the requirement to build and use CUDA optional by properly using [Cargo features](https://doc.rust-lang.org/cargo/reference/features.html) - [x] See if we can also avoid the GPU->GPU copy - [ ] Derive the CUDA device ID from the `render_node` param --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:47:36 +00:00