[PR #43] [CLOSED] fix(cuda): free the transfer-full pool config that pinned the GstCudaContext (for feat/vulkan-encode) #44

New issue

Closed

opened 2026-10-15 12:47:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/43
Author: @salty2011
Created: 7/26/2026
Status: ❌ Closed

Base: feat/vulkan-encode ← Head: fix/cuda-context-refcount-vulkan-encode

📝 Commits (1)

b3da88e fix(cuda): free the transfer-full pool config that pinned the GstCudaContext

📊 Changes

2 files changed (+23 additions, -1 deletions)

View changed files

📝 gst-plugin-wayland-display/src/waylandsrc/imp.rs (+7 -1)
📝 wayland-display-core/src/utils/allocator/cuda/mod.rs (+16 -0)

📄 Description

Same fix as #42, applied directly to feat/vulkan-encode so this branch does not have to wait for a rebase to pick it up. Take whichever is more convenient — or close this one if you would rather let #42 flow in naturally.

Both bugs predate this branch (they entered in 63688e6, 2025-10-08, and wayland-display-core/src/utils/allocator/cuda/mod.rs is byte-identical between stable and this branch), but exposure is highest here, because a Vulkan host falls back to the NVENC/CUDA path for AV1.

The leak

CUDABufferPool::get_updated_size() calls gst_buffer_pool_get_config() and never frees the result. That function is (transfer full) — a fresh gst_structure_copy() of the pool's config that the caller owns. The sibling configure() makes the same call but hands its copy to gst_buffer_pool_set_config(), which consumes it, so get_updated_size() is the only leaking caller.

gst_structure_copy() copies every field value, and configure() has just installed a cuda-stream field via gst_buffer_pool_config_set_cuda_stream(). GST_TYPE_CUDA_STREAM is a boxed type whose copy/free are gst_mini_object_ref/gst_mini_object_unref, so the copy takes a new reference on the GstCudaStream — and gst_cuda_stream_new() holds gst_object_ref (context) on its GstCudaContext, released only in _gst_cuda_stream_free():

leaked GstStructure (config copy)  ->  cuda-stream field  ->  GstCudaStream  ->  GstCudaContext

decide_allocation() negotiates more than once per stream, so several config copies leak per stream, each pinning the stream object and through it one reference to the CUDA context. When the element creates its own context this is invisible (process-lifetime anyway). When the application injects one per session via gst_element_set_context(), that session's context is never finalized: roughly 500 MiB of VRAM plus one cuda-EvtHandlr driver thread leaked per session, growing linearly.

Second hunk, and why it must land with the first

Same function: the allocation query's caps was adopted with Caps::from_glib_full, consuming a reference the element never owned and leaving the negotiated caps one short. That is latent while the config copy leaks, because the copy holds a compensating reference on the same caps.

Fix only the leak and the caps drops below its true count. The failure is quiet: streams still negotiate, connect and decode at full frame rate, but buffers are never released and every session's pipelines are retained (measured on my leak-only build: 843 MiB VRAM, ~3000 live GstMemory, ~1000 live GstBuffer and 6 live GstPipeline after two sessions, plus one free_priv_data: object finalizing but still has 1 parents per session).

Verification — on this branch

Built from this branch's head with --features cuda and run on an RTX 5090: waylanddisplaysrc ! cudaconvert ! nvcudah264enc behind an interpipe boundary, application-injected per-session GstCudaContext, leaks tracer unfiltered, SIGUSR1 dump 35 s after each teardown had settled.

	before	after
live `GstCudaStream` after 1 session	1	0
live `GstCudaStream` after 2 sessions	2	0
`GstCudaContext` ref-count	2 after 1 session, 3 after 2 (+1 per session)	flat at 1 across 3 sessions
settled VRAM	rising per session	flat
GStreamer criticals / `free_priv_data` warnings	0	0
stream health	decoding, 59–60 fps	decoding, 59–60 fps

The residual ref-count of 1 is my own reference to the injected context — by design, not a leak.

Finding it

Worth recording, because the standard recipe hides it: a GstStructure is not refcounted and GstCudaStream is a GstMiniObject, so leaks(filters=GstObject) shows one GstCudaContext alive at ref-count 1 with no GstObject retaining it, which reads as an impossible leak. filters= also restricts what the tracer tracks, so hand-naming candidate types still hides holders you did not guess. Running leaks unfiltered is what made the chain visible.

Because the leaked structure is detached from every object, no teardown path can reach it — the element's own lifecycle is fine (CUDAContext::drop runs, GsCUDABuf drops, and clearing the element's context reference in stop() changes nothing).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/43 **Author:** [@salty2011](https://github.com/salty2011) **Created:** 7/26/2026 **Status:** ❌ Closed **Base:** `feat/vulkan-encode` ← **Head:** `fix/cuda-context-refcount-vulkan-encode` --- ### 📝 Commits (1) - [`b3da88e`](https://github.com/games-on-whales/gst-wayland-display/commit/b3da88efeb84867fa80d231e49b05f7e74ac9063) fix(cuda): free the transfer-full pool config that pinned the GstCudaContext ### 📊 Changes **2 files changed** (+23 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `gst-plugin-wayland-display/src/waylandsrc/imp.rs` (+7 -1) 📝 `wayland-display-core/src/utils/allocator/cuda/mod.rs` (+16 -0) </details> ### 📄 Description Same fix as #42, applied directly to `feat/vulkan-encode` so this branch does not have to wait for a rebase to pick it up. Take whichever is more convenient — or close this one if you would rather let #42 flow in naturally. Both bugs predate this branch (they entered in `63688e6`, 2025-10-08, and `wayland-display-core/src/utils/allocator/cuda/mod.rs` is byte-identical between `stable` and this branch), but **exposure is highest here**, because a Vulkan host falls back to the NVENC/CUDA path for AV1. ### The leak `CUDABufferPool::get_updated_size()` calls `gst_buffer_pool_get_config()` and never frees the result. That function is **`(transfer full)`** — a fresh `gst_structure_copy()` of the pool's config that the caller owns. The sibling `configure()` makes the same call but hands its copy to `gst_buffer_pool_set_config()`, which consumes it, so `get_updated_size()` is the only leaking caller. `gst_structure_copy()` copies every field value, and `configure()` has just installed a `cuda-stream` field via `gst_buffer_pool_config_set_cuda_stream()`. `GST_TYPE_CUDA_STREAM` is a boxed type whose copy/free are `gst_mini_object_ref`/`gst_mini_object_unref`, so the copy takes a new reference on the `GstCudaStream` — and `gst_cuda_stream_new()` holds `gst_object_ref (context)` on its `GstCudaContext`, released only in `_gst_cuda_stream_free()`: ``` leaked GstStructure (config copy) -> cuda-stream field -> GstCudaStream -> GstCudaContext ``` `decide_allocation()` negotiates more than once per stream, so several config copies leak per stream, each pinning the stream object and through it one reference to the CUDA context. When the element creates its own context this is invisible (process-lifetime anyway). When the application injects one per session via `gst_element_set_context()`, that session's context is never finalized: roughly **500 MiB of VRAM plus one `cuda-EvtHandlr` driver thread leaked per session**, growing linearly. ### Second hunk, and why it must land with the first Same function: the allocation query's caps was adopted with `Caps::from_glib_full`, consuming a reference the element never owned and leaving the negotiated caps one short. That is latent while the config copy leaks, because the copy holds a compensating reference on the *same* caps. Fix only the leak and the caps drops below its true count. The failure is quiet: streams still negotiate, connect and decode at full frame rate, but buffers are never released and every session's pipelines are retained (measured on my leak-only build: 843 MiB VRAM, ~3000 live `GstMemory`, ~1000 live `GstBuffer` and 6 live `GstPipeline` after two sessions, plus one `free_priv_data: object finalizing but still has 1 parents` per session). ### Verification — on this branch Built from this branch's head with `--features cuda` and run on an RTX 5090: `waylanddisplaysrc ! cudaconvert ! nvcudah264enc` behind an `interpipe` boundary, application-injected per-session `GstCudaContext`, `leaks` tracer unfiltered, `SIGUSR1` dump 35 s after each teardown had settled. | | before | after | |---|---|---| | live `GstCudaStream` after 1 session | 1 | 0 | | live `GstCudaStream` after 2 sessions | 2 | 0 | | `GstCudaContext` ref-count | 2 after 1 session, 3 after 2 (+1 per session) | flat at 1 across 3 sessions | | settled VRAM | rising per session | flat | | GStreamer criticals / `free_priv_data` warnings | 0 | 0 | | stream health | decoding, 59–60 fps | decoding, 59–60 fps | The residual ref-count of 1 is my own reference to the injected context — by design, not a leak. ### Finding it Worth recording, because the standard recipe hides it: a `GstStructure` is not refcounted and `GstCudaStream` is a `GstMiniObject`, so `leaks(filters=GstObject)` shows one `GstCudaContext` alive at ref-count 1 with **no `GstObject` retaining it**, which reads as an impossible leak. `filters=` also restricts what the tracer *tracks*, so hand-naming candidate types still hides holders you did not guess. Running `leaks` **unfiltered** is what made the chain visible. Because the leaked structure is detached from every object, no teardown path can reach it — the element's own lifecycle is fine (`CUDAContext::drop` runs, `GsCUDABuf` drops, and clearing the element's context reference in `stop()` changes nothing). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:47:51 +00:00

bsakaguchi referenced this issue

2026-10-15 12:47:51 +00:00

[PR #44] [MERGED] fix(compositor): pointer focus for newly mapped windows — bbox refresh, map-time enter, and a get_pointer-race guard #45

bsakaguchi closed this issue

2026-10-15 12:47:51 +00:00