[PR #42] [MERGED] fix(cuda): free the transfer-full pool config that pinned the GstCudaContext #42

New issue

Closed

opened 2026-10-15 12:47:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:50 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/42
Author: @salty2011
Created: 7/26/2026
Status: ✅ Merged
Merged: 8/16/2026
Merged by: @JBailes

Base: stable ← Head: fix/cuda-context-refcount

📝 Commits (1)

ce6535d fix(cuda): free the transfer-full pool config that pinned the GstCudaContext

📊 Changes

2 files changed (+23 additions, -1 deletions)

View changed files

📝 gst-plugin-wayland-display/src/waylandsrc/imp.rs (+7 -1)
📝 wayland-display-core/src/utils/allocator/cuda/mod.rs (+16 -0)

📄 Description

CUDABufferPool::get_updated_size() calls gst_buffer_pool_get_config() and never frees the
result. That function is (transfer full) — it returns a fresh gst_structure_copy() of the
pool's config and the caller owns it. The sibling configure() makes the same call but hands its
copy to gst_buffer_pool_set_config(), which consumes it, so get_updated_size() is the only
leaking caller.

The leak is not just a stray GstStructure. gst_structure_copy() copies every field value, and
configure() has just installed a cuda-stream field via
gst_buffer_pool_config_set_cuda_stream(). GST_TYPE_CUDA_STREAM is a boxed type whose copy/free
are gst_mini_object_ref/gst_mini_object_unref, so the copy takes a new reference on the
GstCudaStream — and gst_cuda_stream_new() holds gst_object_ref (context) on its
GstCudaContext, released only in _gst_cuda_stream_free(). The retaining chain:

leaked GstStructure (config copy)  ->  cuda-stream field  ->  GstCudaStream  ->  GstCudaContext

decide_allocation() negotiates more than once per stream, so several config copies leak per
stream, each pinning the stream object and through it one reference to the CUDA context.

Impact depends on who owns the context

When waylanddisplaysrc creates its own context it is process-lifetime anyway and the leak is
invisible — which is probably why this has gone unnoticed. When the application injects one per
session via gst_element_set_context() (the sharing path this element already supports through
cuda-device-id), that session's GstCudaContext is never finalized: roughly 500 MiB of VRAM
plus one cuda-EvtHandlr driver thread leaked per session, growing linearly. I hit it as an
unbounded VRAM staircase across sessions on a multi-session host.

Second hunk, and why it must land with the first

Same function: the allocation query's caps was adopted with Caps::from_glib_full, consuming a
reference the element never owned and leaving the negotiated caps one short. That is latent while
the config copy leaks, because the copy holds a compensating reference on the same caps.

Fix only the leak and the caps drops below its true count. The failure is quiet and misleading:
streams still negotiate, connect and decode at full frame rate, but buffers are never released and
every session's pipelines are retained. On my leak-only build: 843 MiB VRAM, ~3000 live
GstMemory, ~1000 live GstBuffer and 6 live GstPipeline after two sessions, plus one
free_priv_data: object finalizing but still has 1 parents per session. So the two hunks are one
change; splitting them is worse than neither.

Why it is hard to see with the leaks tracer

Worth recording for the next person: a GstStructure is not refcounted and GstCudaStream is a
GstMiniObject, so leaks(filters=GstObject) shows one GstCudaContext alive at ref-count 1
with no GstObject retaining it — which reads as an impossible leak. filters= also restricts
what the tracer tracks, so hand-naming candidate types still hides holders you did not guess.
Running leaks unfiltered is what made the chain visible: after one session it reported
exactly three live objects (GstCudaStream, GstCudaContext, GstContext).

Because the leaked structure is detached from every object, no teardown path can reach it. For
anyone who suspects a lifecycle bug here: the element's own lifecycle is fine — instrumented builds
confirm CUDAContext::drop runs and the compositor's GsCUDABuf drops, and clearing the element's
context reference in stop() changes nothing.

Verification

RTX 5090, waylanddisplaysrc ! cudaconvert ! nvcudah264enc behind an interpipe boundary, with
the application injecting one GstCudaContext per session; leaks tracer unfiltered, SIGUSR1
dump 35 s after each teardown had settled.

	before	after
live `GstCudaStream` after 1 session	1	0
live `GstCudaStream` after 2 sessions	2	0
`GstCudaContext` ref-count	2 after 1 session, 3 after 2 (+1 per session)	flat at 1 across 3 sessions
settled VRAM	rising per session	flat
GStreamer criticals / `free_priv_data` warnings	0	0
stream health	decoding, 59–60 fps	decoding, 59–60 fps

The residual ref-count of 1 is my own reference to the injected context — by design, not a leak.

Also compile-verified against this branch's base with --features cuda.

Note for #37

The same two lines are present, unchanged, on feat/vulkan-encode (cuda/mod.rs is
byte-identical between the two branches), and this patch applies cleanly there too. Exposure is
highest on that branch, since Vulkan hosts fall back to the NVENC/CUDA path for AV1. Fixing it
here means #37 inherits it on its next rebase.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/42 **Author:** [@salty2011](https://github.com/salty2011) **Created:** 7/26/2026 **Status:** ✅ Merged **Merged:** 8/16/2026 **Merged by:** [@JBailes](https://github.com/JBailes) **Base:** `stable` ← **Head:** `fix/cuda-context-refcount` --- ### 📝 Commits (1) - [`ce6535d`](https://github.com/games-on-whales/gst-wayland-display/commit/ce6535d56c2a1d43ec935b5e67ba4bc88cc70e5d) fix(cuda): free the transfer-full pool config that pinned the GstCudaContext ### 📊 Changes **2 files changed** (+23 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `gst-plugin-wayland-display/src/waylandsrc/imp.rs` (+7 -1) 📝 `wayland-display-core/src/utils/allocator/cuda/mod.rs` (+16 -0) </details> ### 📄 Description `CUDABufferPool::get_updated_size()` calls `gst_buffer_pool_get_config()` and never frees the result. That function is **`(transfer full)`** — it returns a fresh `gst_structure_copy()` of the pool's config and the caller owns it. The sibling `configure()` makes the same call but hands its copy to `gst_buffer_pool_set_config()`, which consumes it, so `get_updated_size()` is the only leaking caller. The leak is not just a stray `GstStructure`. `gst_structure_copy()` copies every field value, and `configure()` has just installed a `cuda-stream` field via `gst_buffer_pool_config_set_cuda_stream()`. `GST_TYPE_CUDA_STREAM` is a boxed type whose copy/free are `gst_mini_object_ref`/`gst_mini_object_unref`, so the copy takes a new reference on the `GstCudaStream` — and `gst_cuda_stream_new()` holds `gst_object_ref (context)` on its `GstCudaContext`, released only in `_gst_cuda_stream_free()`. The retaining chain: ``` leaked GstStructure (config copy) -> cuda-stream field -> GstCudaStream -> GstCudaContext ``` `decide_allocation()` negotiates more than once per stream, so several config copies leak per stream, each pinning the stream object and through it one reference to the CUDA context. ### Impact depends on who owns the context When `waylanddisplaysrc` creates its own context it is process-lifetime anyway and the leak is invisible — which is probably why this has gone unnoticed. When the application **injects one per session** via `gst_element_set_context()` (the sharing path this element already supports through `cuda-device-id`), that session's `GstCudaContext` is never finalized: roughly **500 MiB of VRAM plus one `cuda-EvtHandlr` driver thread leaked per session**, growing linearly. I hit it as an unbounded VRAM staircase across sessions on a multi-session host. ### Second hunk, and why it must land with the first Same function: the allocation query's caps was adopted with `Caps::from_glib_full`, consuming a reference the element never owned and leaving the negotiated caps one short. That is latent while the config copy leaks, because the copy holds a compensating reference on the *same* caps. Fix only the leak and the caps drops below its true count. The failure is quiet and misleading: streams still negotiate, connect and decode at full frame rate, but buffers are never released and every session's pipelines are retained. On my leak-only build: 843 MiB VRAM, ~3000 live `GstMemory`, ~1000 live `GstBuffer` and 6 live `GstPipeline` after two sessions, plus one `free_priv_data: object finalizing but still has 1 parents` per session. So the two hunks are one change; splitting them is worse than neither. ### Why it is hard to see with the leaks tracer Worth recording for the next person: a `GstStructure` is not refcounted and `GstCudaStream` is a `GstMiniObject`, so `leaks(filters=GstObject)` shows one `GstCudaContext` alive at **ref-count 1 with no `GstObject` retaining it** — which reads as an impossible leak. `filters=` also restricts what the tracer *tracks*, so hand-naming candidate types still hides holders you did not guess. Running `leaks` **unfiltered** is what made the chain visible: after one session it reported exactly three live objects (`GstCudaStream`, `GstCudaContext`, `GstContext`). Because the leaked structure is detached from every object, no teardown path can reach it. For anyone who suspects a lifecycle bug here: the element's own lifecycle is fine — instrumented builds confirm `CUDAContext::drop` runs and the compositor's `GsCUDABuf` drops, and clearing the element's context reference in `stop()` changes nothing. ### Verification RTX 5090, `waylanddisplaysrc ! cudaconvert ! nvcudah264enc` behind an `interpipe` boundary, with the application injecting one `GstCudaContext` per session; `leaks` tracer unfiltered, `SIGUSR1` dump 35 s after each teardown had settled. | | before | after | |---|---|---| | live `GstCudaStream` after 1 session | 1 | 0 | | live `GstCudaStream` after 2 sessions | 2 | 0 | | `GstCudaContext` ref-count | 2 after 1 session, 3 after 2 (+1 per session) | flat at 1 across 3 sessions | | settled VRAM | rising per session | flat | | GStreamer criticals / `free_priv_data` warnings | 0 | 0 | | stream health | decoding, 59–60 fps | decoding, 59–60 fps | The residual ref-count of 1 is my own reference to the injected context — by design, not a leak. Also compile-verified against this branch's base with `--features cuda`. ### Note for #37 The same two lines are present, unchanged, on `feat/vulkan-encode` (`cuda/mod.rs` is byte-identical between the two branches), and this patch applies cleanly there too. Exposure is highest on that branch, since Vulkan hosts fall back to the NVENC/CUDA path for AV1. Fixing it here means #37 inherits it on its next rebase. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:47:50 +00:00