[PR #54] [MERGED] fix(cuda): stop clearing a pool the query still owns #55

New issue

Closed

opened 2026-10-15 12:47:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:47:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/gst-wayland-display/pull/54
Author: @JBailes
Created: 8/16/2026
Status: ✅ Merged
Merged: 8/16/2026
Merged by: @JBailes

Base: stable ← Head: fix/cuda-pool-clear-object

📝 Commits (1)

01027ba fix(cuda): stop clearing a pool the query still owns

📊 Changes

1 file changed (+3 additions, -1 deletions)

View changed files

📝 gst-plugin-wayland-display/src/waylandsrc/imp.rs (+3 -1)

📄 Description

decide_allocation() released a downstream pool it does not own, through a call whose type is wrong in a way the compiler cannot see:

gst_clear_object(pool.as_ptr() as *mut _)

gst_clear_object takes a GstObject **. pool.as_ptr() is a GstBufferPool *. The as *mut _ lets inference supply *mut *mut GstObject from the parameter type, so the cast is legal and it compiles without a warning. At runtime the call dereferences one level too far and treats the pool's first field, GTypeInstance.g_class, as the pointer to clear.

Measured on GStreamer 1.26.2 against a plain gst_buffer_pool_new() with the same cast:

g_object_unref: assertion 'G_IS_OBJECT (object)' failed, as the call tries to unref the class vtable. GLib's check refuses it, so the vtable itself survives.
The pool's class pointer is then overwritten with NULL.
Every later call on the pool fails its type check: gst_buffer_pool_set_active: assertion 'GST_IS_BUFFER_POOL (pool)' failed.
The owner's own unref fails the same assertion, so the pool can never be freed.

That is the outcome with GLib checks compiled in, which is the Debian and Ubuntu default. Under G_DISABLE_CHECKS the first unref is not refused and the class vtable's refcount is decremented instead. I have not built that configuration.

The release was never needed, which is the part worth keeping. query.allocation_pools() parses with gst_query_parse_nth_allocation_pool, which is (transfer full), so the returned Vec owns the BufferPool and drops it at the end of the scope. Deleting the line is the whole fix.

Reachable only when the CUDA path is live and downstream offers a pool that is not a CUDA buffer pool, so it has stayed quiet.

The same line is on feat/vulkan-encode at imp.rs:1100 and needs the same removal there.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/gst-wayland-display/pull/54 **Author:** [@JBailes](https://github.com/JBailes) **Created:** 8/16/2026 **Status:** ✅ Merged **Merged:** 8/16/2026 **Merged by:** [@JBailes](https://github.com/JBailes) **Base:** `stable` ← **Head:** `fix/cuda-pool-clear-object` --- ### 📝 Commits (1) - [`01027ba`](https://github.com/games-on-whales/gst-wayland-display/commit/01027baddc7b7e6f52612719555205bed9d295fd) fix(cuda): stop clearing a pool the query still owns ### 📊 Changes **1 file changed** (+3 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `gst-plugin-wayland-display/src/waylandsrc/imp.rs` (+3 -1) </details> ### 📄 Description `decide_allocation()` released a downstream pool it does not own, through a call whose type is wrong in a way the compiler cannot see: gst_clear_object(pool.as_ptr() as *mut _) `gst_clear_object` takes a `GstObject **`. `pool.as_ptr()` is a `GstBufferPool *`. The `as *mut _` lets inference supply `*mut *mut GstObject` from the parameter type, so the cast is legal and it compiles without a warning. At runtime the call dereferences one level too far and treats the pool's first field, `GTypeInstance.g_class`, as the pointer to clear. Measured on GStreamer 1.26.2 against a plain `gst_buffer_pool_new()` with the same cast: - `g_object_unref: assertion 'G_IS_OBJECT (object)' failed`, as the call tries to unref the class vtable. GLib's check refuses it, so the vtable itself survives. - The pool's class pointer is then overwritten with NULL. - Every later call on the pool fails its type check: `gst_buffer_pool_set_active: assertion 'GST_IS_BUFFER_POOL (pool)' failed`. - The owner's own unref fails the same assertion, so the pool can never be freed. That is the outcome with GLib checks compiled in, which is the Debian and Ubuntu default. Under `G_DISABLE_CHECKS` the first unref is not refused and the class vtable's refcount is decremented instead. I have not built that configuration. The release was never needed, which is the part worth keeping. `query.allocation_pools()` parses with `gst_query_parse_nth_allocation_pool`, which is (transfer full), so the returned `Vec` owns the `BufferPool` and drops it at the end of the scope. Deleting the line is the whole fix. Reachable only when the CUDA path is live and downstream offers a pool that is not a CUDA buffer pool, so it has stayed quiet. The same line is on feat/vulkan-encode at imp.rs:1100 and needs the same removal there. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 12:47:58 +00:00