[GH-ISSUE #392] bug(ublk-daemon): undelivered acquire responses leak devices and shared references #100

New issue

Closed

opened 2026-10-29 14:19:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:35 +00:00

Owner

Originally created by @dushulin on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/392

Summary

A shared OverlayBD acquisition can complete after its caller has timed out or disconnected. The daemon records the acquired device/reference before sending the response. If response delivery fails, the handler returns without releasing the acquisition. The caller never receives the result needed to own and release it.

This leaves an active acquisition with no corresponding client owner. Repeated failures can accumulate devices or shared references.

Environment and verification scope

Component: ublk-daemon, shared memory devices used for snapshot restore, with device pooling enabled.
Environment: Linux x86_64, kernel 6.8, ublk_drv, OverlayBD.
Runtime evidence comes from an older downstream deployment. The affected path was independently inspected at public revision 0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45.
A standalone reproducer has not yet been run against that upstream revision. The first affected revision and last known good revision are unknown.

Observed behavior

The following sequence was observed repeatedly. Logs are normalized, with timestamps and identifiers removed:

client: acquire shared memory ublk device: daemon RPC timed out after 30s
daemon: acquired shared overlaybd device (...) dev_id=<id>
daemon: daemon connection handler failed
        write message length prefix
        Broken pipe (os error 32)

The device remained LIVE and held by the daemon. No subsequent release appeared in the retained lifecycle logs, and no Firecracker FD or mmap referencing it was observed. Normal idle placeholder devices were excluded from this classification.

For an affected device whose backing layer later became inaccessible, a host inventory operation (fdisk -l) triggered header reads and exposed the following errors:

overlaybd request failed op=Read offset=0 len=4096
    No such file or directory (os error 2) errno=-2
Buffer I/O error on dev ublkb<id>, logical block 0, async page read

The inventory read exposed the retained device; it did not cause the leak. An orphan with readable backing data can persist without this warning.

The original reason acquisition exceeded 30 seconds, and why the backing layer later became inaccessible, have not been established. The -ENOENT observation does not establish that unlinking an open local file caused the I/O failure.

Source analysis

At the reviewed public revision:

UblkDaemonClient::call wraps the RPC in a timeout; the inner operation owns the connection.
acquire_shared inserts a new active shared entry or increments an existing shared reference before returning DeviceAcquired.
The connection handler calls send_message(...).await?. A failed send exits the handler without rolling back that acquisition.

The acquisition remains recorded as active, so the normal client-release path cannot return the abandoned acquisition to the pool. The runtime observation concerns an acquired shared device; the existing-reference leak is a source-level finding. Other resource-creating handlers have not been reproduced here.

Proposed regression coverage

This procedure is proposed and has not yet been executed:

Start an isolated daemon with pooling and a disposable read-only OverlayBD image.
Add a test barrier after the acquisition is registered, immediately before response delivery.
Send AcquireOverlaybd with shared access. At the barrier, close the client connection without consuming the result.
Resume the handler and verify response delivery fails.
Verify the abandoned active entry/reference is released. Account for legitimate idle-pool refill when comparing device counts.
Repeat while another client holds a valid reference to the same image. Only the abandoned reference should be released; the valid client's device must remain usable.

Coverage should also include timeout/cancellation and a response successfully written before the caller abandons the result. Failed-send rollback alone does not cover that last ownership handoff window; the fix should define how abandoned acquisitions are identified and released.

Expected behavior and impact

An acquisition whose result is abandoned is eventually released, preserving legitimate shared users.
Repeated timeouts or disconnects do not accumulate devices or references beyond valid acquisitions and the configured idle pool.
Acquisition ownership remains recoverable when response delivery or caller cancellation interrupts the handoff.

Unreleased devices can retain device slots, file descriptors, worker threads, and buffers, potentially preventing later sandbox creation or restore. Leaked references to an existing shared device can prevent its release after valid users finish. If an orphan's backing data becomes inaccessible, later reads can additionally produce kernel I/O warnings.

Originally created by @dushulin on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/392 ### Summary A shared OverlayBD acquisition can complete after its caller has timed out or disconnected. The daemon records the acquired device/reference before sending the response. If response delivery fails, the handler returns without releasing the acquisition. The caller never receives the result needed to own and release it. This leaves an active acquisition with no corresponding client owner. Repeated failures can accumulate devices or shared references. ### Environment and verification scope - Component: `ublk-daemon`, shared memory devices used for snapshot restore, with device pooling enabled. - Environment: Linux x86_64, kernel 6.8, `ublk_drv`, OverlayBD. - Runtime evidence comes from an older downstream deployment. The affected path was independently inspected at public revision [`0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45`](https://github.com/kvcache-ai/AgentENV/commit/0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45). - A standalone reproducer has not yet been run against that upstream revision. The first affected revision and last known good revision are unknown. ### Observed behavior The following sequence was observed repeatedly. Logs are normalized, with timestamps and identifiers removed: ```text client: acquire shared memory ublk device: daemon RPC timed out after 30s daemon: acquired shared overlaybd device (...) dev_id=<id> daemon: daemon connection handler failed write message length prefix Broken pipe (os error 32) ``` The device remained LIVE and held by the daemon. No subsequent release appeared in the retained lifecycle logs, and no Firecracker FD or mmap referencing it was observed. Normal idle placeholder devices were excluded from this classification. For an affected device whose backing layer later became inaccessible, a host inventory operation (`fdisk -l`) triggered header reads and exposed the following errors: ```text overlaybd request failed op=Read offset=0 len=4096 No such file or directory (os error 2) errno=-2 Buffer I/O error on dev ublkb<id>, logical block 0, async page read ``` The inventory read exposed the retained device; it did not cause the leak. An orphan with readable backing data can persist without this warning. The original reason acquisition exceeded 30 seconds, and why the backing layer later became inaccessible, have not been established. The `-ENOENT` observation does not establish that unlinking an open local file caused the I/O failure. ### Source analysis At the reviewed public revision: 1. [`UblkDaemonClient::call`](https://github.com/kvcache-ai/AgentENV/blob/0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45/storage/ublk-daemon/src/client.rs#L693) wraps the RPC in a timeout; the inner operation owns the connection. 2. [`acquire_shared`](https://github.com/kvcache-ai/AgentENV/blob/0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45/storage/ublk-daemon/src/server.rs#L1576) inserts a new active shared entry or increments an existing shared reference before returning `DeviceAcquired`. 3. The connection handler calls [`send_message(...).await?`](https://github.com/kvcache-ai/AgentENV/blob/0ce8eb9bd94d8656b0a40e5be7cdfadfd6e78b45/storage/ublk-daemon/src/server.rs#L735). A failed send exits the handler without rolling back that acquisition. The acquisition remains recorded as active, so the normal client-release path cannot return the abandoned acquisition to the pool. The runtime observation concerns an acquired shared device; the existing-reference leak is a source-level finding. Other resource-creating handlers have not been reproduced here. ### Proposed regression coverage This procedure is proposed and has not yet been executed: 1. Start an isolated daemon with pooling and a disposable read-only OverlayBD image. 2. Add a test barrier after the acquisition is registered, immediately before response delivery. 3. Send `AcquireOverlaybd` with shared access. At the barrier, close the client connection without consuming the result. 4. Resume the handler and verify response delivery fails. 5. Verify the abandoned active entry/reference is released. Account for legitimate idle-pool refill when comparing device counts. 6. Repeat while another client holds a valid reference to the same image. Only the abandoned reference should be released; the valid client's device must remain usable. Coverage should also include timeout/cancellation and a response successfully written before the caller abandons the result. Failed-send rollback alone does not cover that last ownership handoff window; the fix should define how abandoned acquisitions are identified and released. ### Expected behavior and impact - An acquisition whose result is abandoned is eventually released, preserving legitimate shared users. - Repeated timeouts or disconnects do not accumulate devices or references beyond valid acquisitions and the configured idle pool. - Acquisition ownership remains recoverable when response delivery or caller cancellation interrupts the handoff. Unreleased devices can retain device slots, file descriptors, worker threads, and buffers, potentially preventing later sandbox creation or restore. Leaked references to an existing shared device can prevent its release after valid users finish. If an orphan's backing data becomes inaccessible, later reads can additionally produce kernel I/O warnings.

bsakaguchi closed this issue

2026-10-29 14:19:35 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:42 +00:00

[PR #100] [CLOSED] fix(orchestrator): restore pause state after allocation failure #178