[PR #335] [MERGED] perf(ublk-daemon): stop devices concurrently on shutdown

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/335 **Author:** [@odysseusy](https://github.com/odysseusy) **Created:** 10/1/2026 **Status:** ✅ Merged **Merged:** 10/1/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `perf/ublk-daemon-parallel-shutdown` --- ### 📝 Commits (1) - [`f3f5dcf`](https://github.com/kvcache-ai/AgentENV/commit/f3f5dcfebf2fbfa6c2787436a1d1c586ae5695f2) perf(ublk-daemon): stop devices concurrently on shutdown ### 📊 Changes **1 file changed** (+43 additions, -14 deletions) <details> <summary>View changed files</summary> 📝 `storage/ublk-daemon/src/server.rs` (+43 -14) </details> ### 📄 Description ## What Tear down devices concurrently in `UblkDaemonServer::stop_all_devices()`, at most 16 at a time. The change uses a `JoinSet` with a `Semaphore`. Each task keeps the existing per-device sequence: quiesce, drop the char-device fd, then `delete_dev`. ## Why Shutdown stops raw, exclusive, shared and idle pooled devices one after another. Each STOP_DEV/DEL_DEV pair waits for block-layer RCU/SRCU grace periods. Those waits are global and can overlap, but serial teardown makes shutdown time grow linearly: about 45 s for 2000 devices. ## Related issue None. This is a small, self-contained change; happy to open an issue if preferred. ## Scope and non-goals Only the shutdown path changes. The runtime `Delete` path is untouched, because requests on separate connections are already handled concurrently. ## Design and behavior changes - The kernel punts ublk control commands that may sleep to io-wq, so the teardowns make progress in parallel on the shared control ring. - Bounding concurrency at 16 avoids spawning an io-wq worker per device on hosts with thousands of devices. - Failures are still logged per device, and a panicked task is logged instead of being ignored silently. ## Compatibility and operations - Public API or generated protocol: N/A - Configuration or defaults: N/A - Snapshot manifest, artifact layout, or storage format: N/A - Upgrade and rollback: N/A - Host requirements, permissions, ports, or dependencies: N/A ## Validation - [ ] `make fmt` - [ ] `make clippy` - [ ] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [ ] Documentation updated - [x] Benchmarks or performance comparison completed Commands and results: ```text cargo fmt -p uvm-ublk-daemon -- --check ok cargo clippy -p uvm-ublk-daemon --all-targets -- -D warnings ok cargo test -p uvm-ublk-daemon 86 passed Create N raw overlaybd devices over the daemon socket (sparse 8 GiB upper), send SIGTERM, time until exit. Median of 3, VM with 16 vCPUs. No ublkc* nodes left behind in any run. kernel devices before after speedup 6.6 500 10.80 s 0.68 s 15.9x 6.6 2000 44.77 s 2.75 s 16.3x 7.3-rc5 500 9.74 s 2.37 s 4.1x 7.3-rc5 2000 42.69 s 8.97 s 4.8x ``` Skipped checks and reasons: `make fmt`/`make clippy`/`make test-unit` cover the whole workspace including Go services; I ran the cargo equivalents scoped to the changed crate above. Integration tests need a full Firecracker host. The shutdown path was exercised directly by the A/B runs above. ## Risks and reviewer notes The gain is smaller on 7.x because the kernel releases the last device reference under the global `ublk_ctl_mutex` and calls `srcu_barrier()` from there. A kernel fix that moves the put out of the lock is under review upstream: https://lore.kernel.org/r/20261001-bug-ublk-del-dev-put-unlocked-v1-1-18e9bb9be727@gmail.com ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi commented

2026-10-29 14:22:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/335
Author: @odysseusy
Created: 10/1/2026
Status: ✅ Merged
Merged: 10/1/2026
Merged by: @yingdi-shan

Base: main ← Head: perf/ublk-daemon-parallel-shutdown

📝 Commits (1)

f3f5dcf perf(ublk-daemon): stop devices concurrently on shutdown

📊 Changes

1 file changed (+43 additions, -14 deletions)

View changed files

📝 storage/ublk-daemon/src/server.rs (+43 -14)

📄 Description

What

Tear down devices concurrently in UblkDaemonServer::stop_all_devices(), at most 16 at a time. The change uses a JoinSet with a Semaphore. Each task keeps the existing per-device sequence: quiesce, drop the char-device fd, then delete_dev.

Why

Shutdown stops raw, exclusive, shared and idle pooled devices one after another. Each STOP_DEV/DEL_DEV pair waits for block-layer RCU/SRCU grace periods. Those waits are global and can overlap, but serial teardown makes shutdown time grow linearly: about 45 s for 2000 devices.

None. This is a small, self-contained change; happy to open an issue if preferred.

Scope and non-goals

Only the shutdown path changes. The runtime Delete path is untouched, because requests on separate connections are already handled concurrently.

Design and behavior changes

The kernel punts ublk control commands that may sleep to io-wq, so the teardowns make progress in parallel on the shared control ring.
Bounding concurrency at 16 avoids spawning an io-wq worker per device on hosts with thousands of devices.
Failures are still logged per device, and a panicked task is logged instead of being ignored silently.

Compatibility and operations

Public API or generated protocol: N/A
Configuration or defaults: N/A
Snapshot manifest, artifact layout, or storage format: N/A
Upgrade and rollback: N/A
Host requirements, permissions, ports, or dependencies: N/A

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

cargo fmt -p uvm-ublk-daemon -- --check                      ok
cargo clippy -p uvm-ublk-daemon --all-targets -- -D warnings ok
cargo test -p uvm-ublk-daemon                               86 passed

Create N raw overlaybd devices over the daemon socket (sparse 8 GiB upper),
send SIGTERM, time until exit. Median of 3, VM with 16 vCPUs.
No ublkc* nodes left behind in any run.

kernel    devices   before    after    speedup
6.6       500       10.80 s   0.68 s   15.9x
6.6       2000      44.77 s   2.75 s   16.3x
7.3-rc5   500        9.74 s   2.37 s    4.1x
7.3-rc5   2000      42.69 s   8.97 s    4.8x

Skipped checks and reasons: make fmt/make clippy/make test-unit cover the whole workspace including Go services; I ran the cargo equivalents scoped to the changed crate above. Integration tests need a full Firecracker host. The shutdown path was exercised directly by the A/B runs above.

Risks and reviewer notes

The gain is smaller on 7.x because the kernel releases the last device reference under the global ublk_ctl_mutex and calls srcu_barrier() from there. A kernel fix that moves the put out of the lock is under review upstream: https://lore.kernel.org/r/20261001-bug-ublk-del-dev-put-unlocked-v1-1-18e9bb9be727@gmail.com

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

bsakaguchi

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-10-29 14:22:45 +00:00

[PR #357] [CLOSED] chore(deps): bump hickory-resolver from 0.26.1 to 0.26.2 #376

Rows
Columns

[PR #335] [MERGED] perf(ublk-daemon): stop devices concurrently on shutdown #357