[GH-ISSUE #333] perf(ublk): default ublk block devices to the none I/O scheduler #85

New issue

Closed

opened 2026-10-29 14:19:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:23 +00:00

Owner

Originally created by @odysseusy on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/333

Affected component

ublk / ublk-daemon (host setup)

Problem statement

Every ublk device AgentENV creates has one hardware queue, so the kernel attaches mq-deadline to /dev/ublkbN by default. ublk forwards each request to the userspace overlaybd target, which does its own ordering, so the elevator adds only per-request locking and dispatch overhead. On Linux 7.2 and later the cost is much larger: commit 9cbbac29d752 ("block: Remove redundant plug in __submit_bio()") makes every direct I/O to a device with an elevator dispatch asynchronously through kblockd, which adds a context switch per request.

Measured on uvm-ublk from v0.2.3 (6cccaa7):

Target: overlaybd with a sparse 8 GiB upper.
Device parameters: the daemon defaults of 1 queue, depth 16 and 256 KiB maximum I/O.
fio: --direct=1 --ioengine=io_uring --bs=4k.
Pinning: each device server and each fio job ran on its own CPU, on a VM with 16 vCPUs.
Each figure is the median of 3 rounds.

kernel	workload	mq-deadline (default)	none
6.6	randread qd16, 1 dev	169K IOPS	304K IOPS (+80%)
6.6	randread qd16, 4 devs	690K IOPS	1247K IOPS (+81%)
6.6	randwrite qd16, 1 dev	108K IOPS	175K IOPS (+63%)
7.3-rc5	randread qd16, 1 dev	107K IOPS	307K IOPS (+186%)
7.3-rc5	randread qd16, 4 devs	434K IOPS	1232K IOPS (+184%)
7.3-rc5	randwrite qd16, 1 dev	105K IOPS	181K IOPS (+73%)

Use case

Any host running sandboxes on ublk-backed rootfs or extra drives. Small random I/O from the guest is the common case for agent workloads.

Current behavior and workarounds

Devices come up with mq-deadline. An operator can add their own udev rule or write queue/scheduler by hand, but nothing in AgentENV does it. The daemon runs as an unprivileged user and can't write sysfs.

Desired behavior

New /dev/ublkb* devices use the none scheduler before the daemon hands them out.

Proposed approach

server --setup-host already installs /etc/udev/rules.d/99-agentenv-ublk.rules to grant device access. Add one line to it, and to scripts/tests/setup-ublk-access.sh:

ACTION=="add", SUBSYSTEM=="block", KERNEL=="ublkb*", ATTR{queue/scheduler}="none"

udev applies the attribute in the same event that sets the node's group. The daemon runs without CAP_DAC_OVERRIDE, so wait_for_ublk_dev() can't open the node before that event, and the scheduler is already none by then. I checked this on 10 devices with the production privilege model (non-root user with CAP_SYS_ADMIN and CAP_NET_ADMIN).

A patch is ready if this approach is acceptable.

Compatibility and operational impact

Hosts provisioned earlier keep the old behavior until setup runs again.
Operators who want another scheduler can override it with a later-ordered rule.
Permissions are unchanged.

Alternatives considered

Writing sysfs from the daemon: this needs privileges the daemon doesn't have.
A kernel-side default: a ublk patch that sets BLK_MQ_F_NO_SCHED_BY_DEFAULT, like loop and null_blk, is under review upstream. It won't reach the 6.x kernels that hosts run today.

Non-goals

Tuning other queue attributes, and changing the scheduler of non-ublk devices. This is independent of the userspace weighted I/O scheduling proposed in #47, which runs inside the daemon.

Originally created by @odysseusy on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/333 ### Affected component ublk / ublk-daemon (host setup) ### Problem statement Every ublk device AgentENV creates has one hardware queue, so the kernel attaches `mq-deadline` to `/dev/ublkbN` by default. ublk forwards each request to the userspace overlaybd target, which does its own ordering, so the elevator adds only per-request locking and dispatch overhead. On Linux 7.2 and later the cost is much larger: commit 9cbbac29d752 ("block: Remove redundant plug in __submit_bio()") makes every direct I/O to a device with an elevator dispatch asynchronously through kblockd, which adds a context switch per request. Measured on `uvm-ublk` from v0.2.3 (`6cccaa7`): - Target: overlaybd with a sparse 8 GiB upper. - Device parameters: the daemon defaults of 1 queue, depth 16 and 256 KiB maximum I/O. - fio: `--direct=1 --ioengine=io_uring --bs=4k`. - Pinning: each device server and each fio job ran on its own CPU, on a VM with 16 vCPUs. - Each figure is the median of 3 rounds. | kernel | workload | mq-deadline (default) | none | |---|---|---|---| | 6.6 | randread qd16, 1 dev | 169K IOPS | 304K IOPS (+80%) | | 6.6 | randread qd16, 4 devs | 690K IOPS | 1247K IOPS (+81%) | | 6.6 | randwrite qd16, 1 dev | 108K IOPS | 175K IOPS (+63%) | | 7.3-rc5 | randread qd16, 1 dev | 107K IOPS | 307K IOPS (+186%) | | 7.3-rc5 | randread qd16, 4 devs | 434K IOPS | 1232K IOPS (+184%) | | 7.3-rc5 | randwrite qd16, 1 dev | 105K IOPS | 181K IOPS (+73%) | ### Use case Any host running sandboxes on ublk-backed rootfs or extra drives. Small random I/O from the guest is the common case for agent workloads. ### Current behavior and workarounds Devices come up with `mq-deadline`. An operator can add their own udev rule or write `queue/scheduler` by hand, but nothing in AgentENV does it. The daemon runs as an unprivileged user and can't write sysfs. ### Desired behavior New `/dev/ublkb*` devices use the `none` scheduler before the daemon hands them out. ### Proposed approach `server --setup-host` already installs `/etc/udev/rules.d/99-agentenv-ublk.rules` to grant device access. Add one line to it, and to `scripts/tests/setup-ublk-access.sh`: ``` ACTION=="add", SUBSYSTEM=="block", KERNEL=="ublkb*", ATTR{queue/scheduler}="none" ``` udev applies the attribute in the same event that sets the node's group. The daemon runs without CAP_DAC_OVERRIDE, so `wait_for_ublk_dev()` can't open the node before that event, and the scheduler is already `none` by then. I checked this on 10 devices with the production privilege model (non-root user with CAP_SYS_ADMIN and CAP_NET_ADMIN). A patch is ready if this approach is acceptable. ### Compatibility and operational impact - Hosts provisioned earlier keep the old behavior until setup runs again. - Operators who want another scheduler can override it with a later-ordered rule. - Permissions are unchanged. ### Alternatives considered - Writing sysfs from the daemon: this needs privileges the daemon doesn't have. - A kernel-side default: a ublk patch that sets `BLK_MQ_F_NO_SCHED_BY_DEFAULT`, like loop and null_blk, is under review upstream. It won't reach the 6.x kernels that hosts run today. ### Non-goals Tuning other queue attributes, and changing the scheduler of non-ublk devices. This is independent of the userspace weighted I/O scheduling proposed in #47, which runs inside the daemon.

bsakaguchi closed this issue

2026-10-29 14:19:24 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:36 +00:00

[PR #85] [MERGED] feat(overlaybd): introduce index() and flatten_with_args() from LSMTFile #168