[GH-ISSUE #203] ublk: proactive refill causes device churn without UBLK_F_UPDATE_SIZE #65

New issue

Closed

opened 2026-10-29 14:19:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:04 +00:00

Owner

Originally created by @ztting01 on GitHub (Aug 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/203

Affected component

ublk / ublk-daemon

AgentENV version

Version or commit: 1a58ff95e17b45c4178cd87ceaca61386f7a8c2c
Local modifications: none for the baseline run

Host environment

OS: openEuler 24.03 LTS-SP3
Kernel: 6.6.0-132.0.0.111.oe2403sp3.aarch64
Architecture: aarch64
Host: bare metal, Kunpeng 920, 502 GiB memory
Storage/filesystem: ext4
ublk features: 0x1fe (UBLK_F_UPDATE_SIZE is unavailable)

Operation and configuration

We repeatedly paused and resumed sandboxes using 64 GiB rootfs devices and 2 GiB memory-snapshot devices with the block-device pool enabled:

[pool]
low_watermark = 2
high_watermark = 64

[pool.block]
enabled = true
startup_prewarm = true

Steps to reproduce

Run AgentENV on a kernel without UBLK_F_UPDATE_SIZE.
Start 24 sandboxes from the same snapshot template.
Run 11 command/pause/resume interactions in each sandbox.
Inspect ublk-daemon lifecycle logs and the remaining devices with lsblk.

Expected behavior

A single device size should not monopolize the global idle pool when idle ublk devices cannot be resized. Different image sizes should be reusable without continuous device creation and destruction.

Actual behavior

The refill target grows from the low watermark toward the high watermark and refill uses one observed image size. In this test, 64 GiB devices filled all 64 idle slots.

Because the kernel cannot resize those devices, later 2 GiB memory-snapshot requests cannot reuse them. A 2 GiB device is created on resume, rejected when it is returned to the full pool on pause, and then stopped. The cycle repeats on the next resume.

Logs and diagnostics

The same host, snapshot template and workload were used for both runs. As a diagnostic, we disabled proactive block-device refill when UBLK_F_UPDATE_SIZE was unavailable and measured the naturally warmed stable state.

Metric	Original behavior	Proactive refill disabled
Successful sandboxes	24/24	24/24
New/reused shared acquisitions	262/6	0/265
Devices returned to idle pool	6	553
Devices stopped	553	0
Idle device distribution	64 x 64 GiB	24 x 2 GiB + 24 x 64 GiB
Total elapsed time	77.78 s	71.93 s
Pause p95	1.68 s	1.27 s
Resume p95	3.19 s	0.73 s

No unexpected errors or warnings occurred in the diagnostic run.

Reproduction frequency

Always under this workload.

Regression information

Last known good: unknown
First known bad: unknown

Additional context

UBLK_U_CMD_UPDATE_SIZE / UBLK_F_UPDATE_SIZE was added upstream in Linux 6.16. AgentENV currently documents Linux 6.8+ as its host requirement, so kernels from 6.8 through 6.15 do not have this capability unless it has been backported.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @ztting01 on GitHub (Aug 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/203 ## Affected component ublk / ublk-daemon ## AgentENV version - Version or commit: `1a58ff95e17b45c4178cd87ceaca61386f7a8c2c` - Local modifications: none for the baseline run ## Host environment - OS: openEuler 24.03 LTS-SP3 - Kernel: `6.6.0-132.0.0.111.oe2403sp3.aarch64` - Architecture: aarch64 - Host: bare metal, Kunpeng 920, 502 GiB memory - Storage/filesystem: ext4 - ublk features: `0x1fe` (`UBLK_F_UPDATE_SIZE` is unavailable) ## Operation and configuration We repeatedly paused and resumed sandboxes using 64 GiB rootfs devices and 2 GiB memory-snapshot devices with the block-device pool enabled: ```toml [pool] low_watermark = 2 high_watermark = 64 [pool.block] enabled = true startup_prewarm = true ``` ## Steps to reproduce 1. Run AgentENV on a kernel without `UBLK_F_UPDATE_SIZE`. 2. Start 24 sandboxes from the same snapshot template. 3. Run 11 command/pause/resume interactions in each sandbox. 4. Inspect ublk-daemon lifecycle logs and the remaining devices with `lsblk`. ## Expected behavior A single device size should not monopolize the global idle pool when idle ublk devices cannot be resized. Different image sizes should be reusable without continuous device creation and destruction. ## Actual behavior The refill target grows from the low watermark toward the high watermark and refill uses one observed image size. In this test, 64 GiB devices filled all 64 idle slots. Because the kernel cannot resize those devices, later 2 GiB memory-snapshot requests cannot reuse them. A 2 GiB device is created on resume, rejected when it is returned to the full pool on pause, and then stopped. The cycle repeats on the next resume. ## Logs and diagnostics The same host, snapshot template and workload were used for both runs. As a diagnostic, we disabled proactive block-device refill when `UBLK_F_UPDATE_SIZE` was unavailable and measured the naturally warmed stable state. | Metric | Original behavior | Proactive refill disabled | |---|---:|---:| | Successful sandboxes | 24/24 | 24/24 | | New/reused shared acquisitions | 262/6 | 0/265 | | Devices returned to idle pool | 6 | 553 | | Devices stopped | 553 | 0 | | Idle device distribution | 64 x 64 GiB | 24 x 2 GiB + 24 x 64 GiB | | Total elapsed time | 77.78 s | 71.93 s | | Pause p95 | 1.68 s | 1.27 s | | Resume p95 | 3.19 s | 0.73 s | No unexpected errors or warnings occurred in the diagnostic run. ## Reproduction frequency Always under this workload. ## Regression information - Last known good: unknown - First known bad: unknown ## Additional context [`UBLK_U_CMD_UPDATE_SIZE` / `UBLK_F_UPDATE_SIZE` was added upstream in Linux 6.16](https://kernelnewbies.org/Linux_6.16). AgentENV currently documents [Linux 6.8+](https://github.com/kvcache-ai/AgentENV#-prerequisites) as its host requirement, so kernels from 6.8 through 6.15 do not have this capability unless it has been backported. ## Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi closed this issue

2026-10-29 14:19:04 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:16 +00:00

[PR #65] [MERGED] perf: stabilize concurrent snapshot resume #150