[GH-ISSUE #47] feat: support weight-based I/O scheduling across sandboxes at ublk layer #23

New issue

Open

opened 2026-10-29 14:18:29 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented 2026-10-29 14:18:29 +00:00

Owner

Copy link

Originally created by @rogeroger-yu on GitHub (Jul 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/47

Summary

Add a node-wide weighted fair I/O scheduler at the ublk device layer. Each sandbox's block device is assigned a weight, and the total node bandwidth budget is distributed proportionally among active devices. The scheduler is work-conserving: idle sandboxes' bandwidth is automatically redistributed to active ones.

Motivation

While absolute rate limiting (Firecracker TokenBucket) prevents starvation, it wastes bandwidth when sandboxes are idle. A weight-based scheduler enables:

1. Fair sharing: Equal-weight sandboxes automatically get equal bandwidth
2. Work conservation: No bandwidth wasted when sandboxes are idle
3. Priority support: Higher-weight sandboxes get proportionally more I/O
4. Global control: Single knob (total_bandwidth) to tune node-level I/O pressure

Proposed Solution

Implement IoWeightScheduler as a global singleton in the ublk daemon process:

• Token bucket per device: Each device gets tokens proportional to device_weight / sum(active_weights)
• Refill cycle: Every refill_interval_ms (default 100ms), tokens are replenished
• Async acquire: handle_io_request calls acquire(bytes) before each I/O operation
• Automatic registration: Devices register on pool acquire, unregister on drop

Configuration

[machine.disk_weight_scheduler]
enabled = true
total_bandwidth_bytes_per_sec = 209715200  # 200 MB/s total budget
default_weight = 100
refill_interval_ms = 100

Test Results

Scenario	Baseline	With Scheduler	Notes
Single sandbox	~1.8 GB/s	~111 MB/s	Capped to budget
2 concurrent (equal weight)	~1.8 GB/s each	~76 MB/s each	Fairness ratio: 1.00
Total concurrent throughput	~3.6 GB/s	~152 MB/s	Within budget

Originally created by @rogeroger-yu on GitHub (Jul 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/47 ## Summary Add a node-wide weighted fair I/O scheduler at the ublk device layer. Each sandbox's block device is assigned a weight, and the total node bandwidth budget is distributed proportionally among active devices. The scheduler is work-conserving: idle sandboxes' bandwidth is automatically redistributed to active ones. ## Motivation While absolute rate limiting (Firecracker TokenBucket) prevents starvation, it wastes bandwidth when sandboxes are idle. A weight-based scheduler enables: 1. **Fair sharing**: Equal-weight sandboxes automatically get equal bandwidth 2. **Work conservation**: No bandwidth wasted when sandboxes are idle 3. **Priority support**: Higher-weight sandboxes get proportionally more I/O 4. **Global control**: Single knob (total_bandwidth) to tune node-level I/O pressure ## Proposed Solution Implement `IoWeightScheduler` as a global singleton in the ublk daemon process: - **Token bucket per device**: Each device gets tokens proportional to `device_weight / sum(active_weights)` - **Refill cycle**: Every `refill_interval_ms` (default 100ms), tokens are replenished - **Async acquire**: `handle_io_request` calls `acquire(bytes)` before each I/O operation - **Automatic registration**: Devices register on pool acquire, unregister on drop ### Configuration ```toml [machine.disk_weight_scheduler] enabled = true total_bandwidth_bytes_per_sec = 209715200 # 200 MB/s total budget default_weight = 100 refill_interval_ms = 100 ``` ## Test Results | Scenario | Baseline | With Scheduler | Notes | |----------|----------|----------------|-------| | Single sandbox | ~1.8 GB/s | ~111 MB/s | Capped to budget | | 2 concurrent (equal weight) | ~1.8 GB/s each | ~76 MB/s each | Fairness ratio: 1.00 | | Total concurrent throughput | ~3.6 GB/s | ~152 MB/s | Within budget |

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Jul 28, 2026):

Thanks for the proposal and benchmark data. My main concern is the placement of the scheduler.

At the ublk boundary, we only see the logical byte count requested by the guest. That does not necessarily represent the node resource actually consumed. Different ublk devices can resolve to the same immutable OverlayBD lower layer and shared cache entry. The same-sized request may be:

• served entirely from page cache
• read from a local OverlayBD file;
• fetched from a remote registry or P2P source;
• expanded into cache refill, prefetch, decompression, or additional metadata I/O.

For example, one device may trigger a remote fetch and warm a shared layer, while another device later reads the same data from cache. Charging both devices the same number of tokens at the ublk layer does not reflect their disk or network consumption. Consequently, per-device ublk limiting controls guest-visible block throughput, but it can drift significantly from weighted sharing of the node’s actual storage and network resources.

I suggest first adding per-device metrics for logical ublk bytes, local backend bytes, and remote backend bytes. That will show how much these values diverge under realistic shared-image workloads before we choose the scheduling boundary.

<!-- gh-comment-id:5105990118 --> @yingdi-shan commented on GitHub (Jul 28, 2026): Thanks for the proposal and benchmark data. My main concern is the placement of the scheduler. At the ublk boundary, we only see the logical byte count requested by the guest. That does not necessarily represent the node resource actually consumed. Different ublk devices can resolve to the same immutable OverlayBD lower layer and shared cache entry. The same-sized request may be: - served entirely from page cache - read from a local OverlayBD file; - fetched from a remote registry or P2P source; - expanded into cache refill, prefetch, decompression, or additional metadata I/O. For example, one device may trigger a remote fetch and warm a shared layer, while another device later reads the same data from cache. Charging both devices the same number of tokens at the ublk layer does not reflect their disk or network consumption. Consequently, per-device ublk limiting controls guest-visible block throughput, but it can drift significantly from weighted sharing of the node’s actual storage and network resources. I suggest first adding per-device metrics for logical ublk bytes, local backend bytes, and remote backend bytes. That will show how much these values diverge under realistic shared-image workloads before we choose the scheduling boundary.

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@rogeroger-yu commented on GitHub (Jul 29, 2026):

Thanks for the detailed feedback — this is a valid concern.

You're right that at the ublk boundary, logical bytes don't always correlate with actual backend resource consumption, especially under shared-image workloads with mixed cache/remote hits.

I'll take the metrics-first approach:

1. Add per-device I/O source metrics in the overlaybd read path, breaking down each request into:

   • logical_bytes: guest-requested size at ublk layer
   • local_bytes: served from local file (committed layers + upper)
   • cache_bytes: served from block cache / page cache
   • remote_bytes: fetched from registry or P2P

2. Expose via Prometheus on the existing daemon metrics endpoint (0.0.0.0:9103), labeled by dev_id.

3. Run a shared-image benchmark — multiple sandboxes from the same base image doing concurrent reads (cold start + warm steady state) — and report the divergence between logical and actual I/O.

Once we have real data showing how much these values drift, we can make an informed decision on where to place the scheduling boundary (ublk entry vs. backend I/O vs. a hybrid approach).

I'll update PR #49 to start with the metrics instrumentation instead of the scheduler, and share the benchmark results here.

<!-- gh-comment-id:5112397451 --> @rogeroger-yu commented on GitHub (Jul 29, 2026): Thanks for the detailed feedback — this is a valid concern. You're right that at the ublk boundary, logical bytes don't always correlate with actual backend resource consumption, especially under shared-image workloads with mixed cache/remote hits. I'll take the metrics-first approach: 1. **Add per-device I/O source metrics** in the overlaybd read path, breaking down each request into: - `logical_bytes`: guest-requested size at ublk layer - `local_bytes`: served from local file (committed layers + upper) - `cache_bytes`: served from block cache / page cache - `remote_bytes`: fetched from registry or P2P 2. **Expose via Prometheus** on the existing daemon metrics endpoint (`0.0.0.0:9103`), labeled by `dev_id`. 3. **Run a shared-image benchmark** — multiple sandboxes from the same base image doing concurrent reads (cold start + warm steady state) — and report the divergence between logical and actual I/O. Once we have real data showing how much these values drift, we can make an informed decision on where to place the scheduling boundary (ublk entry vs. backend I/O vs. a hybrid approach). I'll update PR #49 to start with the metrics instrumentation instead of the scheduler, and share the benchmark results here.

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Jul 29, 2026):

Thank you so much!

<!-- gh-comment-id:5112700336 --> @yingdi-shan commented on GitHub (Jul 29, 2026): Thank you so much!

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@rogeroger-yu commented on GitHub (Jul 30, 2026):

Follow-up: per-device logical / local / remote byte metrics + measured divergence

Per review feedback, I added the three per-device metrics @yingdi-shan asked for and
measured how much they diverge under a realistic shared-image workload before we lock
in the scheduling boundary.

New metrics

• agentenv_ublk_device_read_bytes_total{dev_id} — logical bytes read by the guest (already existed)
• agentenv_overlaybd_device_local_read_bytes_total{dev_id} — bytes served by the LocalFile backend
• agentenv_overlaybd_device_remote_read_bytes_total{dev_id} — bytes served by the registryfs backend

dev_id is threaded through IoCtx across ublk → ImageFile → LSMT → leaf backends
(LocalFile / registryfs), so reads are attributed per device without touching the
backends' existing global metrics.

Method

Same overlaybd-native image (8 compressed zfile layers, rootfs ~700 MB) and the same
read workload (find /usr -type f | xargs cat, ~127 MiB across ~5,200 reads). One
device measured per backend state, each after drop_caches:

• Device A: everything served from remote registryfs (cold, no local cache)
• Device B: everything served from local commits (LocalFile)

Results

Scenario	Logical (guest)	Actual backend read	Divergence
Remote · cold boot	7.5 MiB	37.0 MiB (remote)	4.94x
Remote · after bulk read	127.1 MiB	197.2 MiB (remote)	1.55x
Local · cold boot	11.6 MiB	11.6 MiB (local)	1.00x
Local · after bulk read	127.1 MiB	127.1 MiB (local)	1.00x

Conclusion: divergence is significant — the current ublk-layer scheduling approach needs adjustment

1. The divergence is real and strongly depends on backend/cache state. On a local cache
   hit, backend ≈ logical (1.0x); on a remote/cold read, backend I/O is 1.5x (bulk
   sequential) to ~5x (cold boot, small random metadata reads) the logical bytes, due
   to block-granularity fetches + decompression + over-fetch.
2. Consequently, weighting on logical bytes at the ublk layer under-accounts the true
   node and network cost by up to ~5x for cold/remote tenants, while remaining accurate
   for local-cache-hit tenants. Fairness at the logical layer is not the same as fairness
   over the physical resource (registry bandwidth / network egress / decompression CPU).

Proposed adjustment directions (for discussion)

• Option 1: move/augment scheduling accounting to backend bytes (weight local and
  remote separately, with a higher weight on remote to reflect network cost), so throttling
  tracks real node cost.
• Option 2: keep ublk logical-layer scheduling for guest-visible throughput fairness,
  and add a separate remote-bandwidth budget in the overlaybd backend, treating cold fetch
  as a one-time warmup cost that caching amortizes.

Which direction do we prefer? I'll adjust the implementation once we align.

<!-- gh-comment-id:5131729281 --> @rogeroger-yu commented on GitHub (Jul 30, 2026): ## Follow-up: per-device logical / local / remote byte metrics + measured divergence Per review feedback, I added the three per-device metrics @yingdi-shan asked for and measured how much they diverge under a realistic shared-image workload before we lock in the scheduling boundary. ### New metrics - `agentenv_ublk_device_read_bytes_total{dev_id}` — logical bytes read by the guest (already existed) - `agentenv_overlaybd_device_local_read_bytes_total{dev_id}` — bytes served by the LocalFile backend - `agentenv_overlaybd_device_remote_read_bytes_total{dev_id}` — bytes served by the registryfs backend `dev_id` is threaded through `IoCtx` across ublk → ImageFile → LSMT → leaf backends (LocalFile / registryfs), so reads are attributed per device without touching the backends' existing global metrics. ### Method Same overlaybd-native image (8 compressed zfile layers, rootfs ~700 MB) and the same read workload (`find /usr -type f | xargs cat`, ~127 MiB across ~5,200 reads). One device measured per backend state, each after `drop_caches`: - Device A: everything served from remote registryfs (cold, no local cache) - Device B: everything served from local commits (LocalFile) ### Results | Scenario | Logical (guest) | Actual backend read | Divergence | |---|---|---|---| | Remote · cold boot | 7.5 MiB | 37.0 MiB (remote) | **4.94x** | | Remote · after bulk read | 127.1 MiB | 197.2 MiB (remote) | **1.55x** | | Local · cold boot | 11.6 MiB | 11.6 MiB (local) | **1.00x** | | Local · after bulk read | 127.1 MiB | 127.1 MiB (local) | **1.00x** | ### Conclusion: divergence is significant — the current ublk-layer scheduling approach needs adjustment 1. The divergence is real and strongly depends on backend/cache state. On a local cache hit, backend ≈ logical (1.0x); on a remote/cold read, backend I/O is **1.5x (bulk sequential) to ~5x (cold boot, small random metadata reads)** the logical bytes, due to block-granularity fetches + decompression + over-fetch. 2. Consequently, weighting on logical bytes at the ublk layer **under-accounts the true node and network cost by up to ~5x for cold/remote tenants**, while remaining accurate for local-cache-hit tenants. Fairness at the logical layer is not the same as fairness over the physical resource (registry bandwidth / network egress / decompression CPU). ### Proposed adjustment directions (for discussion) - **Option 1:** move/augment scheduling accounting to **backend bytes** (weight local and remote separately, with a higher weight on remote to reflect network cost), so throttling tracks real node cost. - **Option 2:** keep ublk logical-layer scheduling for guest-visible throughput fairness, and add a separate remote-bandwidth budget in the overlaybd backend, treating cold fetch as a one-time warmup cost that caching amortizes. Which direction do we prefer? I'll adjust the implementation once we align.

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@drmfsltdoh commented on GitHub (Jul 30, 2026):

It is okay

<!-- gh-comment-id:5132196691 --> @drmfsltdoh commented on GitHub (Jul 30, 2026): It is okay

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Jul 30, 2026):

Before moving forward, we should also account for host page cache. Since multiple sandboxes can share the same block layers, one sandbox may end up populating the cache on behalf of others. The LocalFile metric counts backend requests, but not all of those requests translate into physical disk I/O.

Kernel readahead adds another layer of complexity on top of page caching. One potential approach is to run each device's or tenant's local-read worker in a separate cgroup, then use cgroup v2's io.stat for physical-read accounting and io.max/io.weight for control — under this model, page-cache hits would naturally incur no disk charge. That said, this approach would likely require a substantial refactor of the current I/O model, so I'm also considering whether a better solution exists.

<!-- gh-comment-id:5132454595 --> @yingdi-shan commented on GitHub (Jul 30, 2026): Before moving forward, we should also account for host page cache. Since multiple sandboxes can share the same block layers, one sandbox may end up populating the cache on behalf of others. The LocalFile metric counts backend requests, but not all of those requests translate into physical disk I/O. Kernel readahead adds another layer of complexity on top of page caching. One potential approach is to run each device's or tenant's local-read worker in a separate cgroup, then use cgroup v2's io.stat for physical-read accounting and io.max/io.weight for control — under this model, page-cache hits would naturally incur no disk charge. That said, this approach would likely require a substantial refactor of the current I/O model, so I'm also considering whether a better solution exists.

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@rogeroger-yu commented on GitHub (Aug 3, 2026):

Thanks for pushing on this — I ran the boundary validation end-to-end and want to share the raw findings across the approaches I tried.

Test setup: local overlaybd-native image (8 compressed zfile layers), buffered ioEngine (page cache on), guest dd on the extra drive, host cache dropped between cold runs. Ground truth for "physical disk bytes" is the kernel's own /proc/self/io read_bytes/write_bytes (the same counter cgroup io.stat draws from).

Approaches evaluated:

#	Approach	Result	Defect
A	Per-device byte metrics at LocalFile (*_local_read_bytes_total{dev_id})	Works, attributes correctly per dev_id	Counts backend request bytes, not physical disk I/O — blind to page cache + readahead. This is the gap flagged in the review.
B	Per-device physical-read time histogram (*_local_read_seconds{dev_id})	Not usable as a cache/disk signal	Size-dominated, not cache-vs-disk. Avg physical read ≈170 KiB (compressed extents), so even fully-warm reads sit in 0.1–1ms (memcpy cost), not µs. Warm dd 36.1 MB/s ≈ cold 37.5 MB/s (decompression-bound). Time can't discriminate cache from disk here.
C	RWF_NOWAIT in-process cache-residency byte meter (probe cache; miss → real read + charge, hit → free)	Partially works, then breaks down	(1) Clean binary case holds (cold 100% miss, warm 100% hit). (2) Steady-state it undercounts physical reads by 36–66% vs kernel read_bytes (readahead fills neighbors "for free" to the probe). (3) Read-only: pwritev2(RWF_NOWAIT) returns ENOTSUP on ext4 buffered writes — no write-path analog.
D	cgroup v2 io.stat / io.max / io.weight	Correct signal	Faithful physical read + write bytes/iops per group, and can throttle both. Cost: substantial refactor (per-device cgroup + worker attribution).

Measured RWF_NOWAIT charged/actual (kernel read_bytes): sequential+readahead 34.1%, sequential+FADV_RANDOM 50.3%, random 64.1% — all material undercounts. On the write side there's no in-process analog at all (pwritev2(RWF_NOWAIT) → ENOTSUP), and physical write cost is decoupled from the write() call anyway (writeback + journal + LSMT amplification).

Question for the community: given that physical disk-I/O weighting inherently spans both reads and writes, and cgroup v2 is the only signal above that faithfully covers both — is there an existing plan or preferred direction for adopting cgroup v2 (io.weight/io.max) here? I'd like to align on scope before investing in the per-device cgroup/worker refactor.

<!-- gh-comment-id:5164712795 --> @rogeroger-yu commented on GitHub (Aug 3, 2026): Thanks for pushing on this — I ran the boundary validation end-to-end and want to share the raw findings across the approaches I tried. **Test setup:** local overlaybd-native image (8 compressed zfile layers), buffered ioEngine (page cache on), guest `dd` on the extra drive, host cache dropped between cold runs. Ground truth for "physical disk bytes" is the kernel's own `/proc/self/io` `read_bytes`/`write_bytes` (the same counter `cgroup io.stat` draws from). **Approaches evaluated:** | # | Approach | Result | Defect | |---|---|---|---| | A | Per-device **byte** metrics at LocalFile (`*_local_read_bytes_total{dev_id}`) | Works, attributes correctly per dev_id | Counts **backend request bytes**, not physical disk I/O — blind to page cache + readahead. This is the gap flagged in the review. | | B | Per-device physical-read **time** histogram (`*_local_read_seconds{dev_id}`) | Not usable as a cache/disk signal | Size-dominated, not cache-vs-disk. Avg physical read ≈170 KiB (compressed extents), so even fully-warm reads sit in 0.1–1ms (memcpy cost), not µs. Warm `dd` 36.1 MB/s ≈ cold 37.5 MB/s (decompression-bound). Time can't discriminate cache from disk here. | | C | `RWF_NOWAIT` in-process cache-residency **byte meter** (probe cache; miss → real read + charge, hit → free) | Partially works, then breaks down | (1) Clean binary case holds (cold 100% miss, warm 100% hit). (2) Steady-state it **undercounts physical reads by 36–66%** vs kernel `read_bytes` (readahead fills neighbors "for free" to the probe). (3) **Read-only**: `pwritev2(RWF_NOWAIT)` returns `ENOTSUP` on ext4 buffered writes — no write-path analog. | | D | **cgroup v2** `io.stat` / `io.max` / `io.weight` | Correct signal | Faithful physical **read + write** bytes/iops per group, and can throttle both. Cost: substantial refactor (per-device cgroup + worker attribution). | Measured `RWF_NOWAIT` charged/actual (kernel `read_bytes`): sequential+readahead **34.1%**, sequential+`FADV_RANDOM` **50.3%**, random **64.1%** — all material undercounts. On the write side there's no in-process analog at all (`pwritev2(RWF_NOWAIT)` → `ENOTSUP`), and physical write cost is decoupled from the `write()` call anyway (writeback + journal + LSMT amplification). **Question for the community:** given that physical disk-I/O weighting inherently spans both reads and writes, and cgroup v2 is the only signal above that faithfully covers both — is there an existing plan or preferred direction for adopting cgroup v2 (`io.weight`/`io.max`) here? I'd like to align on scope before investing in the per-device cgroup/worker refactor.

bsakaguchi commented 2026-10-29 14:18:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Aug 4, 2026):

Refactoring the I/O model around cgroups carries real risk and will likely require substantial testing before it can be safely enabled. That said, I believe this work is necessary to properly support multi-tenancy and enforce per-tenant quotas. Given the scope, I'd suggest draft an RFC first to align on the approach before proceeding.

<!-- gh-comment-id:5180694484 --> @yingdi-shan commented on GitHub (Aug 4, 2026): Refactoring the I/O model around cgroups carries real risk and will likely require substantial testing before it can be safely enabled. That said, I believe this work is necessary to properly support multi-tenancy and enforce per-tenant quotas. Given the scope, I'd suggest draft an RFC first to align on the approach before proceeding.

bsakaguchi referenced this issue 2026-10-29 14:19:58 +00:00

[PR #23] [CLOSED] feat: support virtual-host addressing for S3-compatible snapshot back #125

bsakaguchi referenced this issue 2026-10-29 14:21:15 +00:00

[PR #176] [MERGED] Support configurable S3 addressing style #228

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#23

No description provided.