[GH-ISSUE #274] OverlayBD direct I/O hard-codes 512-byte alignment and fails with EINVAL on 4096-byte-sector block devices #75

New issue

Open

opened 2026-10-29 14:19:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:13 +00:00

Owner

Originally created by @ethe on GitHub (Sep 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/274

Affected component

OverlayBD / image resolution

AgentENV version

Version or commit: main at b0d649c411 (2026-09-10)

Host environment

OS: Ubuntu 24.04 (Depot CI runner image depot-ubuntu-24.04)
Kernel (uname -a): 6.12.8+ x86_64
Architecture: x86_64
Bare metal, VM, or cloud: cloud VM (Depot-hosted GitHub Actions runner), KVM available
ublk support: yes
CPU: 8 vCPU
Memory: 31 GiB

Storage/filesystem: root is ext4 on /dev/mapper/root (dm thin pool over virtio disks vda/vdb). The block device reports 4096-byte logical and physical sectors:

$ lsblk -t
NAME       ALIGNMENT MIN-IO  OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE   RA WSAME
vda                0 262144 2097152    4096    4096    1 mq-deadline     256 4096    0B
└─thinpool         0 262144 2097152    4096    4096    1                     4096    0B
  └─root           0 262144 2097152    4096    4096    1                     4096    0B
$ findmnt -no SOURCE,FSTYPE,OPTIONS --target /tmp
/dev/mapper/root ext4   rw,noatime,lazytime,nobarrier

Operation and configuration

# Test suite (TMPDIR defaults to /tmp on the 4096-byte-sector filesystem)
make test-ublk PROFILE=debug

# Minimal syscall-level reproduction of the same failure, no AgentENV involved
fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio --name=p --directory=/tmp --ioengine=libaio   --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio --name=p --directory=/tmp --ioengine=psync    --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
# The same commands with --bs=4096 succeed.

Production exposure with default configuration is limited: LocalFile::direct_io(true) is only used for lower layers when the image config selects ioEngine libaio (image_file.rs, direct_io = io_engine == IO_ENGINE_LIBAIO), plus the lsmt direct-I/O paths. On such a host, enabling libaio makes every lower-layer open fail with Invalid argument, and the test suite always fails.

Steps to reproduce

Use a Linux host whose filesystem for TMPDIR sits on a block device with blockdev --getss = 4096 (4K-native NVMe namespaces and some cloud/virtio disks.
Check out main and run make test-ublk PROFILE=debug.
Observe the O_DIRECT tests in storage/overlaybd fail with EINVAL/EIO.
Optionally run the fio commands above: 512-byte direct I/O fails, 4096-byte direct I/O succeeds, on any of the io_uring, libaio, or psync engines.

Expected behavior

storage/overlaybd/src/backend/local.rs should derive the direct-I/O alignment from the backing device instead of hard-coding it:

const DIRECT_IO_ALIGNMENT: usize = 512;

Linux (ext4/xfs via iomap) rejects O_DIRECT requests unless offset, length and buffer are aligned to the device's logical block size (bdev_logical_block_size), which is 4096 on these hosts. The backend could query BLKSSZGET on the underlying device (or use statx's stx_dio_mem_align/stx_dio_offset_align on kernels ≥ 6.1, which report exactly these limits for a file) when opening a direct-I/O file, and size AlignedBuffer and the offset/length checks from that value. Failing that, opening with direct_io(true) on such a device should return a clear error naming the required alignment rather than surfacing raw EINVAL on the first write.

Actual behavior

Eight tests fail, every direct-I/O write or open returns EINVAL (or EIO from the lsmt create path):

test backend::local::tests::test_write_bytes_at_direct_io ... FAILED
test backend::local::tests::test_write_direct_at_aligned_ptr_fast_path ... FAILED
test backend::local::tests::test_write_direct_at_unaligned_ptr_bounce ... FAILED
test image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine ... FAILED
test lsmt::file::tests::test_direct_io_close_seal ... FAILED
test lsmt::file::tests::test_direct_io_create_open ... FAILED
test lsmt::file::tests::test_direct_io_create_sparse ... FAILED
test lsmt::file::tests::test_direct_io_write_and_read ... FAILED

thread 'backend::local::tests::test_write_bytes_at_direct_io' panicked at storage/overlaybd/src/backend/local.rs:1060:44:
write_bytes_at: Invalid argument (os error 22)
thread 'backend::local::tests::test_write_direct_at_aligned_ptr_fast_path' panicked at storage/overlaybd/src/backend/local.rs:977:14:
aligned write: Invalid argument (os error 22)
thread 'backend::local::tests::test_write_direct_at_unaligned_ptr_bounce' panicked at storage/overlaybd/src/backend/local.rs:1007:14:
unaligned-ptr write via bounce buffer: Invalid argument (os error 22)
thread 'image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine' panicked at storage/overlaybd/src/image/image_file.rs:3004:14:
open image with libaio: failed to open lower layer 0
    Invalid argument (os error 22)
thread 'lsmt::file::tests::test_direct_io_close_seal' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3172:10:
create direct-io LSMTFile failed: Input/output error (os error 5)
thread 'lsmt::file::tests::test_direct_io_write_and_read' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3207:38:
direct-io write: Invalid argument (os error 22)

test result: FAILED. 310 passed; 8 failed; 1 ignored; 0 measured; 0 filtered out

Logs and diagnostics

$ fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=512, buflen=512
$ fio --name=p --directory=/tmp --ioengine=libaio --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512
$ fio --name=p --directory=/tmp --ioengine=psync --direct=1 --bs=512 --size=1M --rw=write --iodepth=4
fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512

# python3: os.pwrite on an O_DIRECT fd with a page-aligned buffer
/tmp: pwrite 512 aligned -> EINVAL
/tmp: pwrite 4096 aligned -> ok (4096)

# Same tests on a 512-byte-sector loop device mounted over /tmp on the same host
$ truncate -s 24G img && losetup --find --show --sector-size 512 img && mkfs.ext4 -q -F /dev/loopN && mount /dev/loopN /tmp
$ make test-ublk PROFILE=debug
test result: ok. 318 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

Reproduction frequency

Always

Regression information

Last known good: not a regression as far as I can tell, the constant has been 512 since the direct-I/O backend was introduced. The tests pass on hosts whose device reports 512-byte logical sectors (GitHub-hosted runners, 512e NVMe).
First known bad: n/a

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @ethe on GitHub (Sep 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/274 ### Affected component OverlayBD / image resolution ### AgentENV version - Version or commit: `main` at b0d649c41112217ae3354eb189a6fb9996e5529b (2026-09-10) ### Host environment - OS: Ubuntu 24.04 (Depot CI runner image `depot-ubuntu-24.04`) - Kernel (`uname -a`): `6.12.8+` x86_64 - Architecture: x86_64 - Bare metal, VM, or cloud: cloud VM (Depot-hosted GitHub Actions runner), KVM available - ublk support: yes - CPU: 8 vCPU - Memory: 31 GiB - Storage/filesystem: root is ext4 on `/dev/mapper/root` (dm thin pool over virtio disks `vda`/`vdb`). The block device reports **4096-byte logical and physical sectors**: ``` $ lsblk -t NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME vda 0 262144 2097152 4096 4096 1 mq-deadline 256 4096 0B └─thinpool 0 262144 2097152 4096 4096 1 4096 0B └─root 0 262144 2097152 4096 4096 1 4096 0B $ findmnt -no SOURCE,FSTYPE,OPTIONS --target /tmp /dev/mapper/root ext4 rw,noatime,lazytime,nobarrier ``` ### Operation and configuration ```shell # Test suite (TMPDIR defaults to /tmp on the 4096-byte-sector filesystem) make test-ublk PROFILE=debug # Minimal syscall-level reproduction of the same failure, no AgentENV involved fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 fio --name=p --directory=/tmp --ioengine=libaio --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 fio --name=p --directory=/tmp --ioengine=psync --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 # The same commands with --bs=4096 succeed. ``` Production exposure with default configuration is limited: `LocalFile::direct_io(true)` is only used for lower layers when the image config selects `ioEngine` libaio (`image_file.rs`, `direct_io = io_engine == IO_ENGINE_LIBAIO`), plus the lsmt direct-I/O paths. On such a host, enabling libaio makes every lower-layer open fail with `Invalid argument`, and the test suite always fails. ### Steps to reproduce 1. Use a Linux host whose filesystem for `TMPDIR` sits on a block device with `blockdev --getss` = 4096 (4K-native NVMe namespaces and some cloud/virtio disks. 2. Check out `main` and run `make test-ublk PROFILE=debug`. 3. Observe the O_DIRECT tests in `storage/overlaybd` fail with `EINVAL`/`EIO`. 4. Optionally run the `fio` commands above: 512-byte direct I/O fails, 4096-byte direct I/O succeeds, on any of the io_uring, libaio, or psync engines. ### Expected behavior `storage/overlaybd/src/backend/local.rs` should derive the direct-I/O alignment from the backing device instead of hard-coding it: ```rust const DIRECT_IO_ALIGNMENT: usize = 512; ``` Linux (ext4/xfs via iomap) rejects O_DIRECT requests unless offset, length and buffer are aligned to the device's logical block size (`bdev_logical_block_size`), which is 4096 on these hosts. The backend could query `BLKSSZGET` on the underlying device (or use `statx`'s `stx_dio_mem_align`/`stx_dio_offset_align` on kernels ≥ 6.1, which report exactly these limits for a file) when opening a direct-I/O file, and size `AlignedBuffer` and the offset/length checks from that value. Failing that, opening with `direct_io(true)` on such a device should return a clear error naming the required alignment rather than surfacing raw `EINVAL` on the first write. ### Actual behavior Eight tests fail, every direct-I/O write or open returns `EINVAL` (or `EIO` from the lsmt create path): ``` test backend::local::tests::test_write_bytes_at_direct_io ... FAILED test backend::local::tests::test_write_direct_at_aligned_ptr_fast_path ... FAILED test backend::local::tests::test_write_direct_at_unaligned_ptr_bounce ... FAILED test image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine ... FAILED test lsmt::file::tests::test_direct_io_close_seal ... FAILED test lsmt::file::tests::test_direct_io_create_open ... FAILED test lsmt::file::tests::test_direct_io_create_sparse ... FAILED test lsmt::file::tests::test_direct_io_write_and_read ... FAILED thread 'backend::local::tests::test_write_bytes_at_direct_io' panicked at storage/overlaybd/src/backend/local.rs:1060:44: write_bytes_at: Invalid argument (os error 22) thread 'backend::local::tests::test_write_direct_at_aligned_ptr_fast_path' panicked at storage/overlaybd/src/backend/local.rs:977:14: aligned write: Invalid argument (os error 22) thread 'backend::local::tests::test_write_direct_at_unaligned_ptr_bounce' panicked at storage/overlaybd/src/backend/local.rs:1007:14: unaligned-ptr write via bounce buffer: Invalid argument (os error 22) thread 'image::image_file::tests::test_image_file_open_local_lower_with_libaio_io_engine' panicked at storage/overlaybd/src/image/image_file.rs:3004:14: open image with libaio: failed to open lower layer 0 Invalid argument (os error 22) thread 'lsmt::file::tests::test_direct_io_close_seal' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3172:10: create direct-io LSMTFile failed: Input/output error (os error 5) thread 'lsmt::file::tests::test_direct_io_write_and_read' panicked at storage/overlaybd/src/lsmt/file/tests.rs:3207:38: direct-io write: Invalid argument (os error 22) test result: FAILED. 310 passed; 8 failed; 1 ignored; 0 measured; 0 filtered out ``` ### Logs and diagnostics ``` $ fio --name=p --directory=/tmp --ioengine=io_uring --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512 fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=512, buflen=512 $ fio --name=p --directory=/tmp --ioengine=libaio --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512 $ fio --name=p --directory=/tmp --ioengine=psync --direct=1 --bs=512 --size=1M --rw=write --iodepth=4 fio: io_u error on file /tmp/p.0.0: Invalid argument: write offset=0, buflen=512 # python3: os.pwrite on an O_DIRECT fd with a page-aligned buffer /tmp: pwrite 512 aligned -> EINVAL /tmp: pwrite 4096 aligned -> ok (4096) # Same tests on a 512-byte-sector loop device mounted over /tmp on the same host $ truncate -s 24G img && losetup --find --show --sector-size 512 img && mkfs.ext4 -q -F /dev/loopN && mount /dev/loopN /tmp $ make test-ublk PROFILE=debug test result: ok. 318 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out ``` ### Reproduction frequency Always ### Regression information - Last known good: not a regression as far as I can tell, the constant has been 512 since the direct-I/O backend was introduced. The tests pass on hosts whose device reports 512-byte logical sectors (GitHub-hosted runners, 512e NVMe). - First known bad: n/a ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.