[GH-ISSUE #302] bug(ublk): pooled devices can expose stale placeholder pages after image switch #81

New issue

Open

opened 2026-10-29 14:19:19 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:19:19 +00:00

Owner

Copy link

Originally created by @dushulin on GitHub (Sep 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/302

Summary

A pooled ublk device can return cached zero pages from its idle placeholder after being rebound to a nonzero business image. In an isolated one-device experiment, buffered reads returned zeros while O_DIRECT reads of the same offset returned the correct snapshot content. Flushing only that device's page cache restored correct buffered reads.

We investigated this after intermittent guest kernel panics during concurrent snapshot restores, surfaced to the caller as timed out waiting for envd. The stale-page mechanism is reproduced; its connection to the original guest panics is strongly supported but not fully proven. We have not identified the reader that populated the placeholder cache during the original failures or reproduced a complete VM panic in the reduced experiment.

Affected component

ublk / ublk-daemon, specifically pooled device image switching and host block-device page-cache consistency.

AgentENV version

• Source inspected: upstream main at e330917f17d9f5103066df2aea5c2e172d6c6527.
• Runtime reproduction used a downstream build. Its prepare_overlaybd_device, idle_released_device, and clear_page_cache functions are byte-for-byte identical to those in the upstream revision above. This is a source comparison, not a claim that the complete upstream revision was runtime-tested.
• Local modifications: present outside these compared functions.

Host environment

• Host kernel: Linux 6.8.0-101-generic.
• Architecture: x86_64; host page size: 4096 bytes.
• ublk enabled; device pool enabled.
• Guest: 1 vCPU, 2048 MiB, Linux 6.1.175; Firecracker v1.15.1-based build.
• Memory restore: OverlayBD memory image exposed through /dev/ublkbN, used as Firecracker file-backed memory.
• The tested memory lower was an immutable filesystem-backed OverlayBD commit (file lower, empty repoBlobUrl), not a registry-backed source.

Operation and configuration

The original operation was creating sandboxes from committed snapshots. The reduced experiment only starts a separate daemon with one private device; it does not start Firecracker, envd, or readiness probes.

Relevant isolated daemon options:

--enable-pool
--pool-low-watermark 0
--pool-high-watermark 1
--pool-startup-prewarm false

Use a separate socket and configuration/cache directory for the experiment. All flushes below target only the experiment's own device.

Steps to reproduce

Supply a valid read-only OverlayBD image with a known nonzero, 4096-byte-aligned page. We used a 2 GiB memory image and offset 0x7ffce000.

1. Start the isolated daemon with the pool configuration above.
2. Acquire the image in shared mode (acquire_overlaybd, access_mode: "shared", virtual_size: 2147483648). Read the selected page using aligned O_DIRECT I/O and save its content as original; verify it is nonzero.
3. Release the device using release_overlaybd. It remains in the pool bound to the empty placeholder.
4. Open a separate read-only FD to this idle device and keep it open across the subsequent acquire. This models a concurrent reader whose FD/cache lifetime spans the handover. Verify that an O_DIRECT read now returns zeros. In the controlled experiment, wait 200 ms and issue BLKFLSBUF on this private device to discard any old business-image read-ahead; then perform a buffered read and verify it returns zeros. The test intentionally fills the placeholder page cache at this point.
5. Acquire the original image again. Verify that the daemon returns the same device ID.
6. Read the selected offset through the buffered path: it still returns zeros. Read the same offset using O_DIRECT: it returns original.
7. Issue BLKFLSBUF on this private device after the switch. The buffered read now returns original as well.
8. Close the retained FD, release the device, and shut down the isolated daemon.

The retained reader FD and explicit placeholder read are part of the reproducer, not an observation about which process was responsible in the original workload. The experiment does not assume a particular udev or block-probing implementation.

Expected behavior

After an acquire completes, buffered and direct reads must refer to the newly bound image. Cache pages populated while the device was idle must not be served as guest memory from a different image.

Actual behavior

At the same offset of the same private device:

Binding / read path	First little-endian u64
Original snapshot, O_DIRECT	0x7ffcd067
Idle placeholder, direct and buffered	0x0
Reacquired snapshot, buffered	0x0 (incorrect)
Reacquired snapshot, O_DIRECT	0x7ffcd067 (correct)
Reacquired snapshot, buffered after device-specific BLKFLSBUF	0x7ffcd067 (correct)

The complete 4096-byte pages matched before the switch, on the direct-read control, and after the flush. The stale buffered page was entirely zero.

An earlier experiment encountered old business-image data while the device was already on the placeholder, so the final controlled experiment explicitly discarded that old cache before filling the placeholder page. Both observations indicate that changing the target alone does not establish a new page-cache identity.

Logs and diagnostics

Sanitized guest error excerpt from the original restore failures:

BUG: unable to handle page fault
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 7ffce067 P4D 7ffce067 PUD 0
Kernel panic - not syncing: Fatal exception in interrupt
Firecracker exiting successfully. exit_code=0

Several failures first faulted in receive_mergeable; another first faulted in kfree on an ACPI GED call path and subsequently faulted in multiple kernel paths. The common evidence is the missing page-table entry, not a proven virtio-net defect. The tested offset 0x7ffce000 is the PUD-table location indicated by this page-table walk (PUD index 0).

In one representative case, snapshot load returned successfully in approximately 30 ms, the VM resumed, and Firecracker exited after the guest panic approximately 2.3 seconds later. The caller received the envd readiness timeout about 57.7 seconds after that exit. Snapshot load uses demand-backed memory, so its short duration alone does not prove all memory reads were complete.

Suspected mechanism in current upstream code

• idle_released_device clears the page cache before switching to the idle placeholder.
• prepare_overlaybd_device switches the target to the requested image and returns the device, without a post-switch cache invalidation.
• clear_page_cache is currently a best-effort BLKFLSBUF.

Clearing before release does not cover cache pages created during the placeholder phase and retained across checkout. Newly created/prewarmed placeholder devices also need a consistent handover; the issue is not necessarily limited to devices previously used by another VM.

A fix needs to consider old in-flight I/O, target switching, and cache invalidation together before the new image is exposed. The flush control in this report demonstrates the cache mismatch; it does not establish that adding one unsynchronized flush is a complete fix.

Reproduction frequency

Intermittent guest failures in the original workload. The reduced one-device experiment reproduced the incorrect buffered read with the explicit retained-reader/cache-fill sequence above. No claim is made that the original VM failure rate or trigger has been reproduced deterministically.

Regression information

• Last known good: unknown.
• First known bad: unknown.
• The relevant switching/flush functions are still present unchanged in the inspected upstream revision.

Additional context

The reduced cache experiment does not depend on readiness probes. Separately, EnvdInstance::wait_for_ready only polls health with a deadline; observing Firecracker process exit during this wait would help report the original VM failure promptly instead of masking it as an envd timeout. That would improve diagnostics but would not fix the stale memory contents.

The experiment's private daemon exited successfully and its device was removed. No global page-cache flush was used.

Pre-submission checklist

• I searched existing open and closed issues and did not find a duplicate.
• I provided a reduced reproduction procedure and stated its limits.
• I removed credentials, private image references, deployment identifiers, hostnames, addresses, and private paths.
• This report concerns a reproduced data-correctness defect; no security vulnerability is being reported.

Originally created by @dushulin on GitHub (Sep 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/302 ## Summary A pooled ublk device can return cached zero pages from its idle placeholder after being rebound to a nonzero business image. In an isolated one-device experiment, buffered reads returned zeros while `O_DIRECT` reads of the same offset returned the correct snapshot content. Flushing only that device's page cache restored correct buffered reads. We investigated this after intermittent guest kernel panics during concurrent snapshot restores, surfaced to the caller as `timed out waiting for envd`. The stale-page mechanism is reproduced; its connection to the original guest panics is strongly supported but not fully proven. We have not identified the reader that populated the placeholder cache during the original failures or reproduced a complete VM panic in the reduced experiment. ## Affected component ublk / ublk-daemon, specifically pooled device image switching and host block-device page-cache consistency. ## AgentENV version - Source inspected: upstream `main` at [`e330917f17d9f5103066df2aea5c2e172d6c6527`](https://github.com/kvcache-ai/AgentENV/commit/e330917f17d9f5103066df2aea5c2e172d6c6527). - Runtime reproduction used a downstream build. Its `prepare_overlaybd_device`, `idle_released_device`, and `clear_page_cache` functions are byte-for-byte identical to those in the upstream revision above. This is a source comparison, not a claim that the complete upstream revision was runtime-tested. - Local modifications: present outside these compared functions. ## Host environment - Host kernel: Linux `6.8.0-101-generic`. - Architecture: x86_64; host page size: 4096 bytes. - ublk enabled; device pool enabled. - Guest: 1 vCPU, 2048 MiB, Linux `6.1.175`; Firecracker v1.15.1-based build. - Memory restore: OverlayBD memory image exposed through `/dev/ublkbN`, used as Firecracker file-backed memory. - The tested memory lower was an immutable filesystem-backed OverlayBD commit (`file` lower, empty `repoBlobUrl`), not a registry-backed source. ## Operation and configuration The original operation was creating sandboxes from committed snapshots. The reduced experiment only starts a separate daemon with one private device; it does not start Firecracker, envd, or readiness probes. Relevant isolated daemon options: ```text --enable-pool --pool-low-watermark 0 --pool-high-watermark 1 --pool-startup-prewarm false ``` Use a separate socket and configuration/cache directory for the experiment. All flushes below target only the experiment's own device. ## Steps to reproduce Supply a valid read-only OverlayBD image with a known nonzero, 4096-byte-aligned page. We used a 2 GiB memory image and offset `0x7ffce000`. 1. Start the isolated daemon with the pool configuration above. 2. Acquire the image in shared mode (`acquire_overlaybd`, `access_mode: "shared"`, `virtual_size: 2147483648`). Read the selected page using aligned `O_DIRECT` I/O and save its content as `original`; verify it is nonzero. 3. Release the device using `release_overlaybd`. It remains in the pool bound to the empty placeholder. 4. Open a separate read-only FD to this idle device and **keep it open across the subsequent acquire**. This models a concurrent reader whose FD/cache lifetime spans the handover. Verify that an `O_DIRECT` read now returns zeros. In the controlled experiment, wait 200 ms and issue `BLKFLSBUF` on this private device to discard any old business-image read-ahead; then perform a buffered read and verify it returns zeros. The test intentionally fills the placeholder page cache at this point. 5. Acquire the original image again. Verify that the daemon returns the same device ID. 6. Read the selected offset through the buffered path: it still returns zeros. Read the same offset using `O_DIRECT`: it returns `original`. 7. Issue `BLKFLSBUF` on this private device after the switch. The buffered read now returns `original` as well. 8. Close the retained FD, release the device, and shut down the isolated daemon. The retained reader FD and explicit placeholder read are part of the reproducer, not an observation about which process was responsible in the original workload. The experiment does not assume a particular udev or block-probing implementation. ## Expected behavior After an acquire completes, buffered and direct reads must refer to the newly bound image. Cache pages populated while the device was idle must not be served as guest memory from a different image. ## Actual behavior At the same offset of the same private device: | Binding / read path | First little-endian u64 | |---|---| | Original snapshot, `O_DIRECT` | `0x7ffcd067` | | Idle placeholder, direct and buffered | `0x0` | | Reacquired snapshot, buffered | **`0x0` (incorrect)** | | Reacquired snapshot, `O_DIRECT` | **`0x7ffcd067` (correct)** | | Reacquired snapshot, buffered after device-specific `BLKFLSBUF` | **`0x7ffcd067` (correct)** | The complete 4096-byte pages matched before the switch, on the direct-read control, and after the flush. The stale buffered page was entirely zero. An earlier experiment encountered old business-image data while the device was already on the placeholder, so the final controlled experiment explicitly discarded that old cache before filling the placeholder page. Both observations indicate that changing the target alone does not establish a new page-cache identity. ## Logs and diagnostics Sanitized guest error excerpt from the original restore failures: ```text BUG: unable to handle page fault #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 7ffce067 P4D 7ffce067 PUD 0 Kernel panic - not syncing: Fatal exception in interrupt Firecracker exiting successfully. exit_code=0 ``` Several failures first faulted in `receive_mergeable`; another first faulted in `kfree` on an ACPI GED call path and subsequently faulted in multiple kernel paths. The common evidence is the missing page-table entry, not a proven virtio-net defect. The tested offset `0x7ffce000` is the PUD-table location indicated by this page-table walk (PUD index 0). In one representative case, snapshot load returned successfully in approximately 30 ms, the VM resumed, and Firecracker exited after the guest panic approximately 2.3 seconds later. The caller received the envd readiness timeout about 57.7 seconds after that exit. Snapshot load uses demand-backed memory, so its short duration alone does not prove all memory reads were complete. ## Suspected mechanism in current upstream code - [`idle_released_device`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1307) clears the page cache **before** switching to the idle placeholder. - [`prepare_overlaybd_device`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1080) switches the target to the requested image and returns the device, without a post-switch cache invalidation. - [`clear_page_cache`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1603) is currently a best-effort `BLKFLSBUF`. Clearing before release does not cover cache pages created during the placeholder phase and retained across checkout. Newly created/prewarmed placeholder devices also need a consistent handover; the issue is not necessarily limited to devices previously used by another VM. A fix needs to consider old in-flight I/O, target switching, and cache invalidation together before the new image is exposed. The flush control in this report demonstrates the cache mismatch; it does **not** establish that adding one unsynchronized flush is a complete fix. ## Reproduction frequency Intermittent guest failures in the original workload. The reduced one-device experiment reproduced the incorrect buffered read with the explicit retained-reader/cache-fill sequence above. No claim is made that the original VM failure rate or trigger has been reproduced deterministically. ## Regression information - Last known good: unknown. - First known bad: unknown. - The relevant switching/flush functions are still present unchanged in the inspected upstream revision. ## Additional context The reduced cache experiment does not depend on readiness probes. Separately, [`EnvdInstance::wait_for_ready`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/src/sandbox/envd.rs#L109) only polls health with a deadline; observing Firecracker process exit during this wait would help report the original VM failure promptly instead of masking it as an envd timeout. That would improve diagnostics but would not fix the stale memory contents. The experiment's private daemon exited successfully and its device was removed. No global page-cache flush was used. ## Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a reduced reproduction procedure and stated its limits. - [x] I removed credentials, private image references, deployment identifiers, hostnames, addresses, and private paths. - [x] This report concerns a reproduced data-correctness defect; no security vulnerability is being reported.

bsakaguchi commented 2026-10-29 14:19:20 +00:00

Author

Owner

Copy link

@NickNYU commented on GitHub (Oct 5, 2026):

Hi @dushulin

In order to understand this bug comprehensively, I just reproduced

1. on current upstream main (5843159)
2. Linux 7.0.11 x86_64
3. using an in-process UblkDaemonServer with your pool shape (low=0, high=1, prewarm=false) — no Firecracker, envd, or external reader process involved.

Results match your table exactly:

after reacquiring the same pooled device (same dev_id), the buffered read at 0x7ffce000 returned 0x0
while O_DIRECT returned the correct marker, and BLKFLSBUF restored the buffered path.
Since the test itself holds the only reader FD, the poisoning is independent of udev/probing — any buffered read while the device sits on the placeholder is enough.

I have encoded the procedure as an #[ignore]d regression test in #342 (fails on current code at the post-switch buffered assertion; meant to be un-ignored together with the fix). Details and validation results are in the PR.

will follow up on the fix direction separately after looking into the in-flight I/O quiescing question you raised.

<!-- gh-comment-id:5996929550 --> @NickNYU commented on GitHub (Oct 5, 2026): Hi @dushulin In order to understand this bug comprehensively, I just reproduced 1. on current upstream `main` (5843159) 2. Linux 7.0.11 x86_64 3. using an in-process `UblkDaemonServer` with your pool shape (low=0, high=1, prewarm=false) — no Firecracker, envd, or external reader process involved. Results match your table exactly: > after reacquiring the same pooled device (same `dev_id`), the buffered read at `0x7ffce000` returned `0x0` > while `O_DIRECT` returned the correct marker, and `BLKFLSBUF` restored the buffered path. > Since the test itself holds the only reader FD, the poisoning is independent of udev/probing — any buffered read while the device sits on the placeholder is enough. I have encoded the procedure as an `#[ignore]`d regression test in #342 (fails on current code at the post-switch buffered assertion; meant to be un-ignored together with the fix). Details and validation results are in the PR. will follow up on the fix direction separately after looking into the in-flight I/O quiescing question you raised.

bsakaguchi referenced this issue 2026-10-29 14:20:31 +00:00

[PR #81] [CLOSED] fix(orchestrator): roll back network policy when metadata persist fails #164

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#81

No description provided.