[GH-ISSUE #391] bug(overlaybd): failed pause persistence deletes a live disk layer #98

New issue

Open

opened 2026-10-29 14:19:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-29 14:19:34 +00:00

Owner

Copy link

Originally created by @Hygge-Gezelligheid on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/391

When runtime and pause artifacts are on the same filesystem, OverlayBD restacking puts a disk layer still needed by the live sandbox inside the pause artifact directory. If saving the paused record fails, cleanup deletes this directory and AgentENV resumes the sandbox with a missing disk layer.

Affected component

Orchestrator / sandbox lifecycle

AgentENV version

• Commit: f9a39636caa3d234bbb96d2406685d7a6d3dabc4 (0.2.3), unmodified product source.
• Firecracker 1.15.1-patch-v1; guest kernel 6.1.175.

Host environment

• Ubuntu 24.04.4, kernel 7.0.0-30-generic, x86_64, KVM and ublk.
• ext4; writable runtime and pause artifacts on the same filesystem.
• Public ubuntu:24.04 sandbox, 1 vCPU, 256 MiB RAM.

Operation and configuration

POST /sandboxes/{sandbox_id}/pause. Firecracker process pool disabled; block pool enabled. Complete configuration and launch commands are in the folded reproduction below.

Steps to reproduce

1. Create a sandbox from ubuntu:24.04 using the configuration below.
2. Arm the supplied injector to return ENOSPC for exactly one write to the paused-record RocksDB WAL.
3. Pause the sandbox. Check its API state and the disk-layer paths in its live OverlayBD image.json.
4. With the injector disarmed, retry pause, then query the sandbox.

Expected behavior

If pause fails and the original sandbox is resumed, its disk layers remain available. A later pause must not fail because cleanup removed a live layer.

Actual behavior

The first pause returns HTTP 500. The sandbox then reports running, but its live image.json references a deleted snapshot.commit. Retrying pause fails on that missing file, and the following sandbox lookup returns 404.

The fault spans three code paths:

1. OverlayBD snapshot capture uses the snapshot output path for the live sealed layer and adds it to the live disk configuration.
2. persist_paused() deletes artifact_root when put_record() fails, removing that same layer.
3. The pause error handler calls sandbox.resume() without restoring the deleted layer.

Logs and diagnostics

First failure: persist paused sandbox record: put RocksDB value / No space left on device.

Retry: link or copy runtime overlaybd layer fails because the referenced snapshot.commit no longer exists. Full sanitized output is folded below.

Reproduction frequency

Once with the supplied real KVM reproduction. The normal pause/resume control passed; the injector recorded exactly one failed write.

Regression information

Confirmed at the commit above. The introducing commit and last known good version are unknown.

Additional context

Complete reproduction: build and launch commands, two source files, result and logs

Use an empty dedicated test server with working KVM, OverlayBD, ublk and network setup. Prepare the runtime dependencies named below, including the tools image. Choose an unused API port and network ranges. The script creates and removes its own test sandboxes; stop the dedicated server after it finishes.

Build, configuration and launch commands

# The operation under test is POST /sandboxes/{sandbox_id}/pause. Sandbox creation uses POST /sandboxes-cold with:

{"image":"ubuntu:24.04","timeout":600,"cpuCount":1,"memoryMB":256,"secure":true}

# ### Build used for the tested binaries

# Run these in the checkout at the stated commit. The test reused binaries built with Rust 1.93.0 and the following compiler settings:

export RUSTC_BOOTSTRAP=1
export RUSTFLAGS='-Zcrate-attr=feature(atomic_try_update)'
cargo +1.93.0 build --release -p agentenv --bin server
cargo +1.93.0 build --release -p uvm-ublk-daemon --bin uvm-ublk-daemon

# ### Start the test server

# Replace the four absolute paths below with your local paths. The dependency layout is the one used by AgentENV. Save fail_db_write.c and reproduce.py below into the new REPRO_DIR.

export AGENTENV_REPO=/absolute/path/to/AgentENV
export AGENTENV_SERVER=/absolute/path/to/server
export AGENTENV_DAEMON=/absolute/path/to/uvm-ublk-daemon
export AGENTENV_DEPS=/absolute/path/to/agentenv-deps
export REPRO_DIR="$(mktemp -d /tmp/apr.XXXXXX)"
printf '%s\n' "$REPRO_DIR"

# After saving the two source files, run:

gcc -Wall -Wextra -Werror -shared -fPIC -O2 \
  -o "$REPRO_DIR/fail_db_write.so" "$REPRO_DIR/fail_db_write.c"

cat > "$REPRO_DIR/config.toml" <<EOF
home_path = "$REPRO_DIR/home"
runtime_path = "$REPRO_DIR/run"
deps_path = "$AGENTENV_DEPS"
virtualization_mode = "kvm"
[firecracker]
binary_path = "$AGENTENV_DEPS/firecracker/1.15.1-patch-v1/firecracker"
[kernel]
image_path = "$AGENTENV_DEPS/kernel/vmlinux-6.1.175/vmlinux.bin"
[network.internal]
host_interaction_cidr = "10.231.0.0/16"
veth_cidr = "10.232.0.0/16"
[snapshot]
repository_backend = "posix_fs"
[ublk]
daemon_binary_path = "$AGENTENV_DAEMON"
daemon_metrics_listen_addr = ""
[pool]
low_watermark = 1
high_watermark = 2
[pool.firecracker]
enabled = false
[pool.block]
enabled = true
[observability.scheduler_report]
enabled = false
EOF

API_ADDR=127.0.0.1:18782 \
AENV_HOME_PATH="$REPRO_DIR/home" \
AENV_RUNTIME_PATH="$REPRO_DIR/run" \
RUST_LOG=agentenv=debug \
"$AGENTENV_REPO/scripts/run-with-capabilities.sh" env \
  LD_PRELOAD="$REPRO_DIR/fail_db_write.so" \
  AENV_DB_FAIL_PREFIX="$REPRO_DIR/home/persisted-sandboxes/records.db/" \
  AENV_DB_FAIL_MARKER="$REPRO_DIR/fail-next-write" \
  AENV_DB_FAIL_LOG="$REPRO_DIR/faults.log" \
  "$AGENTENV_SERVER" --config "$REPRO_DIR/config.toml" \
  > "$REPRO_DIR/server.log" 2>&1

# In a second terminal, set REPRO_DIR to the directory printed above, wait for a 2xx health response, and run:

curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:18782/health
python3 "$REPRO_DIR/reproduce.py" \
  --url http://127.0.0.1:18782 \
  --home "$REPRO_DIR/home" \
  --fault-dir "$REPRO_DIR"

# The Python script reads the generated API key from the test home and does not print it. Its only Python dependencies are in the standard library.

fail_db_write.c

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/syscall.h>
#include <unistd.h>

/* Inject one ENOSPC into this test database's next numbered WAL write.
 * All other writes go through; unlinking the marker consumes the fault. */
ssize_t write(int fd, const void *data, size_t size) {
    int saved_errno = errno;
    const char *prefix = getenv("AENV_DB_FAIL_PREFIX");
    const char *marker = getenv("AENV_DB_FAIL_MARKER");
    const char *log = getenv("AENV_DB_FAIL_LOG");
    char fd_path[64], path[PATH_MAX], line[PATH_MAX + 64];
    if (!prefix || !marker || !log || access(marker, F_OK) != 0)
        goto passthrough;
    snprintf(fd_path, sizeof(fd_path), "/proc/self/fd/%d", fd);
    ssize_t length = readlink(fd_path, path, sizeof(path) - 1);
    if (length < 0) goto passthrough;
    path[length] = '\0';
    size_t n = strlen(prefix);
    if (strncmp(path, prefix, n) != 0) goto passthrough;
    const char *name = path + n, *suffix = name;
    while (*suffix >= '0' && *suffix <= '9') ++suffix;
    if (suffix == name || strcmp(suffix, ".log") != 0)
        goto passthrough;
    if (unlink(marker) != 0) goto passthrough;
    int log_fd = syscall(SYS_openat, AT_FDCWD, log,
                         O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600);
    if (log_fd >= 0) {
        int count = snprintf(line, sizeof(line), "write ENOSPC %s\n", path);
        if (count > 0 && (size_t)count < sizeof(line))
            syscall(SYS_write, log_fd, line, count);
        syscall(SYS_close, log_fd);
    }
    errno = ENOSPC;
    return -1;
passthrough:
    errno = saved_errno;
    return syscall(SYS_write, fd, data, size);
}

reproduce.py

#!/usr/bin/env python3
"""Reproduce failed pause persistence against a dedicated local AgentENV server.

Python standard library only. See the issue's launch instructions for the
one-shot WAL fault injector. This creates and deletes only its own sandboxes.
"""
import argparse
import http.client
import json
import re
import sys
from pathlib import Path
from urllib.parse import urlparse


parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--url", required=True)
parser.add_argument("--home", required=True, type=Path)
parser.add_argument("--fault-dir", required=True, type=Path)
args = parser.parse_args()
url = urlparse(args.url)
if url.scheme != "http" or url.hostname not in ("127.0.0.1", "localhost", "::1"):
    parser.error("use a dedicated server on loopback")
home = args.home.resolve()
fault_dir = args.fault_dir.resolve()
marker = fault_dir / "fail-next-write"
fault_log = fault_dir / "faults.log"
key_file = home / "secrets/api-key"
key = key_file.read_text().strip() if key_file.exists() else ""
active = set()
identities = {}
result = {"image": "ubuntu:24.04", "fault": "one WAL write returns ENOSPC"}


def clean(text):
    if key:
        text = text.replace(key, "<REDACTED>")
    text = text.replace(str(home), "<TEST_HOME>")
    text = text.replace(str(fault_dir), "<TEST_DIR>")
    def replace_id(match):
        return identities.setdefault(match[0], f"<ID_{len(identities) + 1}>")
    return re.sub(r"\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b",
                  replace_id, text)


def request(method, path, body=None):
    payload = None if body is None else json.dumps(body).encode()
    connection = http.client.HTTPConnection(url.hostname, url.port, timeout=300)
    try:
        connection.request(method, url.path.rstrip("/") + path, body=payload,
                           headers={"Content-Type": "application/json", "X-API-Key": key})
        response = connection.getresponse()
        text = response.read().decode()
        return response.status, json.loads(text) if text else None
    finally:
        connection.close()


def success(method, path, body=None):
    status, value = request(method, path, body)
    assert 200 <= status < 300, (method, path, status, value)
    return value


def create():
    value = success("POST", "/sandboxes-cold", {
        "image": "ubuntu:24.04", "timeout": 600,
        "cpuCount": 1, "memoryMB": 256, "secure": True,
    })
    sid = value["sandboxID"]
    active.add(sid)
    return sid


try:
    assert not marker.exists() and not fault_log.exists(), "use fresh fault state"
    assert success("GET", "/v2/sandboxes") == [], "use an empty test server"
    print("Running a normal pause/resume control...", flush=True)
    control = create()
    success("POST", f"/sandboxes/{control}/pause")
    success("POST", f"/v2/sandboxes/{control}/connect", {"timeout": 600})
    success("DELETE", f"/sandboxes/{control}")
    active.remove(control)
    result["normal_control"] = "passed"

    print("Creating the faulted VM from the same public image...", flush=True)
    sid = create()
    identities[sid] = "<SANDBOX_ID>"
    artifacts = home / "persisted-sandboxes/artifacts"
    runtime = home / "firecracker-work"
    assert artifacts.stat().st_dev == runtime.stat().st_dev, "use one filesystem"
    marker.touch(exist_ok=False)
    try:
        status, error = request("POST", f"/sandboxes/{sid}/pause")
    finally:
        marker.unlink(missing_ok=True)
    result["first_pause"] = {"http_status": status, "error": error}
    assert status == 500 and "persist paused sandbox record" in str(error), result
    injected = fault_log.read_text().splitlines()
    assert len(injected) == 1, injected
    result["injected_operations"] = injected
    state = success("GET", f"/sandboxes/{sid}")["state"]
    assert state.lower() == "running", state
    result["state_after_failed_pause"] = state

    missing = []
    for config in runtime.rglob("image.json"):
        for layer in json.loads(config.read_text()).get("lowers", []):
            path = Path(layer.get("file", ""))
            if str(path).startswith(str(artifacts / sid) + "/") and not path.exists():
                missing.append({"runtime_config": str(config), "missing_layer": str(path)})
    assert missing, "no missing live layer: the bug was not reproduced"
    result["missing_live_layers"] = missing

    status, error = request("POST", f"/sandboxes/{sid}/pause")
    result["retry_pause_without_fault"] = {"http_status": status, "error": error}
    assert status == 500 and any(row["missing_layer"] in str(error) for row in missing)
    assert fault_log.read_text().splitlines() == injected, "unexpected second injection"
    status, _ = request("GET", f"/sandboxes/{sid}")
    result["instance_after_retry_http_status"] = status
    assert status == 404, status
    result["reproduced"] = True
    print(clean(json.dumps(result, indent=2)), flush=True)
except Exception as error:
    print(clean(f"Reproduction did not complete: {error}"), file=sys.stderr)
    sys.exit(1)
finally:
    marker.unlink(missing_ok=True)
    for sid in active:
        try:
            status, _ = request("DELETE", f"/sandboxes/{sid}")
            if not (200 <= status < 300 or status == 404):
                print(clean(f"Cleanup needs attention for {sid}: HTTP {status}"), file=sys.stderr)
        except Exception as error:
            print(clean(f"Cleanup needs attention: {error}"), file=sys.stderr)

Observed result

{
  "image": "ubuntu:24.04",
  "fault": "one WAL write returns ENOSPC",
  "normal_control": "passed",
  "first_pause": {
    "http_status": 500,
    "error": {
      "code": 500,
      "message": "internal error: failed to persist paused sandbox state: paused sandbox store operation failed: persist paused sandbox record: put RocksDB value"
    }
  },
  "injected_operations": [
    "write ENOSPC <TEST_HOME>/persisted-sandboxes/records.db/000004.log"
  ],
  "state_after_failed_pause": "running",
  "missing_live_layers": [
    {
      "runtime_config": "<TEST_HOME>/firecracker-work/agentenv-fc-<RUNTIME_ID>/overlaybd/image.json",
      "missing_layer": "<TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_2>/rootfs/snapshot.commit"
    }
  ],
  "retry_pause_without_fault": {
    "http_status": 500,
    "error": {
      "code": 500,
      "message": "sandbox <SANDBOX_ID> operation Pause failed: link or copy runtime overlaybd layer <TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_2>/rootfs/snapshot.commit -> <TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_3>/rootfs/inherited-layers/0000/snapshot.commit: sparse copy command failed with status exit status: 1"
    }
  },
  "instance_after_retry_http_status": 404,
  "reproduced": true
}

Relevant logs

2026-10-19T02:14:23.123338Z  WARN pause_sandbox: agentenv::orchestrator::service: failed to persist paused sandbox state error=Store { operation: "persist paused sandbox record", source: put RocksDB value
    IO error: No space left on device: While appending to file: <TEST_HOME>/persisted-sandboxes/records.db/000004.log: No space left on device } sandbox_id=<SANDBOX_ID>
2026-10-19T02:14:23.205684Z  WARN agentenv::sandbox::firecracker::overlaybd_snapshot: hard-link runtime overlaybd layer failed; falling back to sparse copy source=<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/snapshot.commit destination=<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/inherited-layers/0000/snapshot.commit error=No such file or directory (os error 2)
cp: cannot stat '<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/snapshot.commit': No such file or directory

Pre-submission checklist

• I searched existing open and closed issues and did not find a duplicate.
• I provided a minimal reproduction or explained why one cannot be provided.
• I removed credentials, tokens, private image references, and other sensitive information.
• This is not a security vulnerability.

Originally created by @Hygge-Gezelligheid on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/391 When runtime and pause artifacts are on the same filesystem, OverlayBD restacking puts a disk layer still needed by the live sandbox inside the pause artifact directory. If saving the paused record fails, cleanup deletes this directory and AgentENV resumes the sandbox with a missing disk layer. ### Affected component Orchestrator / sandbox lifecycle ### AgentENV version - Commit: `f9a39636caa3d234bbb96d2406685d7a6d3dabc4` (0.2.3), unmodified product source. - Firecracker `1.15.1-patch-v1`; guest kernel `6.1.175`. ### Host environment - Ubuntu 24.04.4, kernel `7.0.0-30-generic`, x86_64, KVM and ublk. - ext4; writable runtime and pause artifacts on the same filesystem. - Public `ubuntu:24.04` sandbox, 1 vCPU, 256 MiB RAM. ### Operation and configuration `POST /sandboxes/{sandbox_id}/pause`. Firecracker process pool disabled; block pool enabled. Complete configuration and launch commands are in the folded reproduction below. ### Steps to reproduce 1. Create a sandbox from `ubuntu:24.04` using the configuration below. 2. Arm the supplied injector to return `ENOSPC` for exactly one write to the paused-record RocksDB WAL. 3. Pause the sandbox. Check its API state and the disk-layer paths in its live OverlayBD `image.json`. 4. With the injector disarmed, retry pause, then query the sandbox. ### Expected behavior If pause fails and the original sandbox is resumed, its disk layers remain available. A later pause must not fail because cleanup removed a live layer. ### Actual behavior The first pause returns HTTP 500. The sandbox then reports `running`, but its live `image.json` references a deleted `snapshot.commit`. Retrying pause fails on that missing file, and the following sandbox lookup returns 404. The fault spans three code paths: 1. [OverlayBD snapshot capture](https://github.com/kvcache-ai/AgentENV/blob/f9a39636caa3d234bbb96d2406685d7a6d3dabc4/src/sandbox/firecracker/overlaybd_snapshot.rs#L495) uses the snapshot output path for the live sealed layer and [adds it to the live disk configuration](https://github.com/kvcache-ai/AgentENV/blob/f9a39636caa3d234bbb96d2406685d7a6d3dabc4/src/sandbox/firecracker/overlaybd_snapshot.rs#L560). 2. [`persist_paused()`](https://github.com/kvcache-ai/AgentENV/blob/f9a39636caa3d234bbb96d2406685d7a6d3dabc4/src/orchestrator/persistence/file_backed.rs#L367) deletes `artifact_root` when `put_record()` fails, removing that same layer. 3. [The pause error handler](https://github.com/kvcache-ai/AgentENV/blob/f9a39636caa3d234bbb96d2406685d7a6d3dabc4/src/orchestrator/service.rs#L1564) calls `sandbox.resume()` without restoring the deleted layer. ### Logs and diagnostics First failure: `persist paused sandbox record: put RocksDB value` / `No space left on device`. Retry: `link or copy runtime overlaybd layer` fails because the referenced `snapshot.commit` no longer exists. Full sanitized output is folded below. ### Reproduction frequency Once with the supplied real KVM reproduction. The normal pause/resume control passed; the injector recorded exactly one failed write. ### Regression information Confirmed at the commit above. The introducing commit and last known good version are unknown. ### Additional context <details> <summary>Complete reproduction: build and launch commands, two source files, result and logs</summary> Use an empty dedicated test server with working KVM, OverlayBD, ublk and network setup. Prepare the runtime dependencies named below, including the tools image. Choose an unused API port and network ranges. The script creates and removes its own test sandboxes; stop the dedicated server after it finishes. **Build, configuration and launch commands** ```shell # The operation under test is POST /sandboxes/{sandbox_id}/pause. Sandbox creation uses POST /sandboxes-cold with: {"image":"ubuntu:24.04","timeout":600,"cpuCount":1,"memoryMB":256,"secure":true} # ### Build used for the tested binaries # Run these in the checkout at the stated commit. The test reused binaries built with Rust 1.93.0 and the following compiler settings: export RUSTC_BOOTSTRAP=1 export RUSTFLAGS='-Zcrate-attr=feature(atomic_try_update)' cargo +1.93.0 build --release -p agentenv --bin server cargo +1.93.0 build --release -p uvm-ublk-daemon --bin uvm-ublk-daemon # ### Start the test server # Replace the four absolute paths below with your local paths. The dependency layout is the one used by AgentENV. Save fail_db_write.c and reproduce.py below into the new REPRO_DIR. export AGENTENV_REPO=/absolute/path/to/AgentENV export AGENTENV_SERVER=/absolute/path/to/server export AGENTENV_DAEMON=/absolute/path/to/uvm-ublk-daemon export AGENTENV_DEPS=/absolute/path/to/agentenv-deps export REPRO_DIR="$(mktemp -d /tmp/apr.XXXXXX)" printf '%s\n' "$REPRO_DIR" # After saving the two source files, run: gcc -Wall -Wextra -Werror -shared -fPIC -O2 \ -o "$REPRO_DIR/fail_db_write.so" "$REPRO_DIR/fail_db_write.c" cat > "$REPRO_DIR/config.toml" <<EOF home_path = "$REPRO_DIR/home" runtime_path = "$REPRO_DIR/run" deps_path = "$AGENTENV_DEPS" virtualization_mode = "kvm" [firecracker] binary_path = "$AGENTENV_DEPS/firecracker/1.15.1-patch-v1/firecracker" [kernel] image_path = "$AGENTENV_DEPS/kernel/vmlinux-6.1.175/vmlinux.bin" [network.internal] host_interaction_cidr = "10.231.0.0/16" veth_cidr = "10.232.0.0/16" [snapshot] repository_backend = "posix_fs" [ublk] daemon_binary_path = "$AGENTENV_DAEMON" daemon_metrics_listen_addr = "" [pool] low_watermark = 1 high_watermark = 2 [pool.firecracker] enabled = false [pool.block] enabled = true [observability.scheduler_report] enabled = false EOF API_ADDR=127.0.0.1:18782 \ AENV_HOME_PATH="$REPRO_DIR/home" \ AENV_RUNTIME_PATH="$REPRO_DIR/run" \ RUST_LOG=agentenv=debug \ "$AGENTENV_REPO/scripts/run-with-capabilities.sh" env \ LD_PRELOAD="$REPRO_DIR/fail_db_write.so" \ AENV_DB_FAIL_PREFIX="$REPRO_DIR/home/persisted-sandboxes/records.db/" \ AENV_DB_FAIL_MARKER="$REPRO_DIR/fail-next-write" \ AENV_DB_FAIL_LOG="$REPRO_DIR/faults.log" \ "$AGENTENV_SERVER" --config "$REPRO_DIR/config.toml" \ > "$REPRO_DIR/server.log" 2>&1 # In a second terminal, set REPRO_DIR to the directory printed above, wait for a 2xx health response, and run: curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:18782/health python3 "$REPRO_DIR/reproduce.py" \ --url http://127.0.0.1:18782 \ --home "$REPRO_DIR/home" \ --fault-dir "$REPRO_DIR" # The Python script reads the generated API key from the test home and does not print it. Its only Python dependencies are in the standard library. ``` **fail_db_write.c** ```c #define _GNU_SOURCE #include <errno.h> #include <fcntl.h> #include <limits.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/syscall.h> #include <unistd.h> /* Inject one ENOSPC into this test database's next numbered WAL write. * All other writes go through; unlinking the marker consumes the fault. */ ssize_t write(int fd, const void *data, size_t size) { int saved_errno = errno; const char *prefix = getenv("AENV_DB_FAIL_PREFIX"); const char *marker = getenv("AENV_DB_FAIL_MARKER"); const char *log = getenv("AENV_DB_FAIL_LOG"); char fd_path[64], path[PATH_MAX], line[PATH_MAX + 64]; if (!prefix || !marker || !log || access(marker, F_OK) != 0) goto passthrough; snprintf(fd_path, sizeof(fd_path), "/proc/self/fd/%d", fd); ssize_t length = readlink(fd_path, path, sizeof(path) - 1); if (length < 0) goto passthrough; path[length] = '\0'; size_t n = strlen(prefix); if (strncmp(path, prefix, n) != 0) goto passthrough; const char *name = path + n, *suffix = name; while (*suffix >= '0' && *suffix <= '9') ++suffix; if (suffix == name || strcmp(suffix, ".log") != 0) goto passthrough; if (unlink(marker) != 0) goto passthrough; int log_fd = syscall(SYS_openat, AT_FDCWD, log, O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600); if (log_fd >= 0) { int count = snprintf(line, sizeof(line), "write ENOSPC %s\n", path); if (count > 0 && (size_t)count < sizeof(line)) syscall(SYS_write, log_fd, line, count); syscall(SYS_close, log_fd); } errno = ENOSPC; return -1; passthrough: errno = saved_errno; return syscall(SYS_write, fd, data, size); } ``` **reproduce.py** ```python #!/usr/bin/env python3 """Reproduce failed pause persistence against a dedicated local AgentENV server. Python standard library only. See the issue's launch instructions for the one-shot WAL fault injector. This creates and deletes only its own sandboxes. """ import argparse import http.client import json import re import sys from pathlib import Path from urllib.parse import urlparse parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--url", required=True) parser.add_argument("--home", required=True, type=Path) parser.add_argument("--fault-dir", required=True, type=Path) args = parser.parse_args() url = urlparse(args.url) if url.scheme != "http" or url.hostname not in ("127.0.0.1", "localhost", "::1"): parser.error("use a dedicated server on loopback") home = args.home.resolve() fault_dir = args.fault_dir.resolve() marker = fault_dir / "fail-next-write" fault_log = fault_dir / "faults.log" key_file = home / "secrets/api-key" key = key_file.read_text().strip() if key_file.exists() else "" active = set() identities = {} result = {"image": "ubuntu:24.04", "fault": "one WAL write returns ENOSPC"} def clean(text): if key: text = text.replace(key, "<REDACTED>") text = text.replace(str(home), "<TEST_HOME>") text = text.replace(str(fault_dir), "<TEST_DIR>") def replace_id(match): return identities.setdefault(match[0], f"<ID_{len(identities) + 1}>") return re.sub(r"\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b", replace_id, text) def request(method, path, body=None): payload = None if body is None else json.dumps(body).encode() connection = http.client.HTTPConnection(url.hostname, url.port, timeout=300) try: connection.request(method, url.path.rstrip("/") + path, body=payload, headers={"Content-Type": "application/json", "X-API-Key": key}) response = connection.getresponse() text = response.read().decode() return response.status, json.loads(text) if text else None finally: connection.close() def success(method, path, body=None): status, value = request(method, path, body) assert 200 <= status < 300, (method, path, status, value) return value def create(): value = success("POST", "/sandboxes-cold", { "image": "ubuntu:24.04", "timeout": 600, "cpuCount": 1, "memoryMB": 256, "secure": True, }) sid = value["sandboxID"] active.add(sid) return sid try: assert not marker.exists() and not fault_log.exists(), "use fresh fault state" assert success("GET", "/v2/sandboxes") == [], "use an empty test server" print("Running a normal pause/resume control...", flush=True) control = create() success("POST", f"/sandboxes/{control}/pause") success("POST", f"/v2/sandboxes/{control}/connect", {"timeout": 600}) success("DELETE", f"/sandboxes/{control}") active.remove(control) result["normal_control"] = "passed" print("Creating the faulted VM from the same public image...", flush=True) sid = create() identities[sid] = "<SANDBOX_ID>" artifacts = home / "persisted-sandboxes/artifacts" runtime = home / "firecracker-work" assert artifacts.stat().st_dev == runtime.stat().st_dev, "use one filesystem" marker.touch(exist_ok=False) try: status, error = request("POST", f"/sandboxes/{sid}/pause") finally: marker.unlink(missing_ok=True) result["first_pause"] = {"http_status": status, "error": error} assert status == 500 and "persist paused sandbox record" in str(error), result injected = fault_log.read_text().splitlines() assert len(injected) == 1, injected result["injected_operations"] = injected state = success("GET", f"/sandboxes/{sid}")["state"] assert state.lower() == "running", state result["state_after_failed_pause"] = state missing = [] for config in runtime.rglob("image.json"): for layer in json.loads(config.read_text()).get("lowers", []): path = Path(layer.get("file", "")) if str(path).startswith(str(artifacts / sid) + "/") and not path.exists(): missing.append({"runtime_config": str(config), "missing_layer": str(path)}) assert missing, "no missing live layer: the bug was not reproduced" result["missing_live_layers"] = missing status, error = request("POST", f"/sandboxes/{sid}/pause") result["retry_pause_without_fault"] = {"http_status": status, "error": error} assert status == 500 and any(row["missing_layer"] in str(error) for row in missing) assert fault_log.read_text().splitlines() == injected, "unexpected second injection" status, _ = request("GET", f"/sandboxes/{sid}") result["instance_after_retry_http_status"] = status assert status == 404, status result["reproduced"] = True print(clean(json.dumps(result, indent=2)), flush=True) except Exception as error: print(clean(f"Reproduction did not complete: {error}"), file=sys.stderr) sys.exit(1) finally: marker.unlink(missing_ok=True) for sid in active: try: status, _ = request("DELETE", f"/sandboxes/{sid}") if not (200 <= status < 300 or status == 404): print(clean(f"Cleanup needs attention for {sid}: HTTP {status}"), file=sys.stderr) except Exception as error: print(clean(f"Cleanup needs attention: {error}"), file=sys.stderr) ``` **Observed result** ```json { "image": "ubuntu:24.04", "fault": "one WAL write returns ENOSPC", "normal_control": "passed", "first_pause": { "http_status": 500, "error": { "code": 500, "message": "internal error: failed to persist paused sandbox state: paused sandbox store operation failed: persist paused sandbox record: put RocksDB value" } }, "injected_operations": [ "write ENOSPC <TEST_HOME>/persisted-sandboxes/records.db/000004.log" ], "state_after_failed_pause": "running", "missing_live_layers": [ { "runtime_config": "<TEST_HOME>/firecracker-work/agentenv-fc-<RUNTIME_ID>/overlaybd/image.json", "missing_layer": "<TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_2>/rootfs/snapshot.commit" } ], "retry_pause_without_fault": { "http_status": 500, "error": { "code": 500, "message": "sandbox <SANDBOX_ID> operation Pause failed: link or copy runtime overlaybd layer <TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_2>/rootfs/snapshot.commit -> <TEST_HOME>/persisted-sandboxes/artifacts/<SANDBOX_ID>/<ID_3>/rootfs/inherited-layers/0000/snapshot.commit: sparse copy command failed with status exit status: 1" } }, "instance_after_retry_http_status": 404, "reproduced": true } ``` **Relevant logs** ```text 2026-10-19T02:14:23.123338Z WARN pause_sandbox: agentenv::orchestrator::service: failed to persist paused sandbox state error=Store { operation: "persist paused sandbox record", source: put RocksDB value IO error: No space left on device: While appending to file: <TEST_HOME>/persisted-sandboxes/records.db/000004.log: No space left on device } sandbox_id=<SANDBOX_ID> 2026-10-19T02:14:23.205684Z WARN agentenv::sandbox::firecracker::overlaybd_snapshot: hard-link runtime overlaybd layer failed; falling back to sparse copy source=<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/snapshot.commit destination=<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/inherited-layers/0000/snapshot.commit error=No such file or directory (os error 2) cp: cannot stat '<TEST_HOME>/persisted-sandboxes/artifacts/<ID>/<ID>/rootfs/snapshot.commit': No such file or directory ``` </details> ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi added the

bug

label 2026-10-29 14:19:34 +00:00

bsakaguchi referenced this issue 2026-10-29 14:20:40 +00:00

[PR #98] fix(overlaybd): validate index bounds before allocation #177

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#98

No description provided.