kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #128] OverlayBD cold start of public image fails without Docker credential file #43

New issue

Closed

opened 2026-10-29 14:18:45 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:18:45 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Aug 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/128

Affected component

OverlayBD / image resolution

AgentENV version

Version or commit: Reproduced on 8f028b1; still present by source inspection at 990eb0daea1cac3f49b4dd083467a90f49f8278e
Local modifications: none in the affected code

Host environment

OS: Ubuntu 24.04.4 LTS
Kernel (uname -a): Linux 6.17.0-41-generic x86_64
Architecture: x86_64
Bare metal, VM, or cloud: KVM virtual machine, Docker Compose deployment
ublk support: ublk_drv loaded
CPU: Intel Xeon Gold 6240R
Memory: approximately 31 GiB
Storage/filesystem: Docker volumes; underlying filesystem not collected

Operation and configuration

# agentenv-a and agentenv-b do not contain:
#   ~/.docker/config.json
#   $DOCKER_CONFIG/config.json
#   /opt/overlaybd/cred.json

aenv start --cold overlaybd/ubuntu:22.04_obd

The generated OverlayBD global configuration contains:

{
  "credentialConfig": {
    "mode": "",
    "path": "",
    "timeout": 1
  }
}

Steps to reproduce

Start the standard Docker Compose deployment without mounting a Docker credential file into the runtime containers.
Authenticate the CLI against the gateway.
Run aenv start --cold overlaybd/ubuntu:22.04_obd.
Observe that image resolution succeeds but creation of the OverlayBD runtime device fails.

Expected behavior

The public OverlayBD-native image should be accessed anonymously when no Docker credential file is configured, and the sandbox should start successfully.

Actual behavior

Image resolution succeeds, but the ublk daemon fails before Firecracker starts:

create user image overlaybd runtime device: create overlaybd runtime device via daemon:
daemon: create overlaybd runtime device failed: materialize overlaybd runtime:
resolve overlaybd base virtual size: open overlaybd image config:
failed to open lower layer 0: read credential file:
No such file or directory (os error 2)

Logs and diagnostics

2026-07-31T12:00:27Z WARN regctl failed; retrying
  failed to request auth token: EOF

2026-07-31T12:00:34Z INFO source image is overlaybd-native;
  skipping blob download; runtime registryfs_v2 will fetch on demand

2026-07-31T12:00:34Z INFO image resolved to overlaybd config
  image=docker.io/overlaybd/ubuntu:22.04_obd

2026-07-31T12:00:35Z WARN failed to start sandbox
  failed to open lower layer 0: read credential file:
  No such file or directory (os error 2)

The initial Docker Hub EOF is retried successfully and is not the fatal error.

Reproduction frequency

Always in the tested Docker Compose configuration when no credential file is present.

Regression information

Last known good: unknown
First known bad: present in 8f028b1, the initial open-source release

Additional context

write_generated_overlaybd_global_config intentionally disables credential lookup when no Docker config is detected. However:

GlobalConfig::default() assigns the legacy credentialFilePath value /opt/overlaybd/cred.json.
Serde supplies that default because the generated configuration omits the legacy field.
normalize_compat_fields() sees an empty modern credential mode and a non-empty legacy path, then enables file credentials.
registryfs_v2 tries to read /opt/overlaybd/cred.json, which does not exist.

A permanent fix should ensure an explicitly disabled credentialConfig is not overridden by a defaulted legacy field. The generated configuration could explicitly set "credentialFilePath": "", or deserialization could distinguish an omitted legacy field from one explicitly configured.

A regression test should generate and reload the global config without Docker credentials, then assert that the effective credential mode remains disabled.

Temporary workaround: create /opt/overlaybd/cred.json containing:

{"auths":{}}

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction.
I removed credentials, tokens, private image references, and sensitive information.
This is not a security vulnerability.

Originally created by @yingdi-shan on GitHub (Aug 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/128 ### Affected component OverlayBD / image resolution ### AgentENV version - Version or commit: Reproduced on `8f028b1`; still present by source inspection at `990eb0daea1cac3f49b4dd083467a90f49f8278e` - Local modifications: none in the affected code ### Host environment - OS: Ubuntu 24.04.4 LTS - Kernel (`uname -a`): Linux 6.17.0-41-generic x86_64 - Architecture: x86_64 - Bare metal, VM, or cloud: KVM virtual machine, Docker Compose deployment - ublk support: `ublk_drv` loaded - CPU: Intel Xeon Gold 6240R - Memory: approximately 31 GiB - Storage/filesystem: Docker volumes; underlying filesystem not collected ### Operation and configuration ```shell # agentenv-a and agentenv-b do not contain: # ~/.docker/config.json # $DOCKER_CONFIG/config.json # /opt/overlaybd/cred.json aenv start --cold overlaybd/ubuntu:22.04_obd ``` The generated OverlayBD global configuration contains: ```json { "credentialConfig": { "mode": "", "path": "", "timeout": 1 } } ``` ### Steps to reproduce 1. Start the standard Docker Compose deployment without mounting a Docker credential file into the runtime containers. 2. Authenticate the CLI against the gateway. 3. Run `aenv start --cold overlaybd/ubuntu:22.04_obd`. 4. Observe that image resolution succeeds but creation of the OverlayBD runtime device fails. ### Expected behavior The public OverlayBD-native image should be accessed anonymously when no Docker credential file is configured, and the sandbox should start successfully. ### Actual behavior Image resolution succeeds, but the ublk daemon fails before Firecracker starts: ```text create user image overlaybd runtime device: create overlaybd runtime device via daemon: daemon: create overlaybd runtime device failed: materialize overlaybd runtime: resolve overlaybd base virtual size: open overlaybd image config: failed to open lower layer 0: read credential file: No such file or directory (os error 2) ``` ### Logs and diagnostics ```text 2026-07-31T12:00:27Z WARN regctl failed; retrying failed to request auth token: EOF 2026-07-31T12:00:34Z INFO source image is overlaybd-native; skipping blob download; runtime registryfs_v2 will fetch on demand 2026-07-31T12:00:34Z INFO image resolved to overlaybd config image=docker.io/overlaybd/ubuntu:22.04_obd 2026-07-31T12:00:35Z WARN failed to start sandbox failed to open lower layer 0: read credential file: No such file or directory (os error 2) ``` The initial Docker Hub EOF is retried successfully and is not the fatal error. ### Reproduction frequency Always in the tested Docker Compose configuration when no credential file is present. ### Regression information - Last known good: unknown - First known bad: present in `8f028b1`, the initial open-source release ### Additional context `write_generated_overlaybd_global_config` intentionally disables credential lookup when no Docker config is detected. However: 1. `GlobalConfig::default()` assigns the legacy `credentialFilePath` value `/opt/overlaybd/cred.json`. 2. Serde supplies that default because the generated configuration omits the legacy field. 3. `normalize_compat_fields()` sees an empty modern credential mode and a non-empty legacy path, then enables file credentials. 4. `registryfs_v2` tries to read `/opt/overlaybd/cred.json`, which does not exist. A permanent fix should ensure an explicitly disabled `credentialConfig` is not overridden by a defaulted legacy field. The generated configuration could explicitly set `"credentialFilePath": ""`, or deserialization could distinguish an omitted legacy field from one explicitly configured. A regression test should generate and reload the global config without Docker credentials, then assert that the effective credential mode remains disabled. Temporary workaround: create `/opt/overlaybd/cred.json` containing: ```json {"auths":{}} ``` ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction. - [x] I removed credentials, tokens, private image references, and sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:18:45 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 14:18:46 +00:00

Author

Owner

@ranxi2001 commented on GitHub (Aug 29, 2026):

I'd like to work on this.

I confirmed the reported configuration path is still present on main@9cedfe2: generated OverlayBD configs write an explicit empty credentialConfig but omit credentialFilePath, so deserialization supplies /opt/overlaybd/cred.json and compatibility normalization can restore file mode.

I plan to keep the fix generator-scoped: emit consistent modern and legacy credential fields for anonymous and file-backed modes, without changing the global legacy fallback behavior. I'll add regression coverage that reloads the generated rootfs, memory, conversion, and resize configs and verifies the effective credential mode and path.

Please let me know if this overlaps with ongoing work.

@ranxi2001 commented on GitHub (Aug 29, 2026): I'd like to work on this. I confirmed the reported configuration path is still present on `main@9cedfe2`: generated OverlayBD configs write an explicit empty `credentialConfig` but omit `credentialFilePath`, so deserialization supplies `/opt/overlaybd/cred.json` and compatibility normalization can restore `file` mode. I plan to keep the fix generator-scoped: emit consistent modern and legacy credential fields for anonymous and file-backed modes, without changing the global legacy fallback behavior. I'll add regression coverage that reloads the generated rootfs, memory, conversion, and resize configs and verifies the effective credential mode and path. Please let me know if this overlaps with ongoing work.

bsakaguchi referenced this issue

2026-10-29 14:20:04 +00:00

[PR #43] [MERGED] fix(overlaybd): reject truncated mapped reads #135