[PR #111] [CLOSED] feat(deploy): bootstrap node prerequisites and cap runtime footprint #187

New issue

Closed

opened 2026-10-29 14:20:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:47 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/111
Author: @cleverhu
Created: 8/3/2026
Status: ❌ Closed

Base: main ← Head: feat/k8s-daemonset-node-bootstrap

📝 Commits (1)

1b7fb3e feat(deploy): bootstrap node prerequisites and cap runtime footprint

📊 Changes

1 file changed (+52 additions, -0 deletions)

View changed files

📝 deploy/k8s/base/agentenv-daemonset.yaml (+52 -0)

📄 Description

What

Two additions to the node DaemonSet:

load-ublk and seed-deps initContainers that make a node ready before the server starts.
TOKIO_WORKER_THREADS and _RJEM_MALLOC_CONF on the runtime container to stop thread and arena counts from scaling with the host core count.

Why

Bootstrap. The server's startup check requires ublk_drv to be loaded in the host kernel and fails with an actionable error when it is not. Nothing in the manifest loaded it, so the DaemonSet only came up on nodes where an operator had done it by hand.

Separately, the workspace hostPath mounts over /workspace, which shadows the runtime assets the image bakes into /workspace/env (firecracker, the kernel, the overlaybd package, regctl). The server therefore re-downloaded all of them on every node on first start, despite them being present in the image it was just pulled from.

Footprint. Both the main runtime and the Firecracker pool runtime size their thread pools from the host core count. On a large node that is hundreds of tokio workers, each holding a jemalloc arena, which inflates the address space; the sandbox start path also spawns helper processes (iptables, ip) whose cost scales with it. jemalloc separately defaults to 4x the core count in arenas and holds dirty pages for 10 s, so resident memory stays high long after one-off work such as image conversion.

None. Deployment-manifest change with no code impact.

Scope and non-goals

Included: the initContainers, the host-root volume they need, and the two runtime environment variables.

Explicitly excluded, because they were part of the same internal change but are site-specific:

Image names pointing at a private registry, and imagePullPolicy: Always. The initContainers reuse agentenv-runtime:latest with IfNotPresent, matching the existing container.
A resources.requests.cpu value. It was 8 internally, which would make the DaemonSet unschedulable on small nodes; it belongs in an overlay, not the base.
Snapshot backend, registry and pool-watermark changes from the same internal branch. Those are deployment policy and are not proposed here.

Design and behavior changes

load-ublk runs privileged and chroot /host modprobe ublk_drv, so modprobe resolves modules for the node's running kernel rather than the container image, then lists /dev/ublk-control so a failure is visible in the init logs rather than surfacing later as a confusing server error. It mounts the host root read-only. It requires a new host-root hostPath volume.

seed-deps copies /workspace/env/. from the image into the hostPath with cp -a --update=none, so existing host state is never overwritten — only missing files are filled in. It is not privileged.

Both are initContainers, so the runtime container starts only after they succeed. On a node missing ublk_drv entirely the pod now fails in init with a clear message instead of crash-looping the server.

TOKIO_WORKER_THREADS=32 caps the tokio pool. _RJEM_MALLOC_CONF=narenas:8,dirty_decay_ms:1000,muzzy_decay_ms:0,background_thread:true caps arenas and shortens decay so pages are returned promptly.

Compatibility and operations

Public API or generated protocol: N/A.
Configuration or defaults: no AgentENV config keys change. Two environment variables are added to the manifest; both are overridable by an overlay.
Snapshot manifest, artifact layout, or storage format: N/A.
Upgrade and rollback: rolling update replaces pods. seed-deps is idempotent and never overwrites, so rollback to the previous manifest leaves a working /var/lib/aenv. The added host-root volume is removed with the manifest.
Host requirements, permissions, ports, or dependencies: the DaemonSet already runs privileged and mounts /dev. This adds a read-only mount of the host root for the init step only.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

$ python3 - <<'EOF'
import yaml
d = yaml.safe_load(open('deploy/k8s/base/agentenv-daemonset.yaml'))
spec = d['spec']['template']['spec']
...verify every volumeMount resolves to a declared volume...
EOF
initContainers: ['load-ublk', 'seed-deps']
volumes: ['workspace', 'agentenv-config', 'host-dev', 'host-root']
mounts: {'load-ublk': ['host-root'], 'seed-deps': ['workspace'], 'agentenv': ['workspace', 'agentenv-config', 'host-dev']}
YAML OK, all mounts resolve

Skipped checks and reasons:

No Rust target applies; the PR touches one YAML file.
The equivalent manifest has been running on an internal multi-node cluster, but not this exact sanitized version, so I am not claiming a cluster apply for this diff. kubectl apply --dry-run=server on a real cluster is the check I could not run here.
No footprint measurement is attached. The tokio and jemalloc values were chosen from the reasoning above and observed to keep RSS down on a 128-core host, but I do not have a clean before/after RSS series to publish, so please treat those two variables as a proposal rather than a measured result. I am happy to drop them into a separate PR if maintainers would rather land the bootstrap fix alone.

Risks and reviewer notes

Mounting the host root, even read-only, is the change that deserves the most scrutiny. It is confined to the load-ublk initContainer and is the standard way to modprobe against the node kernel; the alternative is requiring operators to pre-load the module out of band, which is what this replaces.

TOKIO_WORKER_THREADS=32 is a fixed number rather than a fraction of the node. On a node with fewer than 32 cores it oversubscribes slightly; on a very large node it may be conservative. A percentage would be better but is not expressible in a plain manifest.

cp -a --update=none requires a coreutils new enough to support --update=none; the runtime image's Debian base has it.

Most important file: deploy/k8s/base/agentenv-daemonset.yaml.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/111 **Author:** [@cleverhu](https://github.com/cleverhu) **Created:** 8/3/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/k8s-daemonset-node-bootstrap` --- ### 📝 Commits (1) - [`1b7fb3e`](https://github.com/kvcache-ai/AgentENV/commit/1b7fb3e9ec6542b093c8beda34c3feee610b992b) feat(deploy): bootstrap node prerequisites and cap runtime footprint ### 📊 Changes **1 file changed** (+52 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `deploy/k8s/base/agentenv-daemonset.yaml` (+52 -0) </details> ### 📄 Description ## What Two additions to the node DaemonSet: 1. `load-ublk` and `seed-deps` initContainers that make a node ready before the server starts. 2. `TOKIO_WORKER_THREADS` and `_RJEM_MALLOC_CONF` on the runtime container to stop thread and arena counts from scaling with the host core count. ## Why **Bootstrap.** The server's startup check requires `ublk_drv` to be loaded in the host kernel and fails with an actionable error when it is not. Nothing in the manifest loaded it, so the DaemonSet only came up on nodes where an operator had done it by hand. Separately, the `workspace` hostPath mounts over `/workspace`, which shadows the runtime assets the image bakes into `/workspace/env` (firecracker, the kernel, the overlaybd package, regctl). The server therefore re-downloaded all of them on every node on first start, despite them being present in the image it was just pulled from. **Footprint.** Both the main runtime and the Firecracker pool runtime size their thread pools from the host core count. On a large node that is hundreds of tokio workers, each holding a jemalloc arena, which inflates the address space; the sandbox start path also spawns helper processes (`iptables`, `ip`) whose cost scales with it. jemalloc separately defaults to 4x the core count in arenas and holds dirty pages for 10 s, so resident memory stays high long after one-off work such as image conversion. ## Related issue None. Deployment-manifest change with no code impact. ## Scope and non-goals Included: the initContainers, the host-root volume they need, and the two runtime environment variables. Explicitly excluded, because they were part of the same internal change but are site-specific: - Image names pointing at a private registry, and `imagePullPolicy: Always`. The initContainers reuse `agentenv-runtime:latest` with `IfNotPresent`, matching the existing container. - A `resources.requests.cpu` value. It was `8` internally, which would make the DaemonSet unschedulable on small nodes; it belongs in an overlay, not the base. - Snapshot backend, registry and pool-watermark changes from the same internal branch. Those are deployment policy and are not proposed here. ## Design and behavior changes `load-ublk` runs privileged and `chroot /host modprobe ublk_drv`, so modprobe resolves modules for the node's running kernel rather than the container image, then lists `/dev/ublk-control` so a failure is visible in the init logs rather than surfacing later as a confusing server error. It mounts the host root read-only. It requires a new `host-root` hostPath volume. `seed-deps` copies `/workspace/env/.` from the image into the hostPath with `cp -a --update=none`, so existing host state is never overwritten — only missing files are filled in. It is not privileged. Both are initContainers, so the runtime container starts only after they succeed. On a node missing `ublk_drv` entirely the pod now fails in init with a clear message instead of crash-looping the server. `TOKIO_WORKER_THREADS=32` caps the tokio pool. `_RJEM_MALLOC_CONF=narenas:8,dirty_decay_ms:1000,muzzy_decay_ms:0,background_thread:true` caps arenas and shortens decay so pages are returned promptly. ## Compatibility and operations - Public API or generated protocol: N/A. - Configuration or defaults: no AgentENV config keys change. Two environment variables are added to the manifest; both are overridable by an overlay. - Snapshot manifest, artifact layout, or storage format: N/A. - Upgrade and rollback: rolling update replaces pods. `seed-deps` is idempotent and never overwrites, so rollback to the previous manifest leaves a working `/var/lib/aenv`. The added `host-root` volume is removed with the manifest. - Host requirements, permissions, ports, or dependencies: the DaemonSet already runs privileged and mounts `/dev`. This adds a **read-only** mount of the host root for the init step only. ## Validation - [ ] `make fmt` - [ ] `make clippy` - [ ] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [ ] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results: ```text $ python3 - <<'EOF' import yaml d = yaml.safe_load(open('deploy/k8s/base/agentenv-daemonset.yaml')) spec = d['spec']['template']['spec'] ...verify every volumeMount resolves to a declared volume... EOF initContainers: ['load-ublk', 'seed-deps'] volumes: ['workspace', 'agentenv-config', 'host-dev', 'host-root'] mounts: {'load-ublk': ['host-root'], 'seed-deps': ['workspace'], 'agentenv': ['workspace', 'agentenv-config', 'host-dev']} YAML OK, all mounts resolve ``` Skipped checks and reasons: - No Rust target applies; the PR touches one YAML file. - The equivalent manifest has been running on an internal multi-node cluster, but not this exact sanitized version, so I am not claiming a cluster apply for this diff. `kubectl apply --dry-run=server` on a real cluster is the check I could not run here. - No footprint measurement is attached. The tokio and jemalloc values were chosen from the reasoning above and observed to keep RSS down on a 128-core host, but I do not have a clean before/after RSS series to publish, so please treat those two variables as a proposal rather than a measured result. I am happy to drop them into a separate PR if maintainers would rather land the bootstrap fix alone. ## Risks and reviewer notes Mounting the host root, even read-only, is the change that deserves the most scrutiny. It is confined to the `load-ublk` initContainer and is the standard way to `modprobe` against the node kernel; the alternative is requiring operators to pre-load the module out of band, which is what this replaces. `TOKIO_WORKER_THREADS=32` is a fixed number rather than a fraction of the node. On a node with fewer than 32 cores it oversubscribes slightly; on a very large node it may be conservative. A percentage would be better but is not expressible in a plain manifest. `cp -a --update=none` requires a coreutils new enough to support `--update=none`; the runtime image's Debian base has it. Most important file: `deploy/k8s/base/agentenv-daemonset.yaml`. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:47 +00:00