[PR #196] Add heartbeat discovery and provider-neutral fleet planning #243

New issue

Open

opened 2026-10-29 14:21:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:25 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/196
Author: @nikhilsimha
Created: 8/19/2026
Status: 🔄 Open

Base: main ← Head: nikhil/heartbeat-fleet-planning

📝 Commits (1)

f2e0cce feat(services): add heartbeat fleet planning

📊 Changes

22 files changed (+2060 additions, -115 deletions)

View changed files

📝 docs/src/configuration/env-vars.md (+3 -0)
📝 docs/src/configuration/reference.md (+4 -0)
📝 docs/src/internals/architecture.md (+6 -3)
📝 docs/src/internals/services.md (+6 -3)
📝 services/README.md (+20 -1)
📝 services/api/proto/scheduler.pb.go (+473 -42)
📝 services/api/proto/scheduler.proto (+44 -0)
📝 services/api/proto/scheduler_grpc.pb.go (+129 -15)
➕ services/gateway/internal/fleet_admin.go (+147 -0)
➕ services/gateway/internal/fleet_admin_test.go (+86 -0)
📝 services/gateway/internal/server.go (+45 -17)
📝 services/gateway/internal/server_test.go (+24 -0)
📝 services/scheduler/cmd/main.go (+33 -7)
➕ services/scheduler/internal/fleet_plan.go (+311 -0)
➕ services/scheduler/internal/fleet_plan_test.go (+192 -0)
➕ services/scheduler/internal/heartbeat_discovery_test.go (+126 -0)
📝 services/scheduler/internal/node_registry.go (+94 -8)
📝 services/scheduler/internal/service.go (+109 -6)
📝 services/shared/config/config.go (+128 -12)
📝 services/shared/config/config_test.go (+42 -0)

...and 2 more files

📄 Description

Why this change

AgentENV knows which sandboxes exist, which nodes are full, and whether a node is safe to remove. A cloud autoscaler only sees VM-level signals, so it can remove a node that still owns a running or paused sandbox. A static node list also means every node addition or replacement requires a Scheduler config update and restart.

This change lets AgentENV make the sandbox-aware capacity and drain decisions while a small external executor creates and deletes VMs. We get automatic scale-out, safe scale-in, and node replacement without putting cloud credentials or cloud-specific code inside AgentENV.

High-level approach

Each runtime node sends Scheduler a periodic heartbeat containing its private endpoint and a shared registration token.
Scheduler combines those heartbeats with sandbox demand and memory state to decide how many nodes are needed and which empty node is safe to drain.
A separate infrastructure executor asks AgentENV for the plan, lists the VMs from its cloud provider, and applies one change at a time.
When capacity is booting, new sandbox requests wait briefly and retry. When scaling in, AgentENV requires an empty grace period, then a drain grace period, and verifies the exact node generation before allowing deletion.

AgentENV never receives cloud credentials, and the infrastructure executor never guesses whether a sandbox node is safe to remove.

Summary

add authenticated heartbeat-based node discovery with runtime-advertised endpoints
add provider-neutral desired-capacity, cordon, uncordon, and exact-generation delete plans
expose protected Gateway fleet endpoints and wait for booting capacity on create
include node endpoint and registration token in runtime heartbeats

Safety

infrastructure credentials stay outside AgentENV
paused, starting, and running sandboxes block node deletion
scale-in requires empty and drain grace periods and an exact service generation
demand and memory pressure use fixed targets to prevent runaway scale-out

Validation

make -C services test
cargo fmt --all -- --check
cargo clippy -p agentenv --lib -- -D warnings
cargo test -p agentenv observability::reporter --lib
full Rust library run: 740 passed, 5 pre-existing overlay-disk failures reproduced on untouched origin/main, 4 ignored

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/196 **Author:** [@nikhilsimha](https://github.com/nikhilsimha) **Created:** 8/19/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `nikhil/heartbeat-fleet-planning` --- ### 📝 Commits (1) - [`f2e0cce`](https://github.com/kvcache-ai/AgentENV/commit/f2e0cce03ce55ada6f77c5570ba0c254d7ad22cb) feat(services): add heartbeat fleet planning ### 📊 Changes **22 files changed** (+2060 additions, -115 deletions) <details> <summary>View changed files</summary> 📝 `docs/src/configuration/env-vars.md` (+3 -0) 📝 `docs/src/configuration/reference.md` (+4 -0) 📝 `docs/src/internals/architecture.md` (+6 -3) 📝 `docs/src/internals/services.md` (+6 -3) 📝 `services/README.md` (+20 -1) 📝 `services/api/proto/scheduler.pb.go` (+473 -42) 📝 `services/api/proto/scheduler.proto` (+44 -0) 📝 `services/api/proto/scheduler_grpc.pb.go` (+129 -15) ➕ `services/gateway/internal/fleet_admin.go` (+147 -0) ➕ `services/gateway/internal/fleet_admin_test.go` (+86 -0) 📝 `services/gateway/internal/server.go` (+45 -17) 📝 `services/gateway/internal/server_test.go` (+24 -0) 📝 `services/scheduler/cmd/main.go` (+33 -7) ➕ `services/scheduler/internal/fleet_plan.go` (+311 -0) ➕ `services/scheduler/internal/fleet_plan_test.go` (+192 -0) ➕ `services/scheduler/internal/heartbeat_discovery_test.go` (+126 -0) 📝 `services/scheduler/internal/node_registry.go` (+94 -8) 📝 `services/scheduler/internal/service.go` (+109 -6) 📝 `services/shared/config/config.go` (+128 -12) 📝 `services/shared/config/config_test.go` (+42 -0) _...and 2 more files_ </details> ### 📄 Description ## Why this change AgentENV knows which sandboxes exist, which nodes are full, and whether a node is safe to remove. A cloud autoscaler only sees VM-level signals, so it can remove a node that still owns a running or paused sandbox. A static node list also means every node addition or replacement requires a Scheduler config update and restart. This change lets AgentENV make the sandbox-aware capacity and drain decisions while a small external executor creates and deletes VMs. We get automatic scale-out, safe scale-in, and node replacement without putting cloud credentials or cloud-specific code inside AgentENV. ## High-level approach 1. Each runtime node sends Scheduler a periodic heartbeat containing its private endpoint and a shared registration token. 2. Scheduler combines those heartbeats with sandbox demand and memory state to decide how many nodes are needed and which empty node is safe to drain. 3. A separate infrastructure executor asks AgentENV for the plan, lists the VMs from its cloud provider, and applies one change at a time. 4. When capacity is booting, new sandbox requests wait briefly and retry. When scaling in, AgentENV requires an empty grace period, then a drain grace period, and verifies the exact node generation before allowing deletion. AgentENV never receives cloud credentials, and the infrastructure executor never guesses whether a sandbox node is safe to remove. ## Summary - add authenticated heartbeat-based node discovery with runtime-advertised endpoints - add provider-neutral desired-capacity, cordon, uncordon, and exact-generation delete plans - expose protected Gateway fleet endpoints and wait for booting capacity on create - include node endpoint and registration token in runtime heartbeats ## Safety - infrastructure credentials stay outside AgentENV - paused, starting, and running sandboxes block node deletion - scale-in requires empty and drain grace periods and an exact service generation - demand and memory pressure use fixed targets to prevent runaway scale-out ## Validation - `make -C services test` - `cargo fmt --all -- --check` - `cargo clippy -p agentenv --lib -- -D warnings` - `cargo test -p agentenv observability::reporter --lib` - full Rust library run: 740 passed, 5 pre-existing overlay-disk failures reproduced on untouched origin/main, 4 ignored --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 14:21:25 +00:00

bsakaguchi referenced this issue

2026-10-29 14:21:49 +00:00

[PR #243] [MERGED] chore(release): bump version to v0.2.0 #285