[GH-ISSUE #15] Add locality-based scheduling using node cache awareness #9

New issue

Open

opened 2026-10-29 14:18:18 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:18:18 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/15

Problem

AgentENV should add a locality-based scheduling strategy that places new sandboxes on nodes that already hold the most relevant cache for the requested workload.

Today the distributed scheduler can choose strategies such as round-robin, but it does not account for per-node caches. That can place a sandbox on a cold node even when another node already has useful local artifacts, such as overlaybd layers, converted image commits, memory snapshot commits, or fixed snapshot artifacts.

The scheduler should prefer warm nodes when they have enough available resources. It should only place the sandbox on colder or new nodes when the warmer candidates lack sufficient CPU, memory, disk, or other required capacity.

Proposal

Add a locality-aware scheduling strategy, for example locality or cache_aware, that:

Filters candidate nodes by hard resource availability.
Scores feasible nodes by cache coverage for the requested template, snapshot, or userImage.
Chooses the feasible node with the highest cache-locality score.
Falls back to colder or new nodes only when warm nodes do not have enough resources, or when no useful cache metadata is known.

The cache score can consider local availability of rootfs/image layers, overlaybd commits, memory snapshot commits, fixed snapshot artifacts, and related P2P-published artifacts where applicable.

Originally created by @yingdi-shan on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/15 ## Problem AgentENV should add a locality-based scheduling strategy that places new sandboxes on nodes that already hold the most relevant cache for the requested workload. Today the distributed scheduler can choose strategies such as round-robin, but it does not account for per-node caches. That can place a sandbox on a cold node even when another node already has useful local artifacts, such as overlaybd layers, converted image commits, memory snapshot commits, or fixed snapshot artifacts. The scheduler should prefer warm nodes when they have enough available resources. It should only place the sandbox on colder or new nodes when the warmer candidates lack sufficient CPU, memory, disk, or other required capacity. ## Proposal Add a locality-aware scheduling strategy, for example `locality` or `cache_aware`, that: - Filters candidate nodes by hard resource availability. - Scores feasible nodes by cache coverage for the requested template, snapshot, or `userImage`. - Chooses the feasible node with the highest cache-locality score. - Falls back to colder or new nodes only when warm nodes do not have enough resources, or when no useful cache metadata is known. The cache score can consider local availability of rootfs/image layers, overlaybd commits, memory snapshot commits, fixed snapshot artifacts, and related P2P-published artifacts where applicable.

bsakaguchi added the

enhancement

label

2026-10-29 14:18:18 +00:00

bsakaguchi commented

2026-10-29 14:18:19 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Jul 29, 2026):

We have attempted to map sandbox with cache locality awareness. However, we find that a popular image can cause severe load imbalance and eventually crash that node due to OOM. Heartbeat-based resource filtering may react too late during a burst.

A simpler alternative is resource-bounded grouped round-robin.

Each group belongs to one image and one node. Requests for the same image fill its current open group. When that group reaches any configured resource limit, it is closed and the next group is assigned to the next eligible node using a global round-robin cursor.

For example, suppose nodes are selected in the order A -> B -> C, and each group is limited to 2 sandboxes.

Placement could look like:

ubuntu group 1 -> node A
python group 1 -> node B
ubuntu group 2 -> node C
python group 2 -> node A

Requests may arrive in any order. Ubuntu requests continue filling the current Ubuntu group, while Python requests fill a separate Python group. Different images are never mixed within a group.

The scheduling process is:

Find the current open group for the requested image.
Check whether adding the VM would exceed its sandbox, CPU, or memory budget.
If it fits and the node remains eligible, place it in that group.
Otherwise, create a new one on the next eligible node.
Count pending placements immediately so concurrent requests see the reserved group capacity.
Nodes that are full, under memory pressure, starting too many VMs, or reporting stale telemetry are skipped. Runtime pressure can also close a group early.

This keeps several same-image sandboxes together for cache reuse while progressively spreading popular images across the cluster. Configured CPU and memory act as relative weights rather than predictions of actual usage, so the strategy remains simple while handling different VM sizes.

@yingdi-shan commented on GitHub (Jul 29, 2026): We have attempted to map sandbox with cache locality awareness. However, we find that a popular image can cause severe load imbalance and eventually crash that node due to OOM. Heartbeat-based resource filtering may react too late during a burst. A simpler alternative is resource-bounded grouped round-robin. Each group belongs to one image and one node. Requests for the same image fill its current open group. When that group reaches any configured resource limit, it is closed and the next group is assigned to the next eligible node using a global round-robin cursor. For example, suppose nodes are selected in the order A -> B -> C, and each group is limited to 2 sandboxes. Placement could look like: ubuntu group 1 -> node A python group 1 -> node B ubuntu group 2 -> node C python group 2 -> node A Requests may arrive in any order. Ubuntu requests continue filling the current Ubuntu group, while Python requests fill a separate Python group. Different images are never mixed within a group. The scheduling process is: Find the current open group for the requested image. Check whether adding the VM would exceed its sandbox, CPU, or memory budget. If it fits and the node remains eligible, place it in that group. Otherwise, create a new one on the next eligible node. Count pending placements immediately so concurrent requests see the reserved group capacity. Nodes that are full, under memory pressure, starting too many VMs, or reporting stale telemetry are skipped. Runtime pressure can also close a group early. This keeps several same-image sandboxes together for cache reuse while progressively spreading popular images across the cluster. Configured CPU and memory act as relative weights rather than predictions of actual usage, so the strategy remains simple while handling different VM sizes.

bsakaguchi referenced this issue

2026-10-29 14:18:21 +00:00

[GH-ISSUE #31] Support attached drives when creating a sandbox from a template #14

bsakaguchi referenced this issue

2026-10-29 14:18:22 +00:00

[GH-ISSUE #31] Support attached drives when creating a sandbox from a template #14

bsakaguchi referenced this issue

2026-10-29 14:18:22 +00:00

[GH-ISSUE #31] Support attached drives when creating a sandbox from a template #14