[GH-ISSUE #262] Add offline workload replay for scheduler strategy evaluation

bsakaguchi commented

2026-10-29 14:19:13 +00:00

Owner

Originally created by @shudorcl on GitHub (Sep 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/262

Affected component

Gateway / scheduler

Problem statement

Following up on the review of #68: we need a way to compare scheduling strategies against workload traces before deciding whether to merge grouped round-robin. I offered to help in that PR and am opening this issue to settle the initial scope.

The concern raised in #15 is a good starting point: grouping a popular image may improve cache reuse, but a burst can overload a node before the next heartbeat arrives. I'd like the tool to make that tradeoff visible.

Use case

Replay the same workload against round-robin and a candidate strategy. Start with repeated images/templates, bursts of one popular workload, mixed CPU/memory sizes, and enough distinct workloads to cause cache eviction.

Current behavior and workarounds

Existing strategy tests check placement rules. They don't show how load and cache reuse evolve across a workload. Replaying against a live cluster would help, but requires more setup and is harder to make repeatable.

Desired behavior

For a first version, I'd aim for:

An offline run with the same trace, configuration, and seed producing the same placements and report.
Reuse of the actual scheduler strategies and resource filter, with round-robin as the baseline and a way to evaluate #68 through the same runner.
Request arrivals and completions, plus delayed heartbeat observations. The model should expose over-allocation during bursts.
Reports of per-node CPU/memory allocation, capacity exceedance, unplaced requests, and modeled cache hits/misses, with their definitions documented.
Small synthetic traces with expected results we can check by hand, followed by representative traces when available.

Proposed approach

I'd start with a small event-driven Go tool under services/, using a fixed set of healthy nodes. A trace would contain arrival time, request type, image/template identity, requested resources, and lifetime. The strategy should only see hints it would receive in production, even when the trace contains extra information for accounting.

For caching, an initially empty, bounded LRU cache of workload identities seems enough to start discussing reuse. It won't model shared layers or download time. Likewise, requested CPU/memory tells us about allocation, not actual utilization; cache misses won't tell us cold-start latency.

Before writing the prototype, I'd like to confirm:

Is anyone already working on this, and would an offline simulator be a useful first step?
Are there representative traces we can use, or should we start with synthetic ones?
Is this resource/cache model sufficient initially? What scenarios and results would you want to see to revisit #68?

Compatibility and operational impact

This would be an optional development tool requiring Go, with its own trace and model inputs. No production API/configuration, protobuf, snapshot/storage format, or deployment changes; no KVM or privileged host access. There would be no migration to apply or roll back.

Alternatives considered

Live-cluster replay would give us actual startup timings and cache behavior, at the cost of runtime infrastructure. Unit tests remain useful for strategy correctness but don't cover workload-level comparisons. If a framework is already underway, I'd be happy to contribute a smaller piece of that instead.

Non-goals

The first version would leave out production trace collection, new scheduling policies, node failures/HA, and artifact transfer/P2P simulation. Synthetic results would help evaluate the model, not establish a production performance improvement.

Contribution

I can implement it once we've agreed on the scope. I'll wait for feedback here before starting the prototype.

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @shudorcl on GitHub (Sep 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/262 ### Affected component Gateway / scheduler ### Problem statement Following up on the review of #68: we need a way to compare scheduling strategies against workload traces before deciding whether to merge grouped round-robin. I offered to help in that PR and am opening this issue to settle the initial scope. The concern raised in #15 is a good starting point: grouping a popular image may improve cache reuse, but a burst can overload a node before the next heartbeat arrives. I'd like the tool to make that tradeoff visible. ### Use case Replay the same workload against round-robin and a candidate strategy. Start with repeated images/templates, bursts of one popular workload, mixed CPU/memory sizes, and enough distinct workloads to cause cache eviction. ### Current behavior and workarounds Existing strategy tests check placement rules. They don't show how load and cache reuse evolve across a workload. Replaying against a live cluster would help, but requires more setup and is harder to make repeatable. ### Desired behavior For a first version, I'd aim for: - An offline run with the same trace, configuration, and seed producing the same placements and report. - Reuse of the actual scheduler strategies and resource filter, with round-robin as the baseline and a way to evaluate #68 through the same runner. - Request arrivals and completions, plus delayed heartbeat observations. The model should expose over-allocation during bursts. - Reports of per-node CPU/memory allocation, capacity exceedance, unplaced requests, and modeled cache hits/misses, with their definitions documented. - Small synthetic traces with expected results we can check by hand, followed by representative traces when available. ### Proposed approach I'd start with a small event-driven Go tool under `services/`, using a fixed set of healthy nodes. A trace would contain arrival time, request type, image/template identity, requested resources, and lifetime. The strategy should only see hints it would receive in production, even when the trace contains extra information for accounting. For caching, an initially empty, bounded LRU cache of workload identities seems enough to start discussing reuse. It won't model shared layers or download time. Likewise, requested CPU/memory tells us about allocation, not actual utilization; cache misses won't tell us cold-start latency. Before writing the prototype, I'd like to confirm: 1. Is anyone already working on this, and would an offline simulator be a useful first step? 2. Are there representative traces we can use, or should we start with synthetic ones? 3. Is this resource/cache model sufficient initially? What scenarios and results would you want to see to revisit #68? ### Compatibility and operational impact This would be an optional development tool requiring Go, with its own trace and model inputs. No production API/configuration, protobuf, snapshot/storage format, or deployment changes; no KVM or privileged host access. There would be no migration to apply or roll back. ### Alternatives considered Live-cluster replay would give us actual startup timings and cache behavior, at the cost of runtime infrastructure. Unit tests remain useful for strategy correctness but don't cover workload-level comparisons. If a framework is already underway, I'd be happy to contribute a smaller piece of that instead. ### Non-goals The first version would leave out production trace collection, new scheduling policies, node failures/HA, and artifact transfer/P2P simulation. Synthetic results would help evaluate the model, not establish a production performance improvement. ### Contribution I can implement it once we've agreed on the scope. I'll wait for feedback here before starting the prototype. ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi commented

2026-10-29 14:19:14 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Sep 9, 2026):

Thanks for following up and offering to help. A lightweight offline simulator sounds like a good first step.

Let’s keep the initial version simple: start with a trace based on the Terminal-Bench 2.0 workload described in our docs, compare round-robin with grouped round-robin, and report CPU/memory usage and modeled cache hits/misses.

The most important part is collecting real CPU, memory, and I/O usage every second. For memory, we need RSS and SHM, accounting for memory shared across sandboxes.

I/O is especially important for estimating cache hit ratios, but it’s difficult to measure. A good starting point would be to associate I/O traces with their storage layers, recording how much I/O goes to each layer. That would let us model cache hits more precisely and give us a better basis for evaluating #68.

@yingdi-shan commented on GitHub (Sep 9, 2026): Thanks for following up and offering to help. A lightweight offline simulator sounds like a good first step. Let’s keep the initial version simple: start with a trace based on the Terminal-Bench 2.0 workload described in [our docs](https://github.com/radixark/miles/blob/main/examples/experimental/agentenv/README.md), compare round-robin with grouped round-robin, and report CPU/memory usage and modeled cache hits/misses. The most important part is collecting real CPU, memory, and I/O usage every second. For memory, we need RSS and SHM, accounting for memory shared across sandboxes. I/O is especially important for estimating cache hit ratios, but it’s difficult to measure. A good starting point would be to associate I/O traces with their storage layers, recording how much I/O goes to each layer. That would let us model cache hits more precisely and give us a better basis for evaluating #68.

bsakaguchi commented

2026-10-29 14:19:15 +00:00

Author

Owner

@shudorcl commented on GitHub (Sep 16, 2026):

@yingdi-shan Thanks for clarifying. Do you have existing traces from the Terminal-Bench 2.0 runs described in the Miles docs that could be shared? If traces aren't available yet, I would like to implement an MVP using synthetic traces as the first step.

@shudorcl commented on GitHub (Sep 16, 2026): @yingdi-shan Thanks for clarifying. Do you have existing traces from the Terminal-Bench 2.0 runs described in the Miles docs that could be shared? If traces aren't available yet, I would like to implement an MVP using synthetic traces as the first step.

bsakaguchi commented

2026-10-29 14:19:15 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Sep 16, 2026):

You can refer to issue #281 and #282. They use traces from the official solutions of terminal-bench 2.1.

@yingdi-shan commented on GitHub (Sep 16, 2026): You can refer to issue #281 and #282. They use traces from the official solutions of terminal-bench 2.1.

bsakaguchi commented

2026-10-29 14:19:15 +00:00

Author

Owner

@NickNYU commented on GitHub (Sep 22, 2026):

Hi folks,

Thanks for writing this issue up so clearly.

I'd like to help with this if you're open to it. From the last message it seems like we were waiting for the prototype, and @yingdi-shan has since pointed to #281/#282 for that. Have you had a chance to start?

If you're already working on it, I'd be happy to take one part off your plate — for example the trace collection side: a small tool that samples RSS/SHM and per-layer I/O once per second during a replay run and writes out the trace file the simulator reads. @yingdi-shan called that the most important part, so it feels worth getting right early.

If you haven't started yet, no worries at all — I could put together a first rough version of the simulator, and we can split the work from there.

Either way, it might help to agree on the trace format first (arrival time, request type, image/template, requested CPU/memory, lifetime, plus the per-second samples), so the collector and the simulator can be built in parallel.

A bit of background on me: I worked on the aenv CLI dynamic completion (#163, #250) and the benchmark/e2e dependency split (#39).

Looking forward to your thoughts! Thank you

@NickNYU commented on GitHub (Sep 22, 2026): Hi folks, Thanks for writing this issue up so clearly. I'd like to help with this if you're open to it. From the last message it seems like we were waiting for the prototype, and @yingdi-shan has since pointed to #281/#282 for that. Have you had a chance to start? If you're already working on it, I'd be happy to take one part off your plate — for example the trace collection side: a small tool that samples RSS/SHM and per-layer I/O once per second during a replay run and writes out the trace file the simulator reads. @yingdi-shan called that the most important part, so it feels worth getting right early. If you haven't started yet, no worries at all — I could put together a first rough version of the simulator, and we can split the work from there. Either way, it might help to agree on the trace format first (arrival time, request type, image/template, requested CPU/memory, lifetime, plus the per-second samples), so the collector and the simulator can be built in parallel. A bit of background on me: I worked on the aenv CLI dynamic completion (#163, #250) and the benchmark/e2e dependency split (#39). Looking forward to your thoughts! Thank you

bsakaguchi commented

2026-10-29 14:19:15 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Sep 23, 2026):

Hi folks,

Thanks for writing this issue up so clearly.

I'd like to help with this if you're open to it. From the last message it seems like we were waiting for the prototype, and @yingdi-shan has since pointed to #281/#282 for that. Have you had a chance to start?

If you're already working on it, I'd be happy to take one part off your plate — for example the trace collection side: a small tool that samples RSS/SHM and per-layer I/O once per second during a replay run and writes out the trace file the simulator reads. @yingdi-shan called that the most important part, so it feels worth getting right early.

If you haven't started yet, no worries at all — I could put together a first rough version of the simulator, and we can split the work from there.

Either way, it might help to agree on the trace format first (arrival time, request type, image/template, requested CPU/memory, lifetime, plus the per-second samples), so the collector and the simulator can be built in parallel.

A bit of background on me: I worked on the aenv CLI dynamic completion (#163, #250) and the benchmark/e2e dependency split (#39).

Looking forward to your thoughts! Thank you

Thanks a lot for the offer and the careful write-up! We're already working on this and it's nearly done, though it may be a little while before we open the PR and share the results.

@yingdi-shan commented on GitHub (Sep 23, 2026): > Hi folks, > > Thanks for writing this issue up so clearly. > > I'd like to help with this if you're open to it. From the last message it seems like we were waiting for the prototype, and [@yingdi-shan](https://github.com/yingdi-shan) has since pointed to [#281](https://github.com/kvcache-ai/AgentENV/issues/281)/[#282](https://github.com/kvcache-ai/AgentENV/issues/282) for that. Have you had a chance to start? > > If you're already working on it, I'd be happy to take one part off your plate — for example the trace collection side: a small tool that samples RSS/SHM and per-layer I/O once per second during a replay run and writes out the trace file the simulator reads. [@yingdi-shan](https://github.com/yingdi-shan) called that the most important part, so it feels worth getting right early. > > If you haven't started yet, no worries at all — I could put together a first rough version of the simulator, and we can split the work from there. > > Either way, it might help to agree on the trace format first (arrival time, request type, image/template, requested CPU/memory, lifetime, plus the per-second samples), so the collector and the simulator can be built in parallel. > > A bit of background on me: I worked on the aenv CLI dynamic completion ([#163](https://github.com/kvcache-ai/AgentENV/pull/163), [#250](https://github.com/kvcache-ai/AgentENV/pull/250)) and the benchmark/e2e dependency split ([#39](https://github.com/kvcache-ai/AgentENV/pull/39)). > > Looking forward to your thoughts! Thank you Thanks a lot for the offer and the careful write-up! We're already working on this and it's nearly done, though it may be a little while before we open the PR and share the results.

bsakaguchi commented

2026-10-29 14:19:16 +00:00

Author

Owner

@NickNYU commented on GitHub (Sep 23, 2026):

Hi @yingdi-shan,

Good to know it's nearly done — looking forward to the PR! When it's up, I'd be happy to help review it, especially the trace format and the cache-hit accounting.

Thanks for the quick reply!

@NickNYU commented on GitHub (Sep 23, 2026): Hi @yingdi-shan, Good to know it's nearly done — looking forward to the PR! When it's up, I'd be happy to help review it, especially the trace format and the cache-hit accounting. Thanks for the quick reply!

bsakaguchi commented

2026-10-29 14:19:16 +00:00

Author

Owner

@shudorcl commented on GitHub (Sep 23, 2026):

Thanks for the update! Glad to hear it’s nearly done. I am looking forward to seeing the PR and results.

It’ll also be great to have the evaluation in place so we can revisit #68 with more concrete data. Thanks again!

@shudorcl commented on GitHub (Sep 23, 2026): Thanks for the update! Glad to hear it’s nearly done. I am looking forward to seeing the PR and results. It’ll also be great to have the evaluation in place so we can revisit #68 with more concrete data. Thanks again!

bsakaguchi referenced this issue

2026-10-29 14:20:21 +00:00

[PR #70] [Parked] feat(snapshot): E2B alias rebuild semantics on template publish #155

bsakaguchi referenced this issue

2026-10-29 14:20:22 +00:00

[PR #71] [CLOSED] [1/4] feat(snapshot): template build-context archive store (posixfs + oss) #156

bsakaguchi referenced this issue

2026-10-29 14:20:22 +00:00

[PR #72] [CLOSED] [2/4] feat(api): E2B build-context upload endpoints and configuration #157

bsakaguchi referenced this issue

2026-10-29 14:20:26 +00:00

[PR #73] [CLOSED] [3/4] feat(template): host-side COPY plan (archive rewrite to guest paths) #158