[GH-ISSUE #125] Analyze boot/resume latency #41

New issue

Open

opened 2026-10-29 14:18:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:43 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Aug 5, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/125

Summary

A regular Ubuntu 24.04 snapshot resume, with the Firecracker pool and shared memory device warm, takes:

Samples	Mean	p50	Min	Max
10	46.95 ms	44.80 ms	42.83 ms	66.57 ms

Instrumentation accounts for 46.26 ms, or 98.53% of the mean. This covers the common restore path used by snapshot-backed sandbox creation and paused sandbox resume. Fresh VM boot is excluded.

The benchmark measures FirecrackerSandbox::resume_from_snapshot_config: constructing the runtime from an already-resolved snapshot config, restoring Firecracker, waiting for envd health, initializing envd, and returning a ready sandbox.

Reproduction Environment

Hardware and host requirements:

x86_64 host with /dev/kvm access
Linux kernel with ublk support and /dev/ublk-control access
Intel Xeon Gold 5317, 48 logical CPUs, 2 NUMA nodes
128 GiB RAM with swap disabled
NVMe-backed ext4 storage

Software and workload:

Linux 6.8.0-111-generic
Firecracker 1.15.1-patch-v1
Guest kernel vmlinux-6.1.175
envd 0.5.15
Ubuntu 24.04
1 vCPU, 128 MiB, no attached drives
Uncompressed memory snapshot with direct OverlayBD enabled

To reproduce the regular result, warm the Firecracker and network pools and keep one sandbox from the same snapshot alive so the shared memory ublk device remains referenced.

Time Breakdown

Stage	Mean	p50	Share
Firecracker pool acquire	0.338 ms	0.337 ms	0.72%
Tools drive link	0.065 ms	0.062 ms	0.14%
Rootfs OverlayBD bind	2.945 ms	2.876 ms	6.27%
Network policy	3.667 ms	3.662 ms	7.81%
Memory OverlayBD bind	0.034 ms	0.033 ms	0.07%
Firecracker snapshot load	9.579 ms	7.631 ms	20.40%
MMDS update	0.347 ms	0.337 ms	0.74%
Firecracker resume	0.563 ms	0.527 ms	1.20%
Guest/envd health	22.381 ms	22.564 ms	47.67%
Release background downloads	0.626 ms	0.644 ms	1.33%
envd initialization	5.710 ms	5.442 ms	12.16%
Unattributed	0.691 ms	-	1.47%
Total	46.948 ms	44.800 ms	100%

Before the vCPU resumes, setup consumes 17.54 ms. Guest health and envd initialization consume another 28.72 ms after the vCPU starts.

Analysis

The 2.95 ms rootfs OverlayBD bind consists of approximately:

0.54 ms to materialize runtime configuration
1.86 ms to open the OverlayBD image
0.08 ms to bind the warm device
0.46 ms in main-process configuration and IPC

Memory binding costs only 0.034 ms because another live sandbox retains the shared memory ublk device.

guest_ready is the complete GET /health wait while the vCPU is running. It includes guest execution, demand paging, scheduling, networking, envd processing, and occasional retry delay. It is not 22 ms of host-side polling.

A separate trace measured approximately 2.3k KVM faults and 11.60 ms of cumulative kvm_tdp_page_fault handler time.envd_init is a subsequent POST /init call and costs another 5.71 ms.

Follow-up Opportunities

Prefetch the recorded memory working set and prefault KVM mappings before starting the vCPU using KVM_PRE_FAULT_MEMORY.
Cache rootfs materialization and image-open work.
Avoid redundant network-policy updates.

Originally created by @yingdi-shan on GitHub (Aug 5, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/125 ## Summary A regular Ubuntu 24.04 snapshot resume, with the Firecracker pool and shared memory device warm, takes: | Samples | Mean | p50 | Min | Max | | ---: | ---: | ---: | ---: | ---: | | 10 | 46.95 ms | 44.80 ms | 42.83 ms | 66.57 ms | Instrumentation accounts for 46.26 ms, or 98.53% of the mean. This covers the common restore path used by snapshot-backed sandbox creation and paused sandbox resume. Fresh VM boot is excluded. The benchmark measures `FirecrackerSandbox::resume_from_snapshot_config`: constructing the runtime from an already-resolved snapshot config, restoring Firecracker, waiting for envd health, initializing envd, and returning a ready sandbox. ## Reproduction Environment Hardware and host requirements: - x86_64 host with `/dev/kvm` access - Linux kernel with ublk support and `/dev/ublk-control` access - Intel Xeon Gold 5317, 48 logical CPUs, 2 NUMA nodes - 128 GiB RAM with swap disabled - NVMe-backed ext4 storage Software and workload: - Linux `6.8.0-111-generic` - Firecracker `1.15.1-patch-v1` - Guest kernel `vmlinux-6.1.175` - envd `0.5.15` - Ubuntu 24.04 - 1 vCPU, 128 MiB, no attached drives - Uncompressed memory snapshot with direct OverlayBD enabled To reproduce the regular result, warm the Firecracker and network pools and keep one sandbox from the same snapshot alive so the shared memory ublk device remains referenced. ## Time Breakdown | Stage | Mean | p50 | Share | | --- | ---: | ---: | ---: | | Firecracker pool acquire | 0.338 ms | 0.337 ms | 0.72% | | Tools drive link | 0.065 ms | 0.062 ms | 0.14% | | Rootfs OverlayBD bind | 2.945 ms | 2.876 ms | 6.27% | | Network policy | 3.667 ms | 3.662 ms | 7.81% | | Memory OverlayBD bind | 0.034 ms | 0.033 ms | 0.07% | | Firecracker snapshot load | 9.579 ms | 7.631 ms | 20.40% | | MMDS update | 0.347 ms | 0.337 ms | 0.74% | | Firecracker resume | 0.563 ms | 0.527 ms | 1.20% | | Guest/envd health | 22.381 ms | 22.564 ms | 47.67% | | Release background downloads | 0.626 ms | 0.644 ms | 1.33% | | envd initialization | 5.710 ms | 5.442 ms | 12.16% | | Unattributed | 0.691 ms | - | 1.47% | | **Total** | **46.948 ms** | **44.800 ms** | **100%** | Before the vCPU resumes, setup consumes 17.54 ms. Guest health and envd initialization consume another 28.72 ms after the vCPU starts. ## Analysis The 2.95 ms rootfs OverlayBD bind consists of approximately: - 0.54 ms to materialize runtime configuration - 1.86 ms to open the OverlayBD image - 0.08 ms to bind the warm device - 0.46 ms in main-process configuration and IPC Memory binding costs only 0.034 ms because another live sandbox retains the shared memory ublk device. `guest_ready` is the complete `GET /health` wait while the vCPU is running. It includes guest execution, demand paging, scheduling, networking, envd processing, and occasional retry delay. It is not 22 ms of host-side polling. A separate trace measured approximately 2.3k KVM faults and 11.60 ms of cumulative `kvm_tdp_page_fault` handler time.`envd_init` is a subsequent `POST /init` call and costs another 5.71 ms. ## Follow-up Opportunities 1. Prefetch the recorded memory working set and prefault KVM mappings before starting the vCPU using KVM_PRE_FAULT_MEMORY. 2. Cache rootfs materialization and image-open work. 3. Avoid redundant network-policy updates.

bsakaguchi referenced this issue

2026-10-29 14:20:05 +00:00

[PR #44] [CLOSED] fix(orchestrator): resume sandboxes after recoverable pause failures #136

bsakaguchi referenced this issue

2026-10-29 14:20:16 +00:00

[PR #64] [MERGED] fix(sandbox): resume sandbox upon recoverable pause failure #149