[GH-ISSUE #63] Failed to build template on Arm64 server.

bsakaguchi commented

2026-10-29 14:18:31 +00:00

Owner

Originally created by @eric-yq on GitHub (Jul 29, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/63

Affected component

Snapshot / template builder

AgentENV version

main branch

Host environment

OS: Ubuntu 24.04
Kernel (uname -a): 7.0.0-19~24.04.1-Ubuntu SMP PREEMPT Mon Jul 13 22:29:54 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
Architecture: aarch64(AWS Graviton4)
Bare metal, VM, or cloud: AWS EC2 metal instance, c8g.metal-24xlarge.
ublk support: I think yes, I can start the server successfully.
CPU: AWS Graviton4, 96 core
Memory: 192G
Storage/filesystem: AWS EBS

Operation and configuration

aenv pull ubuntu:22.04 --name ubuntu

Steps to reproduce

I have built aenv-server from source code on AWS Graviton4 EC2 instance and run the server successfully.

I tried to download firecracker-v1.15.1-aarch64 and rename to /var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker

when I tried to run
aenv pull ubuntu:22.04 --name ubuntu ，

it thrown errors:
2026-07-29T11:39:59.328138Z WARN agentenv::api::impls::template: template build failed build_id=019fadac-991f-79c1-98c1-c690ed4faa6a reason=template build failed while running the build sandbox failed_step=None internal_causes=Some("build snapshot artifacts from requested base: attach tools drive version '0.1.0' as rootfs.ext4 for fresh sandbox 019fadac-9f8e-7962-af9e-59705cc5ea0e: Failed to add/update drive rootfs: Request failed: 400 Bad Request - {"fault_message":"An error occurred when deserializing the json body of a request: unknown field direct, expected one of drive_id, partuuid, is_root_device, cache_type, is_read_only, path_on_host, rate_limiter, io_engine, socket at line 1 column 119."}")

so, can you check whether it caused by wrong firecracker file without your patch ? v1.15.1 vs. v1.15.1-patch-v1.
And how to fix it.

Anyway, hope a formal release version can be released to support Arm64.
Thanks.

session_agentenv-test-c8g.metal_20260729.log

Expected behavior

aenv pull ubuntu:22.04 --name ubuntu successfully.

Actual behavior

(aenv-venv) root@ip-172-31-45-63:~# aenv pull ubuntu:22.04 --name ubuntu
Created template 019fadac-991f-79c1-98c1-c690ed4faa6a (build 019fadac-991f-79c1-98c1-c690ed4faa6a)
Build status: building
Error: Build failed: template build failed while running the build sandbox

on server:
2026-07-29T11:39:59.328138Z WARN agentenv::api::impls::template: template build failed build_id=019fadac-991f-79c1-98c1-c690ed4faa6a reason=template build failed while running the build sandbox failed_step=None internal_causes=Some("build snapshot artifacts from requested base: attach tools drive version '0.1.0' as rootfs.ext4 for fresh sandbox 019fadac-9f8e-7962-af9e-59705cc5ea0e: Failed to add/update drive rootfs: Request failed: 400 Bad Request - {"fault_message":"An error occurred when deserializing the json body of a request: unknown field direct, expected one of drive_id, partuuid, is_root_device, cache_type, is_read_only, path_on_host, rate_limiter, io_engine, socket at line 1 column 119."}")
2026-07-29T12:05:06.213454Z INFO agentenv::image::cache::store: image cache maintenance pass complete reclaimed=0 freed_bytes=0 retained=1

Logs and diagnostics

Reproduction frequency

Always

Regression information

Last known good:
First known bad:

Additional context

No response

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @eric-yq on GitHub (Jul 29, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/63 ### Affected component Snapshot / template builder ### AgentENV version main branch ### Host environment - OS: Ubuntu 24.04 - Kernel (`uname -a`): 7.0.0-19~24.04.1-Ubuntu SMP PREEMPT Mon Jul 13 22:29:54 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux - Architecture: aarch64(AWS Graviton4) - Bare metal, VM, or cloud: AWS EC2 metal instance, c8g.metal-24xlarge. - ublk support: I think yes, I can start the server successfully. - CPU: AWS Graviton4, 96 core - Memory: 192G - Storage/filesystem: AWS EBS ### Operation and configuration ```shell aenv pull ubuntu:22.04 --name ubuntu ``` ### Steps to reproduce I have built aenv-server from source code on AWS Graviton4 EC2 instance and run the server successfully. I tried to download `firecracker-v1.15.1-aarch64` and rename to `/var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker` when I tried to run `aenv pull ubuntu:22.04 --name ubuntu` ， it thrown errors: `2026-07-29T11:39:59.328138Z WARN agentenv::api::impls::template: template build failed build_id=019fadac-991f-79c1-98c1-c690ed4faa6a reason=template build failed while running the build sandbox failed_step=None internal_causes=Some("build snapshot artifacts from requested base: attach tools drive version '0.1.0' as rootfs.ext4 for fresh sandbox 019fadac-9f8e-7962-af9e-59705cc5ea0e: Failed to add/update drive rootfs: Request failed: 400 Bad Request - {"fault_message":"An error occurred when deserializing the json body of a request: unknown field direct, expected one of drive_id, partuuid, is_root_device, cache_type, is_read_only, path_on_host, rate_limiter, io_engine, socket at line 1 column 119."}")` so, can you check whether it caused by wrong firecracker file without your patch ? v1.15.1 vs. v1.15.1-patch-v1. And how to fix it. Anyway, hope a formal release version can be released to support Arm64. Thanks. [session_agentenv-test-c8g.metal_20260729.log](https://github.com/user-attachments/files/30505134/session_agentenv-test-c8g.metal_54.92.159.128_20260729.log) ### Expected behavior `aenv pull ubuntu:22.04 --name ubuntu` successfully. ### Actual behavior (aenv-venv) root@ip-172-31-45-63:~# aenv pull ubuntu:22.04 --name ubuntu Created template 019fadac-991f-79c1-98c1-c690ed4faa6a (build 019fadac-991f-79c1-98c1-c690ed4faa6a) Build status: building Error: Build failed: template build failed while running the build sandbox on server: 2026-07-29T11:39:59.328138Z WARN agentenv::api::impls::template: template build failed build_id=019fadac-991f-79c1-98c1-c690ed4faa6a reason=template build failed while running the build sandbox failed_step=None internal_causes=Some("build snapshot artifacts from requested base: attach tools drive version '0.1.0' as rootfs.ext4 for fresh sandbox 019fadac-9f8e-7962-af9e-59705cc5ea0e: Failed to add/update drive rootfs: Request failed: 400 Bad Request - {\"fault_message\":\"An error occurred when deserializing the json body of a request: unknown field `direct`, expected one of `drive_id`, `partuuid`, `is_root_device`, `cache_type`, `is_read_only`, `path_on_host`, `rate_limiter`, `io_engine`, `socket` at line 1 column 119.\"}") 2026-07-29T12:05:06.213454Z INFO agentenv::image::cache::store: image cache maintenance pass complete reclaimed=0 freed_bytes=0 retained=1 ### Logs and diagnostics ```text ``` ### Reproduction frequency Always ### Regression information - Last known good: - First known bad: ### Additional context _No response_ ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:18:31 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 14:18:32 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Jul 29, 2026):

We've published a patch to Firecracker at https://github.com/kvcache-ai/firecracker that optimizes snapshotting performance. Without it, memory has to be written to disk twice. Building a binary from that source should resolve this issue.

@yingdi-shan commented on GitHub (Jul 29, 2026): We've published a patch to Firecracker at https://github.com/kvcache-ai/firecracker that optimizes snapshotting performance. Without it, memory has to be written to disk twice. Building a binary from that source should resolve this issue.

bsakaguchi commented

2026-10-29 14:18:33 +00:00

Author

Owner

@Abhishek6002 commented on GitHub (Jul 29, 2026):

Hi! I'd like to work on this issue. Could you assign it to me?

@Abhishek6002 commented on GitHub (Jul 29, 2026): Hi! I'd like to work on this issue. Could you assign it to me?

bsakaguchi commented

2026-10-29 14:18:33 +00:00

Author

Owner

@eric-yq commented on GitHub (Jul 30, 2026):

@yingdi-shan Hi, can you assign this issue to @Abhishek6002 , he may help solve it .

Thanks.

@eric-yq commented on GitHub (Jul 30, 2026): @yingdi-shan Hi, can you assign this issue to @Abhishek6002 , he may help solve it . Thanks.

bsakaguchi commented

2026-10-29 14:18:33 +00:00

Author

Owner

@eric-yq commented on GitHub (Jul 30, 2026):

@yingdi-shan I have build firecracker with https://github.com/kvcache-ai/firecracker and placed to /var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker , but still throw same errors.
Can you help what addtional step I should do ?

I tried to fix it, but found many issues may need to adopt arm64, such vmlinux, envd, busybox etc.
So hope to have a complete release to adopt arm64 .

@eric-yq commented on GitHub (Jul 30, 2026): @yingdi-shan I have build firecracker with https://github.com/kvcache-ai/firecracker and placed to /var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker , but still throw same errors. Can you help what addtional step I should do ? I tried to fix it, but found many issues may need to adopt arm64, such vmlinux, envd, busybox etc. So hope to have a complete release to adopt arm64 .

bsakaguchi commented

2026-10-29 14:18:33 +00:00

Author

Owner

@eric-yq commented on GitHub (Jul 30, 2026):

I have built AgentENV on ARM64 susscessfully with re-building tools.ext4 and replaced to /var/lib/aenv/deps/tools/0.1.0/tools.ext4

root@ip-172-31-43-122:~# aenv cn 019fb21e-4318-7bc3-96b1-3a96d268d0c7
root@instance:/# arch
aarch64

It's awesome project.

@eric-yq commented on GitHub (Jul 30, 2026): I have built AgentENV on ARM64 susscessfully with re-building tools.ext4 and replaced to /var/lib/aenv/deps/tools/0.1.0/tools.ext4 root@ip-172-31-43-122:~# aenv cn 019fb21e-4318-7bc3-96b1-3a96d268d0c7 root@instance:/# arch aarch64 It's awesome project.

bsakaguchi commented

2026-10-29 14:18:33 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Aug 9, 2026):

Resolved in PR #86.

@yingdi-shan commented on GitHub (Aug 9, 2026): Resolved in PR #86.

bsakaguchi referenced this issue

2026-10-29 14:18:34 +00:00

[GH-ISSUE #90] RFC: explicit placement reservation lifecycle and shared ownership #29

bsakaguchi referenced this issue

2026-10-29 14:18:35 +00:00