[GH-ISSUE #407] bug(network): guest udev rewrites the random eth0 MAC on cold boot, stale host neighbour makes the proxy 502 for ~40s #105

New issue

Closed

opened 2026-10-29 14:19:37 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 14:19:37 +00:00

Owner

Copy link

Originally created by @dushulin on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/407

Summary

On a cold boot, AgentENV attaches eth0 without a guest_mac, so the guest kernel assigns a random MAC. Images that boot systemd (>= 242) with systemd-udevd then rewrite that MAC through the stock 99-default.link (MACAddressPolicy=persistent). No gratuitous ARP is sent. The host neighbour entry for the VM IP on tap0 keeps pointing at the boot-time MAC. Host-to-guest frames are dropped until the guest happens to ARP on its own, or the host entry ages out and is re-resolved.

During that window every proxied connection to the sandbox hits the proxy's 5 s connect timeout and returns 502. This happens even though the sandbox was created successfully and had already answered requests. In our reproduction the window lasted about 43 s.

Affected component

Firecracker runtime / Network

Environment and verification scope

• Runtime evidence comes from a downstream deployment. The affected code path is identical at public revision a27b2db.
• Host: Linux x86_64, virtualization_mode = "kvm", Firecracker 1.15.1-patch-v1 and guest kernel vmlinux-6.1.175 (both the deps_manifest.toml defaults).
• Guest images: user images whose pivot-init hand-off ends in systemd as PID 1.
  • Image A: Ubuntu 24.04, systemd 255, avahi-daemon.
  • Image B: Debian 12, systemd 252.
• The live reproduction used one test sandbox at a time on an otherwise lightly loaded node. Node load is not a factor.

How it showed up

• In a large load test, connect-timeout 502s clustered strictly by image.
• A handful of images failed on most cold starts. Dozens of sibling images, running alongside them on the same nodes, never failed.
• Every affected image boots systemd + udevd. Images without systemd, or with systemd but no udevd, were unaffected.
• In the serial logs, the first failing connect always came after systemd-udev-trigger.

Reproduction (Image A, single sandbox, normalized)

Setup:

• ip -timestamp monitor neigh running in the slot netns.
• A probe every 250 ms: GET /proxy/health with e2b-sandbox-id: <id> and e2b-sandbox-port: 49983, sent to the node's API port.
• t=0 is when the create request returned.

t	Event
-1.6 s	Firecracker config: PUT /network-interfaces/eth0 {"host_dev_name":"tap0","iface_id":"eth0"} (no guest_mac)
boot +1.09 s	guest: virtio_net virtio3: Assigned random MAC address <boot-mac>
-0.5 s	host tap0: 169.254.0.21 lladdr <boot-mac> REACHABLE
0 – 0.75 s	4 probes → 204 (~1 ms)
boot +~2.5 s	systemd-udevd applies 99-default.link; eth0 becomes <persistent-mac>
+1.0 s … +39.4 s	154 consecutive probes → 502, each time_total = 5.00 s
+37 s	host: 169.254.0.21 lladdr <boot-mac> STALE (aged out, still the old MAC)
+43 s	host: PROBE → FAILED → broadcast re-resolve → lladdr <persistent-mac> REACHABLE
+43.5 s	probes → 204 again

Guest state afterwards:

/sys/class/net/eth0/addr_assign_type      3        (NET_ADDR_SET: changed from userspace)
udevadm info: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
99-default.link [Link]: MACAddressPolicy=persistent
/proc/sys/net/ipv4/conf/eth0/arp_notify   0

Image A only emits mDNS multicast after boot and never ARPs its gateway. The host therefore has to wait for its neighbour entry to age out.

Image B rewrites the MAC the same way: the result is deterministic, derived from the image's machine-id plus eth0. However, its guest ARPs the gateway 0.7–5.7 s after the rewrite. The host then learns the new MAC (as STALE with the new lladdr), and in our runs the window was usually too short to observe a 502. Whether a given image is affected depends on whether its guest sends unicast traffic soon after udev runs.

Root cause

1. start_fresh attaches eth0 with guest_mac: None. Firecracker therefore does not offer VIRTIO_NET_F_MAC, and virtio_net falls back to a random MAC (addr_assign_type = NET_ADDR_RANDOM). The None has been there since the initial open-source release, so this is not a regression.
2. pivot-init execs the image's own init after envd is up. For systemd images this starts systemd-udevd.
3. Since systemd v242 (systemd/systemd#11382), MACAddressPolicy=persistent also applies to interfaces whose MAC is NET_ADDR_RANDOM. Upstream treats this as intended behaviour (see systemd/systemd#21185), and the logic is unchanged through current releases. A MAC reported as NET_ADDR_PERM is left alone.
4. The guest does not announce the change (arp_notify=0). The host keeps unicasting to the old MAC, the guest drops those frames, and SYNs from the proxy are lost. The proxy connect timeout is 5 s, so each attempt ends in 502.
5. The existing retrans_time_ms tuning on tap0 only shortens the final probe phase. The REACHABLE → STALE → DELAY time still dominates.

Restore and fork paths do not seem to be affected:

• They load the NIC from vm_state.bin with only a host_dev_name override. The guest keeps whatever MAC it already had.
• When the previous Firecracker process exits, tap0 loses carrier and the kernel flushes its neighbour entries; we observed this as Deleted 169.254.0.21 dev tap0 ... events. A restored guest is therefore resolved again by broadcast ARP.

Expected behavior

Once create has returned and envd has answered, the sandbox should stay reachable through the proxy regardless of which init system or udev rules the user image ships.

Proposed fix

Pass a fixed, locally administered unicast MAC (for example 02:fc:00:00:00:05) as guest_mac in start_fresh.

• Firecracker then exposes VIRTIO_NET_F_MAC, the guest reports NET_ADDR_PERM, and MACAddressPolicy=persistent leaves the MAC unchanged.
• A single constant is enough: every slot already has its own netns, tap0 and a fixed VM IP (NetworkAddressPlan::vm_ip), so the MAC is only visible on that private link.
• Related reports from other projects: cilium/cilium#8304 (no endpoint connectivity with systemd 242) and liquidmetal-dev/flintlock#1284 (Firecracker TAP without guest_mac). Firecracker's own CI rootfs works around this by masking 99-default.link (resources/chroot.sh in firecracker-microvm/firecracker#5364).

Compatibility:

• Only cold boots change.
• Existing paused sandboxes and committed snapshots restore with their saved device state, exactly as today.
• Snapshots taken after the change carry the fixed MAC.

Alternatives considered:

• Shortening neighbour timers on tap0: shrinks the window but does not remove it.
• A static host neighbour entry: would break guests restored from older snapshots with a different MAC.
• Injecting a .link override into the guest: intrusive to user rootfs.

Proposed verification (not yet executed): cold-boot Image A with the change. Expect addr_assign_type = 0, the MAC to equal the configured value, a single neighbour entry with no MAC change, and zero 502s over a 90 s probe window.

Reproduction frequency

Always for Image A; intermittent (race-dependent) for images whose guest ARPs soon after udev runs.

Pre-submission checklist

• I searched existing open and closed issues and did not find a duplicate.
• I provided a minimal reproduction or explained why one cannot be provided.
• I removed credentials, tokens, private image references, and other sensitive information.
• This is not a security vulnerability.

Originally created by @dushulin on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/407 ### Summary On a cold boot, AgentENV attaches `eth0` without a `guest_mac`, so the guest kernel assigns a random MAC. Images that boot systemd (>= 242) with `systemd-udevd` then rewrite that MAC through the stock `99-default.link` (`MACAddressPolicy=persistent`). No gratuitous ARP is sent. The host neighbour entry for the VM IP on `tap0` keeps pointing at the boot-time MAC. Host-to-guest frames are dropped until the guest happens to ARP on its own, or the host entry ages out and is re-resolved. During that window every proxied connection to the sandbox hits the proxy's 5 s connect timeout and returns `502`. This happens even though the sandbox was created successfully and had already answered requests. In our reproduction the window lasted about 43 s. ### Affected component Firecracker runtime / Network ### Environment and verification scope - Runtime evidence comes from a downstream deployment. The affected code path is identical at public revision [`a27b2db`](https://github.com/kvcache-ai/AgentENV/commit/a27b2db909b035f58696ba8798ea0e285c7bfa17). - Host: Linux x86_64, `virtualization_mode = "kvm"`, Firecracker `1.15.1-patch-v1` and guest kernel `vmlinux-6.1.175` (both the `deps_manifest.toml` defaults). - Guest images: user images whose `pivot-init` hand-off ends in systemd as PID 1. - **Image A:** Ubuntu 24.04, systemd 255, avahi-daemon. - **Image B:** Debian 12, systemd 252. - The live reproduction used one test sandbox at a time on an otherwise lightly loaded node. Node load is not a factor. ### How it showed up - In a large load test, connect-timeout `502`s clustered strictly by image. - A handful of images failed on most cold starts. Dozens of sibling images, running alongside them on the same nodes, never failed. - Every affected image boots systemd + udevd. Images without systemd, or with systemd but no udevd, were unaffected. - In the serial logs, the first failing connect always came after `systemd-udev-trigger`. ### Reproduction (Image A, single sandbox, normalized) Setup: - `ip -timestamp monitor neigh` running in the slot netns. - A probe every 250 ms: `GET /proxy/health` with `e2b-sandbox-id: <id>` and `e2b-sandbox-port: 49983`, sent to the node's API port. - `t=0` is when the create request returned. | t | Event | |---|---| | -1.6 s | Firecracker config: `PUT /network-interfaces/eth0 {"host_dev_name":"tap0","iface_id":"eth0"}` (no `guest_mac`) | | boot +1.09 s | guest: `virtio_net virtio3: Assigned random MAC address <boot-mac>` | | -0.5 s | host `tap0`: `169.254.0.21 lladdr <boot-mac> REACHABLE` | | 0 – 0.75 s | 4 probes → `204` (~1 ms) | | boot +~2.5 s | systemd-udevd applies `99-default.link`; eth0 becomes `<persistent-mac>` | | +1.0 s … +39.4 s | **154 consecutive probes → `502`, each `time_total` = 5.00 s** | | +37 s | host: `169.254.0.21 lladdr <boot-mac> STALE` (aged out, still the old MAC) | | +43 s | host: `PROBE` → `FAILED` → broadcast re-resolve → `lladdr <persistent-mac> REACHABLE` | | +43.5 s | probes → `204` again | Guest state afterwards: ```text /sys/class/net/eth0/addr_assign_type 3 (NET_ADDR_SET: changed from userspace) udevadm info: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link 99-default.link [Link]: MACAddressPolicy=persistent /proc/sys/net/ipv4/conf/eth0/arp_notify 0 ``` Image A only emits mDNS multicast after boot and never ARPs its gateway. The host therefore has to wait for its neighbour entry to age out. Image B rewrites the MAC the same way: the result is deterministic, derived from the image's machine-id plus `eth0`. However, its guest ARPs the gateway 0.7–5.7 s after the rewrite. The host then learns the new MAC (as `STALE` with the new lladdr), and in our runs the window was usually too short to observe a `502`. Whether a given image is affected depends on whether its guest sends unicast traffic soon after udev runs. ### Root cause 1. [`start_fresh` attaches eth0 with `guest_mac: None`](https://github.com/kvcache-ai/AgentENV/blob/a27b2db909b035f58696ba8798ea0e285c7bfa17/src/sandbox/firecracker/sandbox.rs#L2210). Firecracker therefore does not offer `VIRTIO_NET_F_MAC`, and virtio_net falls back to a random MAC (`addr_assign_type = NET_ADDR_RANDOM`). The `None` has been there since the initial open-source release, so this is not a regression. 2. [`pivot-init` execs the image's own init](https://github.com/kvcache-ai/AgentENV/blob/a27b2db909b035f58696ba8798ea0e285c7bfa17/tools-image/pivot-init#L163) after envd is up. For systemd images this starts `systemd-udevd`. 3. Since systemd v242 ([systemd/systemd#11382](https://github.com/systemd/systemd/pull/11382)), `MACAddressPolicy=persistent` also applies to interfaces whose MAC is `NET_ADDR_RANDOM`. Upstream treats this as intended behaviour (see [systemd/systemd#21185](https://github.com/systemd/systemd/issues/21185)), and the logic is unchanged through current releases. A MAC reported as `NET_ADDR_PERM` is left alone. 4. The guest does not announce the change (`arp_notify=0`). The host keeps unicasting to the old MAC, the guest drops those frames, and SYNs from the proxy are lost. The [proxy connect timeout is 5 s](https://github.com/kvcache-ai/AgentENV/blob/a27b2db909b035f58696ba8798ea0e285c7bfa17/src/api/proxy.rs#L96), so each attempt ends in `502`. 5. The existing [`retrans_time_ms` tuning on `tap0`](https://github.com/kvcache-ai/AgentENV/blob/a27b2db909b035f58696ba8798ea0e285c7bfa17/src/sandbox/network/slot.rs#L263) only shortens the final probe phase. The `REACHABLE` → `STALE` → `DELAY` time still dominates. Restore and fork paths do not seem to be affected: - They load the NIC from `vm_state.bin` with only a [`host_dev_name` override](https://github.com/kvcache-ai/AgentENV/blob/a27b2db909b035f58696ba8798ea0e285c7bfa17/src/sandbox/firecracker/sandbox.rs#L1696). The guest keeps whatever MAC it already had. - When the previous Firecracker process exits, `tap0` loses carrier and the kernel flushes its neighbour entries; we observed this as `Deleted 169.254.0.21 dev tap0 ...` events. A restored guest is therefore resolved again by broadcast ARP. ### Expected behavior Once create has returned and envd has answered, the sandbox should stay reachable through the proxy regardless of which init system or udev rules the user image ships. ### Proposed fix Pass a fixed, locally administered unicast MAC (for example `02:fc:00:00:00:05`) as `guest_mac` in `start_fresh`. - Firecracker then exposes `VIRTIO_NET_F_MAC`, the guest reports `NET_ADDR_PERM`, and `MACAddressPolicy=persistent` leaves the MAC unchanged. - A single constant is enough: every slot already has its own netns, `tap0` and a fixed VM IP (`NetworkAddressPlan::vm_ip`), so the MAC is only visible on that private link. - Related reports from other projects: [cilium/cilium#8304](https://github.com/cilium/cilium/issues/8304) (no endpoint connectivity with systemd 242) and [liquidmetal-dev/flintlock#1284](https://github.com/liquidmetal-dev/flintlock/issues/1284) (Firecracker TAP without `guest_mac`). Firecracker's own CI rootfs works around this by masking `99-default.link` (`resources/chroot.sh` in [firecracker-microvm/firecracker#5364](https://github.com/firecracker-microvm/firecracker/pull/5364)). Compatibility: - Only cold boots change. - Existing paused sandboxes and committed snapshots restore with their saved device state, exactly as today. - Snapshots taken after the change carry the fixed MAC. Alternatives considered: - Shortening neighbour timers on `tap0`: shrinks the window but does not remove it. - A static host neighbour entry: would break guests restored from older snapshots with a different MAC. - Injecting a `.link` override into the guest: intrusive to user rootfs. Proposed verification (not yet executed): cold-boot Image A with the change. Expect `addr_assign_type = 0`, the MAC to equal the configured value, a single neighbour entry with no MAC change, and zero `502`s over a 90 s probe window. ### Reproduction frequency Always for Image A; intermittent (race-dependent) for images whose guest ARPs soon after udev runs. ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi closed this issue 2026-10-29 14:19:38 +00:00

bsakaguchi commented 2026-10-29 14:19:38 +00:00

Author

Owner

Copy link

@dushulin commented on GitHub (Oct 21, 2026):

@yingdi-shan hi～PTAL, thanks
I can give a PR

<!-- gh-comment-id:6315338320 --> @dushulin commented on GitHub (Oct 21, 2026): @yingdi-shan hi～PTAL, thanks I can give a PR

bsakaguchi commented 2026-10-29 14:19:39 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Oct 21, 2026):

Maybe we can just set sysctl.net.ipv4.conf.all.arp_notify=1 in the guest kernel parameters to solve this problem?

<!-- gh-comment-id:6318828731 --> @yingdi-shan commented on GitHub (Oct 21, 2026): Maybe we can just set sysctl.net.ipv4.conf.all.arp_notify=1 in the guest kernel parameters to solve this problem?

bsakaguchi referenced this issue 2026-10-29 14:20:43 +00:00

[PR #105] [MERGED] chore(release): bump version to v0.1.1 #181

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#105

No description provided.