amnezia-vpn/amnezia-client

Fork 0

mirror of https://github.com/amnezia-vpn/amnezia-client.git synced 2026-10-15 05:42:07 +00:00

[GH-ISSUE #2933] Deadlock between AmneziaVPN-service and kernel network locks on lid close (macOS 26.6, app 5.0.0.5, VLESS) — configd killed by WATCHDOG #1337

New issue

Open

opened 2026-10-13 20:40:28 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-13 20:40:28 +00:00

Owner

Originally created by @romanovevgeniy on GitHub (Aug 4, 2026).
Original GitHub issue: https://github.com/amnezia-vpn/amnezia-client/issues/2933

Summary

On macOS 26.6 (25G72), closing the MacBook lid while the VPN is connected reliably
deadlocks the system networking stack. A kernel lock cycle forms between
AmneziaVPN-service and the kernel, configd's IPConfigurationAgentQueue stalls,
and after 60 seconds the watchdog terminates configd.

This happens every single day, on every lid close with the VPN running.

Environment

AmneziaVPN for macOS 5.0.0.5
macOS 26.6 (build 25G72)
MacBookPro18,3 (Apple Silicon)
Protocol: VLESS (XRay)
tun2socks is present in the process list (41 threads), consistent with the
XRay/VLESS userspace tunnel path
Report incident ID: 19D61D4C-C90C-4BF8-AE79-91B9C65CAC96
Report type: bug_type 409 (WATCHDOG stackshot), displayState: OFF

Steps to reproduce

Connect the VPN.
Close the MacBook lid.
Within ~60 seconds a configd watchdog report is generated.

Actual result

configd is killed by the [watchdog]

Root cause from the stackshot — four-way lock cycle

configd thread 103010 (IPConfigurationAgentQueue) waits on kernel mutex
0xfffffe000c571830, owned by kernel thread 103456 (thread call high #27).
Kernel thread 103456 waits on krwlock 0xfffffe000c5a4270 for reading,
owned by thread 104354.
Thread 104354 belongs to AmneziaVPN-service (pid 553) and waits on kernel
mutex 0x5162d882b476c1c1, owned by kernel thread 5256.
Kernel thread 5256 waits on mutex 0xfffffe000c571830 — back to 103456.

The cycle never breaks, so the whole networking subsystem stalls.

Additional observations

AmneziaVPN-service had 259 threads and ~955 MB resident memory, with
~592k page faults. Roughly 200 of those threads were parked on the same
0xfffffe000c571830 mutex, and about a dozen more on the krwlock held by 104354.
The thread and memory growth looks like a leak independent of the deadlock.
tun2socks (41 threads) is blocked on the same mutex.
The AmneziaVPN GUI process is also stuck: thread 16371 waits on krwlock
0xfffffe000c5a4270 for writing, owned by 104354.

Likely trigger

On lid close macOS tears down Wi-Fi and reconfigures interfaces, so configd
dismantles network services while the client simultaneously tears down its utun.
The two sides appear to acquire the network locks in opposite order.

Workaround

Disconnecting the VPN before closing the lid avoids the issue entirely.

Attachment

Full watchdog stackshot attached.

Error.rtf

Originally created by @romanovevgeniy on GitHub (Aug 4, 2026). Original GitHub issue: https://github.com/amnezia-vpn/amnezia-client/issues/2933 ## Summary On macOS 26.6 (25G72), closing the MacBook lid while the VPN is connected reliably deadlocks the system networking stack. A kernel lock cycle forms between `AmneziaVPN-service` and the kernel, `configd`'s `IPConfigurationAgentQueue` stalls, and after 60 seconds the watchdog terminates `configd`. This happens every single day, on every lid close with the VPN running. ## Environment - AmneziaVPN for macOS **5.0.0.5** - macOS 26.6 (build 25G72) - MacBookPro18,3 (Apple Silicon) - Protocol: **VLESS (XRay)** - `tun2socks` is present in the process list (41 threads), consistent with the XRay/VLESS userspace tunnel path - Report incident ID: 19D61D4C-C90C-4BF8-AE79-91B9C65CAC96 - Report type: bug_type 409 (WATCHDOG stackshot), `displayState: OFF` ## Steps to reproduce 1. Connect the VPN. 2. Close the MacBook lid. 3. Within ~60 seconds a `configd` watchdog report is generated. ## Actual result `configd` is killed by the [watchdog] ## Root cause from the stackshot — four-way lock cycle 1. `configd` thread 103010 (`IPConfigurationAgentQueue`) waits on kernel mutex `0xfffffe000c571830`, owned by kernel thread 103456 (`thread call high #27`). 2. Kernel thread 103456 waits on krwlock `0xfffffe000c5a4270` **for reading**, owned by thread 104354. 3. Thread 104354 belongs to **AmneziaVPN-service (pid 553)** and waits on kernel mutex `0x5162d882b476c1c1`, owned by kernel thread 5256. 4. Kernel thread 5256 waits on mutex `0xfffffe000c571830` — back to 103456. The cycle never breaks, so the whole networking subsystem stalls. ## Additional observations - `AmneziaVPN-service` had **259 threads** and **~955 MB** resident memory, with ~592k page faults. Roughly 200 of those threads were parked on the same `0xfffffe000c571830` mutex, and about a dozen more on the krwlock held by 104354. The thread and memory growth looks like a leak independent of the deadlock. - `tun2socks` (41 threads) is blocked on the same mutex. - The `AmneziaVPN` GUI process is also stuck: thread 16371 waits on krwlock `0xfffffe000c5a4270` for writing, owned by 104354. ## Likely trigger On lid close macOS tears down Wi-Fi and reconfigures interfaces, so `configd` dismantles network services while the client simultaneously tears down its `utun`. The two sides appear to acquire the network locks in opposite order. ## Workaround Disconnecting the VPN before closing the lid avoids the issue entirely. ## Attachment Full watchdog stackshot attached. [Error.rtf](https://github.com/user-attachments/files/30687946/Error.rtf)

bsakaguchi commented

2026-10-13 20:40:30 +00:00

Author

Owner

@RiRion commented on GitHub (Aug 6, 2026):

I’m experiencing what appears to be the same issue described in both #2897 and #2933.

My MacBook Pro has an Apple M2 Max chip, and the problem occurred on both macOS 26.5 and macOS 26.6 while using AmneziaVPN 5.0.0.5.

The Mac completely freezes and then reboots with a kernel panic: userspace watchdog timeout: no successful checkins from configd. The panic log shows that configd stopped responding for 180 seconds, after which AppleARMWatchdogTimer triggered the panic.

This seems consistent with a deadlock in the macOS networking stack involving AmneziaVPN-service and configd, as described in these two issues. AmneziaVPN was installed and active when the freezes occurred. I can provide the full panic logs if needed.

@RiRion commented on GitHub (Aug 6, 2026): I’m experiencing what appears to be the same issue described in both #2897 and #2933. My MacBook Pro has an Apple M2 Max chip, and the problem occurred on both macOS 26.5 and macOS 26.6 while using AmneziaVPN 5.0.0.5. The Mac completely freezes and then reboots with a kernel panic: userspace watchdog timeout: no successful checkins from configd. The panic log shows that configd stopped responding for 180 seconds, after which AppleARMWatchdogTimer triggered the panic. This seems consistent with a deadlock in the macOS networking stack involving AmneziaVPN-service and configd, as described in these two issues. AmneziaVPN was installed and active when the freezes occurred. I can provide the full panic logs if needed.

bsakaguchi commented

2026-10-13 20:40:30 +00:00

Author

Owner

@antivanmsk commented on GitHub (Aug 8, 2026):

On my iMac (M4) with macOS 26.6, Amnesia version 5.0.0.5 works for a while, but then it freezes and completely locks up — all apps stop working. Only a hard reboot helps, and I have to force-quit each previously open app individually!

@antivanmsk commented on GitHub (Aug 8, 2026): On my iMac (M4) with macOS 26.6, Amnesia version 5.0.0.5 works for a while, but then it freezes and completely locks up — all apps stop working. Only a hard reboot helps, and I have to force-quit each previously open app individually!

bsakaguchi commented

2026-10-13 20:40:30 +00:00

Author

Owner

@dmshem commented on GitHub (Aug 21, 2026):

Environment

macOS Tahoe 26.1
AmneziaVPN 5.0.0.5
VLESS / XRay
AmneziaVPN installed from the .pkg distribution

Problem

AmneziaVPN occasionally enters a system-level deadlock, usually after the Mac has gone to sleep while VPN is connected.

After waking up:

AmneziaVPN hangs on Disconnect indefinitely.
Network connectivity becomes unusable.
Killing the AmneziaVPN GUI does not restore networking.
pfctl itself hangs, for example:

sudo pfctl -s info

A normal macOS reboot/shutdown also hangs indefinitely.
The only way to recover is a forced power-off by holding the power button.

This has happened multiple times and is reproducible enough to make the client difficult to use.

Interesting diagnostic information

During the most recent occurrence I inspected the processes before rebooting.

AmneziaVPN-service was running as PID 314, with tun2socks as its child:

PID   PPID  STAT  COMMAND
314      1  Us    /Applications/AmneziaVPN.app/Contents/MacOS/AmneziaVPN-service
2988   314  U     /Applications/AmneziaVPN.app/Contents/MacOS/tun2socks -device tun://utun22 ...

The important part is that tun2socks was in uninterruptible wait (U) while using utun22.

sample was still able to collect a sample from tun2socks:

sudo sample 2988 3

I then sent SIGKILL to tun2socks:

sudo kill -9 2988

but the process did not disappear. It changed to an exiting state and remained stuck:

PID   PPID  STAT  COMMAND
2988   314  ?E    (tun2socks)

I then sent SIGKILL to AmneziaVPN-service:

sudo kill -9 314

After that, both processes remained present in exiting state:

PID   PPID  STAT  COMMAND
314      1  ?Es   (AmneziaVPN-servi)
2988   314  ?E    (tun2socks)

They could not be killed even with SIGKILL.

At this point pfctl also hangs and macOS cannot complete a normal reboot.

Possible direction

This looks like the problem is below the AmneziaVPN GUI level.

In particular, tun2socks appears to get stuck in the kernel while tearing down or interacting with the utun interface. Once this happens, both tun2socks and its parent AmneziaVPN-service cannot finish exiting even after SIGKILL.

The fact that pfctl also blocks and macOS cannot shut down suggests a deadlock or blocked kernel path somewhere around utun / networking teardown rather than simply stale routes or firewall rules.

I can reproduce/test this further and provide the sample output or other diagnostics if you tell me what would be useful to collect before the next forced reboot.

@dmshem commented on GitHub (Aug 21, 2026): ### Environment * macOS Tahoe 26.1 * AmneziaVPN 5.0.0.5 * VLESS / XRay * AmneziaVPN installed from the `.pkg` distribution ### Problem AmneziaVPN occasionally enters a system-level deadlock, usually after the Mac has gone to sleep while VPN is connected. After waking up: * AmneziaVPN hangs on **Disconnect** indefinitely. * Network connectivity becomes unusable. * Killing the AmneziaVPN GUI does not restore networking. * `pfctl` itself hangs, for example: ```bash sudo pfctl -s info ``` * A normal macOS reboot/shutdown also hangs indefinitely. * The only way to recover is a forced power-off by holding the power button. This has happened multiple times and is reproducible enough to make the client difficult to use. ### Interesting diagnostic information During the most recent occurrence I inspected the processes before rebooting. `AmneziaVPN-service` was running as PID 314, with `tun2socks` as its child: ```text PID PPID STAT COMMAND 314 1 Us /Applications/AmneziaVPN.app/Contents/MacOS/AmneziaVPN-service 2988 314 U /Applications/AmneziaVPN.app/Contents/MacOS/tun2socks -device tun://utun22 ... ``` The important part is that `tun2socks` was in uninterruptible wait (`U`) while using `utun22`. `sample` was still able to collect a sample from `tun2socks`: ```bash sudo sample 2988 3 ``` I then sent SIGKILL to `tun2socks`: ```bash sudo kill -9 2988 ``` but the process did **not** disappear. It changed to an exiting state and remained stuck: ```text PID PPID STAT COMMAND 2988 314 ?E (tun2socks) ``` I then sent SIGKILL to `AmneziaVPN-service`: ```bash sudo kill -9 314 ``` After that, **both processes remained present in exiting state**: ```text PID PPID STAT COMMAND 314 1 ?Es (AmneziaVPN-servi) 2988 314 ?E (tun2socks) ``` They could not be killed even with SIGKILL. At this point `pfctl` also hangs and macOS cannot complete a normal reboot. ### Possible direction This looks like the problem is below the AmneziaVPN GUI level. In particular, `tun2socks` appears to get stuck in the kernel while tearing down or interacting with the `utun` interface. Once this happens, both `tun2socks` and its parent `AmneziaVPN-service` cannot finish exiting even after SIGKILL. The fact that `pfctl` also blocks and macOS cannot shut down suggests a deadlock or blocked kernel path somewhere around `utun` / networking teardown rather than simply stale routes or firewall rules. I can reproduce/test this further and provide the `sample` output or other diagnostics if you tell me what would be useful to collect before the next forced reboot.

bsakaguchi commented

2026-10-13 20:40:30 +00:00

Author

Owner

@pavelgup-blip commented on GitHub (Sep 9, 2026):

Same deadlock here, on a desktop Mac with no sleep and no lid involved, ending in
the same kernel panic @RiRion described above, twice in three days. What I can add to
the thread is the part that is missing from a panic log: three watchdog stackshots
taken before the panic, one minute apart, with an identical lock graph that names the
threads holding the locks. I also still reproduce this on 5.0.1.5, as does #3130,
so the current release does not fix it.

Environment

AmneziaVPN 5.0.1 build 5 (package 5.0.1.5, released 2026-08-21 — the latest)
macOS 26.6.2 (25G83), Darwin 25.6.0, RELEASE_ARM64_T8132
Mac16,10 (Mac mini M4), 16 GB, Apple Silicon
Protocol: VLESS / XRay, tun2socks running
Site split tunneling enabled (Conf.routeMode=2, Conf.sitesSplitTunnelingEnabled=1),
115 host /32 routes in the table
KillSwitch active (the amn.* pf anchors, pf.token present)
No NetworkExtension involved: systemextensionsctl list reports 0 extensions,
scutil --nc list is empty

What is new relative to this thread

Not sleep, not lid close. This is a Mac mini, there is no lid. Both panic logs
report Sleep : 0x00000000 and Wake : 0x00000000, so the machine never slept
during those boot sessions. VLESS was simply switched on during normal desktop use:
processUptime in the stackshots puts the start of tun2socks at 15:59:36, and the
first stuck process appears at 16:07:36, about eight minutes into a working tunnel.
The escalation to the panic is captured step by step. Restarting configd does
not help while the lock owners are alive, and the stackshots show it directly: at
16:10:36 configd is pid 353, the watchdog crashes it, and the next stackshot at
16:11:46 already shows the fresh pid 8921 stuck in the very same lock graph. After
the third timeout the kernel panics: userspace watchdog timeout: no successful checkins from configd (2 induced crashes) in 180 seconds. Two panics so far,
2026-09-07 01:04:40 and 2026-09-09 16:14:33, both with a hard reboot.
In the 2026-09-07 panic the induced crash did not even take effect: the panic string
reads configd (1 induced crashes) together with configd has not exited since first loaded. That matches @dmshem's finding that the stuck processes survive SIGKILL.
Still present in 5.0.1.5. Every report in this thread is on 5.0.0.5. #3130 is the
other 5.0.1.5 case, and it is on the same macOS build as mine (26.6.2, 25G83) with
the same tun2socks/utun22 path.
tun2socks is a lock owner, not only a victim. In my stackshots a kernel
thread is blocked on a mutex owned by a tun2socks thread.
Thread bloat is not the condition. The two panics on this machine sit on opposite
ends of that scale. On 2026-09-07 AmneziaVPN-service had 307 threads (74 MB,
167k page faults) and tun2socks 17, close to the 259 threads in the original report.
On 2026-09-09 the same deadlock formed with 52 threads in the service (33 MB) and
4 in tun2socks. Same lock cycle either way, so the thread and memory growth is a
separate leak rather than a precondition.
Three pfctl processes are wedged (pids 8897, 8903, 8905), each 1/1 thread in
TH_WAIT, TH_UNINT, with identical pids and tids across all three stackshots taken
a minute apart. By uptime they were started at 16:07:36, 16:08:36 and 16:09:05, so
roughly once a minute, and none of them ever returned. The last one starts about
30 seconds before configd misses its first checkin. This matches @dmshem's
observation that pfctl hangs, and it puts the pf/KillSwitch path inside the window.
Both configd queues stall, in sequence: IPMonitorQueue in the first stackshot,
then IPConfigurationAgentQueue in the next two.
The onset lines up with a burst of outbound connections through the tunnel, which
is what #3130 reports too. At 16:07:36 Steam started (steam_osx plus five
Steam Helper processes, all with processUptime 180 s in the 16:10:36 stackshot),
and the first pfctl hangs in the same second. #3130 describes the same failure on
the same macOS build and client version, with Docker Desktop opening many outbound
connections through utun22 as the trigger. That is a correlation from one incident
here, not proof, but "an application opens a lot of sockets through the tunnel" fits
both reports better than sleep or lid close does.

Lock graph, identical in all three stackshots

Taken from stackshot.processByPid["0"].waitInfo and turnstileInfo, with owner tids
resolved to processes:

Waiter (kernel_task)	Waiting on	Owner thread	Owner process
1270, 2221, 6541, 187619	kernel mutex `0xfffffe000c5d3f50`	8445	AmneziaVPN-service (pid 599)
1272	kernel mutex `0xa98f7ac0af8d3041`	187923	tun2socks (pid 8555)
194180	krwlock `0xfffffe000c606830` for writing	1272	kernel_task

Both owner threads are themselves in TH_WAIT, TH_UNINT, so nothing ever drains. The
same tids and the same lock addresses appear in all three stackshots, 16:10:36, 16:11:46
and 16:12:46, across a restart of configd in between.

configd's stack points at the stock /System/Library/SystemConfiguration/IPConfiguration.bundle
(UUID 7A775BAB-74AD-3FCF-B359-A71072AFD390, verified with dwarfdump), so no
third-party code is loaded inside configd itself. It is a victim, not the cause.

Timeline, 2026-09-09

Times before 16:10:36 are reconstructed from processUptime in the first stackshot.

Time (MSK)	Event
15:59:36	`tun2socks` (pid 8555) starts, i.e. VLESS is connected
16:07:36	Steam starts (6 processes); `pfctl` pid 8897 starts and never returns
16:08:36	`pfctl` pid 8903 starts and never returns
16:09:05	`pfctl` pid 8905 starts and never returns
~16:09:36	`configd` misses its first checkin
16:10:36	stackshot 1: `configd` pid `353`, `IPMonitorQueue` unresponsive, 0 induced crashes, "has not exited since first loaded"
~16:11:46	watchdogd crashes `configd`; stackshot 2 already shows the new pid `8921`, now `IPConfigurationAgentQueue` unresponsive
16:12:46	stackshot 3: pid `8921` still stuck, 1 induced crash, 120 s since last checkin, "60 seconds since last crashed by watchdogd"
16:14:33	kernel panic, 2 induced crashes, 180 s without a checkin

The 2026-09-07 panic has the same signature with 1 induced crash, after an uptime of
~3.4 days, and AmneziaVPN had logged a hang 11 minutes earlier. Its stackshot is
truncated ("missing stackshot buffer or size"), so no uptimes can be recovered from it.

Uninterruptible processes at 16:12:46

kernel_task           pid      0   635/652 threads
AmneziaVPN-service    pid    599    38/48 threads
tun2socks             pid   8555      4/4 threads
pfctl                 pid   8897      1/1 thread
pfctl                 pid   8903      1/1 thread
pfctl                 pid   8905      1/1 thread
mDNSResponder         pid    488      2/5 threads
configd               pid   8921      2/8 threads   (IPConfigurationAgentQueue, main)

Everything else on the machine that touches the network is stuck behind the same graph.

AmneziaWG is unaffected

The same machine with the same split-tunneling settings runs AmneziaWG
(amneziawg-go) for days without a problem. tun2socks is not spawned in that mode.
That narrows the fault to the XRay/tun2socks path rather than to routing or pf alone.

Artifacts

Attaching the three configd watchdog stackshots and both panic logs, plus a plain
text summary of the lock graph. No IP addresses, hostnames or site lists are included.

amnezia-panic-artifacts.zip

@pavelgup-blip commented on GitHub (Sep 9, 2026): Same deadlock here, on a desktop Mac with **no sleep and no lid involved**, ending in the same kernel panic @RiRion described above, twice in three days. What I can add to the thread is the part that is missing from a panic log: **three watchdog stackshots taken before the panic**, one minute apart, with an identical lock graph that names the threads holding the locks. I also still reproduce this on **5.0.1.5**, as does #3130, so the current release does not fix it. ## Environment - AmneziaVPN **5.0.1 build 5** (package 5.0.1.5, released 2026-08-21 — the latest) - macOS **26.6.2 (25G83)**, Darwin 25.6.0, `RELEASE_ARM64_T8132` - **Mac16,10 (Mac mini M4)**, 16 GB, Apple Silicon - Protocol: **VLESS / XRay**, `tun2socks` running - Site split tunneling enabled (`Conf.routeMode=2`, `Conf.sitesSplitTunnelingEnabled=1`), 115 host `/32` routes in the table - KillSwitch active (the `amn.*` pf anchors, `pf.token` present) - No NetworkExtension involved: `systemextensionsctl list` reports 0 extensions, `scutil --nc list` is empty ## What is new relative to this thread 1. **Not sleep, not lid close.** This is a Mac mini, there is no lid. Both panic logs report `Sleep : 0x00000000` and `Wake : 0x00000000`, so the machine never slept during those boot sessions. VLESS was simply switched on during normal desktop use: `processUptime` in the stackshots puts the start of `tun2socks` at 15:59:36, and the first stuck process appears at 16:07:36, about eight minutes into a working tunnel. 2. **The escalation to the panic is captured step by step.** Restarting `configd` does not help while the lock owners are alive, and the stackshots show it directly: at 16:10:36 `configd` is pid `353`, the watchdog crashes it, and the next stackshot at 16:11:46 already shows the fresh pid `8921` stuck in the very same lock graph. After the third timeout the kernel panics: `userspace watchdog timeout: no successful checkins from configd (2 induced crashes) in 180 seconds`. Two panics so far, 2026-09-07 01:04:40 and 2026-09-09 16:14:33, both with a hard reboot. In the 2026-09-07 panic the induced crash did not even take effect: the panic string reads `configd (1 induced crashes)` together with `configd has not exited since first loaded`. That matches @dmshem's finding that the stuck processes survive `SIGKILL`. 3. **Still present in 5.0.1.5.** Every report in this thread is on 5.0.0.5. #3130 is the other 5.0.1.5 case, and it is on the same macOS build as mine (26.6.2, 25G83) with the same `tun2socks`/`utun22` path. 4. **`tun2socks` is a lock owner, not only a victim.** In my stackshots a kernel thread is blocked on a mutex owned by a `tun2socks` thread. 5. **Thread bloat is not the condition.** The two panics on this machine sit on opposite ends of that scale. On 2026-09-07 `AmneziaVPN-service` had **307 threads** (74 MB, 167k page faults) and `tun2socks` 17, close to the 259 threads in the original report. On 2026-09-09 the same deadlock formed with **52 threads** in the service (33 MB) and **4** in `tun2socks`. Same lock cycle either way, so the thread and memory growth is a separate leak rather than a precondition. 6. **Three `pfctl` processes are wedged** (pids 8897, 8903, 8905), each 1/1 thread in `TH_WAIT, TH_UNINT`, with identical pids and tids across all three stackshots taken a minute apart. By uptime they were started at 16:07:36, 16:08:36 and 16:09:05, so roughly once a minute, and none of them ever returned. The last one starts about 30 seconds before `configd` misses its first checkin. This matches @dmshem's observation that `pfctl` hangs, and it puts the pf/KillSwitch path inside the window. 7. **Both configd queues stall, in sequence:** `IPMonitorQueue` in the first stackshot, then `IPConfigurationAgentQueue` in the next two. 8. **The onset lines up with a burst of outbound connections through the tunnel, which is what #3130 reports too.** At 16:07:36 Steam started (`steam_osx` plus five `Steam Helper` processes, all with `processUptime` 180 s in the 16:10:36 stackshot), and the first `pfctl` hangs in the same second. #3130 describes the same failure on the same macOS build and client version, with Docker Desktop opening many outbound connections through `utun22` as the trigger. That is a correlation from one incident here, not proof, but "an application opens a lot of sockets through the tunnel" fits both reports better than sleep or lid close does. ## Lock graph, identical in all three stackshots Taken from `stackshot.processByPid["0"].waitInfo` and `turnstileInfo`, with owner tids resolved to processes: | Waiter (kernel_task) | Waiting on | Owner thread | Owner process | |---|---|---|---| | 1270, 2221, 6541, 187619 | kernel mutex `0xfffffe000c5d3f50` | 8445 | **AmneziaVPN-service** (pid 599) | | 1272 | kernel mutex `0xa98f7ac0af8d3041` | 187923 | **tun2socks** (pid 8555) | | 194180 | krwlock `0xfffffe000c606830` for writing | 1272 | kernel_task | Both owner threads are themselves in `TH_WAIT, TH_UNINT`, so nothing ever drains. The same tids and the same lock addresses appear in all three stackshots, 16:10:36, 16:11:46 and 16:12:46, across a restart of `configd` in between. `configd`'s stack points at the stock `/System/Library/SystemConfiguration/IPConfiguration.bundle` (UUID `7A775BAB-74AD-3FCF-B359-A71072AFD390`, verified with `dwarfdump`), so no third-party code is loaded inside `configd` itself. It is a victim, not the cause. ## Timeline, 2026-09-09 Times before 16:10:36 are reconstructed from `processUptime` in the first stackshot. | Time (MSK) | Event | |---|---| | 15:59:36 | `tun2socks` (pid 8555) starts, i.e. VLESS is connected | | 16:07:36 | Steam starts (6 processes); `pfctl` pid 8897 starts and never returns | | 16:08:36 | `pfctl` pid 8903 starts and never returns | | 16:09:05 | `pfctl` pid 8905 starts and never returns | | ~16:09:36 | `configd` misses its first checkin | | 16:10:36 | stackshot 1: `configd` pid `353`, `IPMonitorQueue` unresponsive, 0 induced crashes, "has not exited since first loaded" | | ~16:11:46 | watchdogd crashes `configd`; stackshot 2 already shows the new pid `8921`, now `IPConfigurationAgentQueue` unresponsive | | 16:12:46 | stackshot 3: pid `8921` still stuck, 1 induced crash, 120 s since last checkin, "60 seconds since last crashed by watchdogd" | | 16:14:33 | kernel panic, 2 induced crashes, 180 s without a checkin | The 2026-09-07 panic has the same signature with 1 induced crash, after an uptime of ~3.4 days, and `AmneziaVPN` had logged a hang 11 minutes earlier. Its stackshot is truncated ("missing stackshot buffer or size"), so no uptimes can be recovered from it. ## Uninterruptible processes at 16:12:46 ``` kernel_task pid 0 635/652 threads AmneziaVPN-service pid 599 38/48 threads tun2socks pid 8555 4/4 threads pfctl pid 8897 1/1 thread pfctl pid 8903 1/1 thread pfctl pid 8905 1/1 thread mDNSResponder pid 488 2/5 threads configd pid 8921 2/8 threads (IPConfigurationAgentQueue, main) ``` Everything else on the machine that touches the network is stuck behind the same graph. ## AmneziaWG is unaffected The same machine with the same split-tunneling settings runs AmneziaWG (`amneziawg-go`) for days without a problem. `tun2socks` is not spawned in that mode. That narrows the fault to the XRay/`tun2socks` path rather than to routing or pf alone. ## Artifacts Attaching the three `configd` watchdog stackshots and both panic logs, plus a plain text summary of the lock graph. No IP addresses, hostnames or site lists are included. [amnezia-panic-artifacts.zip](https://github.com/user-attachments/files/32014790/amnezia-panic-artifacts.zip)

bsakaguchi commented

2026-10-13 20:40:31 +00:00

Author

Owner

@HeisLuka commented on GitHub (Sep 9, 2026):

I rechecked this against current dev, together with the newer reports in this thread and #3130.

I think the scope of this issue can now be narrowed more accurately:

lid close / sleep is a trigger, but it is no longer supported as a required root cause.

The common failure domain looks much more like the macOS XRay tun2socks / utun data path, with a separate Amnezia teardown/liveness problem that makes the resulting failure effectively unrecoverable.

1. Sleep is not required anymore

The original report strongly correlated the failure with lid close.

However, the newer report on macOS 26.6.2 / Amnezia 5.0.1.5 reproduces the same class of lock graph on a Mac mini with no lid and no sleep transition.

In that capture:

tun2socks
AmneziaVPN-service
pfctl
kernel networking threads
configd

remain blocked across multiple watchdog stackshots.

Importantly, tun2socks is present as a lock owner, not only as a process waiting behind another owner.

There is also #3130, where another 5.0.1.5 system reaches an explicit lock cycle involving:

com.docker.backend
-> kernel_task / dlil_input_utun22
-> com.docker.backend

immediately after a burst of outbound connections through Amnezia's utun22.

So I would now describe sleep as one way of stressing/reconfiguring the same networking path, rather than the primary explanation.

2. Current Amnezia code has an independently reproducible liveness problem around `pfctl`

Current MacOSFirewall::execute() runs firewall operations through QProcess and then calls:

process.waitForFinished()

without a timeout.

That means essentially every synchronous PF operation can block the caller indefinitely if pfctl itself becomes stuck in the kernel.

This affects operations including:

install / uninstall
anchor enable/disable
anchor table replacement
PF state checks
root-anchor priority repair

The newer diagnostic in this issue shows exactly that kind of state: several pfctl processes remain wedged while the networking lock cycle persists.

I don't think the source proves that PF creates the original kernel deadlock.

But it does prove that Amnezia currently has no bounded failure boundary once pfctl enters that state.

The invariant should be:

external firewall command may fail or time out
!=
privileged Amnezia control path may block forever

3. XRay teardown order makes recovery worse

Current XrayProtocol::stop() performs roughly:

disableKillSwitch()
    wait synchronously

restore IPv6 routing
    wait synchronously

restore resolvers
    wait synchronously

deleteTun()
    wait synchronously

xrayStop()
    wait synchronously

only then:
    terminate tun2socks
    wait
    kill tun2socks if needed

So if the system is already in the state reported here:

pfctl blocked in kernel
tun2socks still owns/uses utun22

pressing Disconnect does:

XrayProtocol::stop()
        ↓
disableKillSwitch()
        ↓
pfctl
        ↓
unbounded wait

and never reaches:

terminate / kill tun2socks

This matches the user-visible failure very closely:

Disconnect spins forever
pfctl hangs
tun2socks remains alive
network does not recover
normal shutdown also hangs

Even if the original kernel lock cycle originates outside Amnezia's Qt control logic, the teardown ordering currently prevents Amnezia from making a best-effort recovery.

I would invert the dependency:

stop/quiesce packet producer first
-> bounded cleanup of routing/firewall/DNS

A kill switch remaining active for a short period while the data-plane process is being stopped is also safer than keeping tun2socks alive because PF cleanup did not return.

More generally, one failed cleanup stage should not prevent all subsequent cleanup stages from being attempted.

4. The sleep path is still a useful aggravating factor

Current MacOSNetworkWatcher receives:

kIOMessageSystemWillSleep

and immediately acknowledges it with IOAllowPowerChange().

It does not first quiesce XRay, stop tun2socks, or drain the utun data path.

On wake it later emits the normal wakeup signal.

That is not sufficient to explain the Mac mini reproduction, so I would not call it the root cause.

But it means an active tun2socks / utun22 session is intentionally allowed to cross the macOS sleep transition.

Given the lock graphs reported here, this is worth treating as a separate lifecycle stress case.

For XRay on macOS, a safer lifecycle may be:

WILL_SLEEP
-> quiesce/stop tun2socks
-> acknowledge sleep

HAS_POWERED_ON
-> wait for physical networking readiness
-> recreate data plane

rather than leaving the userspace TUN stack live through the transition.

5. I don't think current Amnezia source is enough to identify the first kernel lock owner with certainty

At this point I would distinguish two questions.

The first one is:

What creates the original macOS kernel lock cycle?

The evidence now points strongly toward the external tun2socks / utun path under network churn, but I don't think the Amnezia repository alone proves whether the underlying fault is in:

tun2socks
gVisor/netstack used by tun2socks
macOS 26.x utun/networking
interaction with another networking consumer

The second question is:

Why can't Amnezia recover once the networking path is wedged?

That part is directly visible in current source:

unbounded synchronous pfctl
+
PF cleanup before tun2socks termination
+
serial synchronous teardown

So I would treat those as actionable client bugs even before the lower-level kernel trigger is fully identified.

6. Current `dev` is still pinned to tun2socks 2.6.0

The current Conan recipe contains:

name = "tun2socks"
version = "2.6.0"

Upstream tun2socks 2.7.0 has been available since July.

The 2.6.0 -> 2.7.0 range includes, among other changes:

gVisor dependency updates
Darwin socket-option changes
core networking changes

I am not claiming that 2.7.0 fixes this issue.

But because the evidence now points toward the tun2socks/utun data path, an A/B reproduction between the exact same Amnezia build using:

tun2socks 2.6.0
vs
tun2socks 2.7.0

would be a high-value isolation test.

Suggested containment / diagnostic direction

I would make the following properties explicit:

1. No pfctl subprocess may block the service indefinitely.

2. Data-plane shutdown must still be attempted if firewall,
   DNS or route cleanup fails.

3. tun2socks should be quiesced before cleanup operations
   which themselves depend on a healthy network kernel path.

4. Sleep/wake should have an explicit XRay lifecycle instead
   of leaving the userspace TUN stack active through sleep.

5. tun2socks version should be part of the crash/deadlock
   diagnostics.

6. Kernel/system deadlock reports involving utun22 should be
   correlated as one problem family rather than separate
   "lid close", "Steam" and "Docker" bugs.

Regression / stress tests

I would specifically test:

A. macOS 26.6.x + XRay + tun2socks:
   sustained high TCP connection churn.

B. Same test with large bursts of new connections
   (Docker / Steam-like workload).

C. Kill switch ON vs OFF.

D. Site split tunneling ON vs OFF.

E. tun2socks 2.6.0 vs 2.7.0.

F. Active high-traffic tunnel -> sleep -> wake.

G. Mock/stall pfctl during Disconnect:
   Disconnect must have a bounded completion path and
   tun2socks termination must still be attempted.

H. Make any one teardown stage fail:
   all independent remaining cleanup stages must still run.

For a reproduced kernel hang, the most useful artifact is probably a stackshot/sysdiagnose showing the lock owners around:

utun22
dlil_input_utun22
tun2socks
pfctl
AmneziaVPN-service

rather than only the GUI log.

So my current classification would be:

confirmed, but broader than the original title.

The likely common fault domain is the macOS XRay tun2socks / utun path.

The exact first kernel-level cause still needs runtime isolation, but the current Amnezia client has a source-confirmed recovery/liveness bug: firewall teardown is unbounded and is made a prerequisite for stopping tun2socks, so a networking/PF stall can permanently block Disconnect and prevent best-effort recovery.

@HeisLuka commented on GitHub (Sep 9, 2026): I rechecked this against current `dev`, together with the newer reports in this thread and #3130. I think the scope of this issue can now be narrowed more accurately: **lid close / sleep is a trigger, but it is no longer supported as a required root cause.** The common failure domain looks much more like the macOS XRay `tun2socks` / `utun` data path, with a separate Amnezia teardown/liveness problem that makes the resulting failure effectively unrecoverable. ### 1. Sleep is not required anymore The original report strongly correlated the failure with lid close. However, the newer report on macOS 26.6.2 / Amnezia 5.0.1.5 reproduces the same class of lock graph on a Mac mini with no lid and no sleep transition. In that capture: ```text tun2socks AmneziaVPN-service pfctl kernel networking threads configd ``` remain blocked across multiple watchdog stackshots. Importantly, `tun2socks` is present as a lock owner, not only as a process waiting behind another owner. There is also #3130, where another 5.0.1.5 system reaches an explicit lock cycle involving: ```text com.docker.backend -> kernel_task / dlil_input_utun22 -> com.docker.backend ``` immediately after a burst of outbound connections through Amnezia's `utun22`. So I would now describe sleep as one way of stressing/reconfiguring the same networking path, rather than the primary explanation. ### 2. Current Amnezia code has an independently reproducible liveness problem around `pfctl` Current `MacOSFirewall::execute()` runs firewall operations through `QProcess` and then calls: ```cpp process.waitForFinished() ``` without a timeout. That means essentially every synchronous PF operation can block the caller indefinitely if `pfctl` itself becomes stuck in the kernel. This affects operations including: ```text install / uninstall anchor enable/disable anchor table replacement PF state checks root-anchor priority repair ``` The newer diagnostic in this issue shows exactly that kind of state: several `pfctl` processes remain wedged while the networking lock cycle persists. I don't think the source proves that PF creates the original kernel deadlock. But it does prove that Amnezia currently has no bounded failure boundary once `pfctl` enters that state. The invariant should be: ```text external firewall command may fail or time out != privileged Amnezia control path may block forever ``` ### 3. XRay teardown order makes recovery worse Current `XrayProtocol::stop()` performs roughly: ```text disableKillSwitch() wait synchronously restore IPv6 routing wait synchronously restore resolvers wait synchronously deleteTun() wait synchronously xrayStop() wait synchronously only then: terminate tun2socks wait kill tun2socks if needed ``` So if the system is already in the state reported here: ```text pfctl blocked in kernel tun2socks still owns/uses utun22 ``` pressing Disconnect does: ```text XrayProtocol::stop() ↓ disableKillSwitch() ↓ pfctl ↓ unbounded wait ``` and never reaches: ```text terminate / kill tun2socks ``` This matches the user-visible failure very closely: ```text Disconnect spins forever pfctl hangs tun2socks remains alive network does not recover normal shutdown also hangs ``` Even if the original kernel lock cycle originates outside Amnezia's Qt control logic, the teardown ordering currently prevents Amnezia from making a best-effort recovery. I would invert the dependency: ```text stop/quiesce packet producer first -> bounded cleanup of routing/firewall/DNS ``` A kill switch remaining active for a short period while the data-plane process is being stopped is also safer than keeping `tun2socks` alive because PF cleanup did not return. More generally, one failed cleanup stage should not prevent all subsequent cleanup stages from being attempted. ### 4. The sleep path is still a useful aggravating factor Current `MacOSNetworkWatcher` receives: ```text kIOMessageSystemWillSleep ``` and immediately acknowledges it with `IOAllowPowerChange()`. It does not first quiesce XRay, stop `tun2socks`, or drain the `utun` data path. On wake it later emits the normal wakeup signal. That is not sufficient to explain the Mac mini reproduction, so I would not call it the root cause. But it means an active `tun2socks` / `utun22` session is intentionally allowed to cross the macOS sleep transition. Given the lock graphs reported here, this is worth treating as a separate lifecycle stress case. For XRay on macOS, a safer lifecycle may be: ```text WILL_SLEEP -> quiesce/stop tun2socks -> acknowledge sleep HAS_POWERED_ON -> wait for physical networking readiness -> recreate data plane ``` rather than leaving the userspace TUN stack live through the transition. ### 5. I don't think current Amnezia source is enough to identify the first kernel lock owner with certainty At this point I would distinguish two questions. The first one is: ```text What creates the original macOS kernel lock cycle? ``` The evidence now points strongly toward the external `tun2socks` / `utun` path under network churn, but I don't think the Amnezia repository alone proves whether the underlying fault is in: ```text tun2socks gVisor/netstack used by tun2socks macOS 26.x utun/networking interaction with another networking consumer ``` The second question is: ```text Why can't Amnezia recover once the networking path is wedged? ``` That part *is* directly visible in current source: ```text unbounded synchronous pfctl + PF cleanup before tun2socks termination + serial synchronous teardown ``` So I would treat those as actionable client bugs even before the lower-level kernel trigger is fully identified. ### 6. Current `dev` is still pinned to tun2socks 2.6.0 The current Conan recipe contains: ```python name = "tun2socks" version = "2.6.0" ``` Upstream tun2socks 2.7.0 has been available since July. The 2.6.0 -> 2.7.0 range includes, among other changes: ```text gVisor dependency updates Darwin socket-option changes core networking changes ``` I am **not** claiming that 2.7.0 fixes this issue. But because the evidence now points toward the `tun2socks`/utun data path, an A/B reproduction between the exact same Amnezia build using: ```text tun2socks 2.6.0 vs tun2socks 2.7.0 ``` would be a high-value isolation test. ### Suggested containment / diagnostic direction I would make the following properties explicit: ```text 1. No pfctl subprocess may block the service indefinitely. 2. Data-plane shutdown must still be attempted if firewall, DNS or route cleanup fails. 3. tun2socks should be quiesced before cleanup operations which themselves depend on a healthy network kernel path. 4. Sleep/wake should have an explicit XRay lifecycle instead of leaving the userspace TUN stack active through sleep. 5. tun2socks version should be part of the crash/deadlock diagnostics. 6. Kernel/system deadlock reports involving utun22 should be correlated as one problem family rather than separate "lid close", "Steam" and "Docker" bugs. ``` ### Regression / stress tests I would specifically test: ```text A. macOS 26.6.x + XRay + tun2socks: sustained high TCP connection churn. B. Same test with large bursts of new connections (Docker / Steam-like workload). C. Kill switch ON vs OFF. D. Site split tunneling ON vs OFF. E. tun2socks 2.6.0 vs 2.7.0. F. Active high-traffic tunnel -> sleep -> wake. G. Mock/stall pfctl during Disconnect: Disconnect must have a bounded completion path and tun2socks termination must still be attempted. H. Make any one teardown stage fail: all independent remaining cleanup stages must still run. ``` For a reproduced kernel hang, the most useful artifact is probably a stackshot/sysdiagnose showing the lock owners around: ```text utun22 dlil_input_utun22 tun2socks pfctl AmneziaVPN-service ``` rather than only the GUI log. So my current classification would be: **confirmed, but broader than the original title.** The likely common fault domain is the macOS XRay `tun2socks` / `utun` path. The exact first kernel-level cause still needs runtime isolation, but the current Amnezia client has a source-confirmed recovery/liveness bug: **firewall teardown is unbounded and is made a prerequisite for stopping `tun2socks`, so a networking/PF stall can permanently block Disconnect and prevent best-effort recovery.**

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

amnezia-vpn/amnezia-client#1337

No description provided.

Rows
Columns

[GH-ISSUE #2933] Deadlock between AmneziaVPN-service and kernel network locks on lid close (macOS 26.6, app 5.0.0.5, VLESS) — configd killed by WATCHDOG #1337

Summary

Environment

Steps to reproduce

Actual result

Root cause from the stackshot — four-way lock cycle

Additional observations

Likely trigger

Workaround

Attachment

Environment

Problem

Interesting diagnostic information

Possible direction

Environment

What is new relative to this thread

Lock graph, identical in all three stackshots

Timeline, 2026-09-09

Uninterruptible processes at 16:12:46

AmneziaWG is unaffected

Artifacts

1. Sleep is not required anymore

2. Current Amnezia code has an independently reproducible liveness problem around pfctl

3. XRay teardown order makes recovery worse

4. The sleep path is still a useful aggravating factor

5. I don't think current Amnezia source is enough to identify the first kernel lock owner with certainty

6. Current dev is still pinned to tun2socks 2.6.0

Suggested containment / diagnostic direction

Regression / stress tests

2. Current Amnezia code has an independently reproducible liveness problem around `pfctl`

6. Current `dev` is still pinned to tun2socks 2.6.0