[GH-ISSUE #146] peer_remove_after_dead deadlocks on flush_workqueue(wg-kex) waiting for in-flight wg_packet_handshake_send_worker during peer removal #90

New issue

Open

opened 2026-10-13 21:06:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-13 21:06:26 +00:00

Owner

Originally created by @JohnnyVBut on GitHub (Mar 6, 2026).
Original GitHub issue: https://github.com/amnezia-vpn/amneziawg-linux-kernel-module/issues/146

Summary

Calling awg set <iface> peer <pubkey> remove while the interface is actively
handling handshakes causes a kernel deadlock. The process enters D-state
(uninterruptible sleep) in __flush_workqueue and never returns. The deadlock
consistently reproduces within 2–3 add→remove iterations.

Critically, the deadlock blocks all netlink operations for all interfaces
managed by the amneziawg module — including unrelated wg show calls and
even sshd connections.

Environment


OS	Ubuntu 22.04.2 LTS (Jammy Jellyfish)
Kernel	`6.8.0-101-generic #101~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC`
AWG module	`amneziawg v1.0.20251009` (DKMS, `/lib/modules/6.8.0-101-generic/updates/dkms/amneziawg.ko`)
AWG userspace	`amneziawg-tools v1.0.20210914`
Architecture	x86_64

Steps to Reproduce

IFACE=wg11
PUBKEY=XWEMkcDnHn+8a1UaCyvz615UO20Y3pdcDsisCvKg+VE=
ALLOWED_IP=192.168.2.2/32

# Iteration 1
awg set $IFACE peer $PUBKEY allowed-ips $ALLOWED_IP  # ok
awg set $IFACE peer $PUBKEY remove                   # ok

# Iteration 2
awg set $IFACE peer $PUBKEY allowed-ips $ALLOWED_IP  # ok
awg set $IFACE peer $PUBKEY remove                   # ← HANGS FOREVER



The remove command on iteration 2 enters D-state and never completes.
SIGKILL has no effect (process remains in D-state).

Expected Behavior
awg set peer remove completes successfully on every invocation.

Actual Behavior
Process enters D (disk sleep) state inside __flush_workqueue
Process is unkillable (SIGKILL ignored in D-state)
All subsequent netlink operations on all amneziawg interfaces block,
including unrelated wg show wg0 dump (different interface) and sshd
Server requires container restart or reboot to recover
Root Cause Analysis
From /proc/<pid>/stack and the workqueue dump, the call chain is:

awg set peer remove
  → wg_set_device()        [netlink handler]
  → set_peer()
  → wg_peer_remove()
  → peer_remove_after_dead()
  → __flush_workqueue(wg-kex-wg11)   ← BLOCKS HERE

At the time of the deadlock, the wg-kex-wg11 workqueue has
wg_packet_handshake_send_worker in-flight:

workqueue wg-kex-wg11: flags=0x6
  pwq 2: cpus=0 flags=0x4 nice=0 active=1/256 refcnt=2
    in-flight: 0:wg_packet_handshake_send_worker [amneziawg]

peer_remove_after_dead calls flush_workqueue() synchronously, waiting for
all in-flight work to drain. However, wg_packet_handshake_send_worker likely
needs to acquire the peer lock or device lock already held by wg_peer_remove,
creating a classic ABBA deadlock:

wg_peer_remove holds device/peer lock → waits for workqueue to flush
wg_packet_handshake_send_worker (in workqueue) → waits for device/peer lock
Additional Findings
1. The deadlock blocks ALL interfaces, not just the affected one.

Three processes entered D-state simultaneously:

D  awg   3614  awg set wg11 peer ... remove    ← triggering process
D  wg    3616  wg show wg0 dump                ← DIFFERENT interface (wg0)!
D  awg   3617  awg show wg11 dump

wg show wg0 dump (a read-only operation on a different interface) also
blocked. This suggests the deadlock affects the generic netlink family handler
at a global level, not per-device.

2. sshd also entered D-state (PID 3610), indicating network layer
disruption beyond just WireGuard netlink operations.

3. WireGuard 1.0 (upstream kernel module) does NOT reproduce this issue.
The same add→remove pattern with wg set peer remove on a WireGuard 1.0
interface completes indefinitely without hanging.

Kernel Stack Trace
/proc/3614/stack (the hung awg set peer remove process):

[<0>] __flush_workqueue+0x14a/0x3e0
[<0>] peer_remove_after_dead+0x88/0x100 [amneziawg]
[<0>] wg_peer_remove+0x7f/0xb0 [amneziawg]
[<0>] set_peer+0x46b/0x650 [amneziawg]
[<0>] wg_set_device+0x68a/0x870 [amneziawg]
[<0>] genl_family_rcv_msg_doit+0xfa/0x160
[<0>] genl_family_rcv_msg+0x189/0x260
[<0>] genl_rcv_msg+0x4c/0xb0
[<0>] netlink_rcv_skb+0x5d/0x110
[<0>] genl_rcv+0x28/0x50
[<0>] netlink_unicast+0x1f0/0x2e0
[<0>] netlink_sendmsg+0x214/0x470
[<0>] __sys_sendto+0x21e/0x230
[<0>] __x64_sys_sendto+0x24/0x40
[<0>] x64_sys_call+0x1f5c/0x2480
[<0>] do_syscall_64+0x81/0x170
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

/proc/3614/wchan: __flush_workqueue

Suggested Fix Direction
peer_remove_after_dead should not call flush_workqueue() synchronously
while holding locks that wg_packet_handshake_send_worker may need.

Possible approaches:

Use drain_workqueue() with a timeout instead of blocking flush_workqueue()
Cancel pending/in-flight handshake work before flushing (cancel_work_sync)
Release the device lock before flushing, re-acquire after
The upstream WireGuard module (wireguard.ko) handles this correctly —
comparing peer_remove_after_dead implementations between upstream WireGuard
and AmneziaWG may reveal the regression point.

Workaround
Use awg-quick down <iface> && awg-quick up <iface> (full interface restart)
instead of awg set peer remove. This avoids the deadlock but resets all
peer statistics.


---

Diagnostic file is attached

[deadlock_awg.txt](https://github.com/user-attachments/files/25801309/deadlock_awg.txt)

Originally created by @JohnnyVBut on GitHub (Mar 6, 2026). Original GitHub issue: https://github.com/amnezia-vpn/amneziawg-linux-kernel-module/issues/146 ## Summary Calling `awg set <iface> peer <pubkey> remove` while the interface is actively handling handshakes causes a kernel deadlock. The process enters D-state (uninterruptible sleep) in `__flush_workqueue` and never returns. The deadlock consistently reproduces within 2–3 add→remove iterations. Critically, the deadlock **blocks all netlink operations for all interfaces** managed by the `amneziawg` module — including unrelated `wg show` calls and even `sshd` connections. --- ## Environment | | | |---|---| | **OS** | Ubuntu 22.04.2 LTS (Jammy Jellyfish) | | **Kernel** | `6.8.0-101-generic #101~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC` | | **AWG module** | `amneziawg v1.0.20251009` (DKMS, `/lib/modules/6.8.0-101-generic/updates/dkms/amneziawg.ko`) | | **AWG userspace** | `amneziawg-tools v1.0.20210914` | | **Architecture** | x86_64 | --- ## Steps to Reproduce ```bash IFACE=wg11 PUBKEY=XWEMkcDnHn+8a1UaCyvz615UO20Y3pdcDsisCvKg+VE= ALLOWED_IP=192.168.2.2/32 # Iteration 1 awg set $IFACE peer $PUBKEY allowed-ips $ALLOWED_IP # ok awg set $IFACE peer $PUBKEY remove # ok # Iteration 2 awg set $IFACE peer $PUBKEY allowed-ips $ALLOWED_IP # ok awg set $IFACE peer $PUBKEY remove # ← HANGS FOREVER The remove command on iteration 2 enters D-state and never completes. SIGKILL has no effect (process remains in D-state). Expected Behavior awg set peer remove completes successfully on every invocation. Actual Behavior Process enters D (disk sleep) state inside __flush_workqueue Process is unkillable (SIGKILL ignored in D-state) All subsequent netlink operations on all amneziawg interfaces block, including unrelated wg show wg0 dump (different interface) and sshd Server requires container restart or reboot to recover Root Cause Analysis From /proc/<pid>/stack and the workqueue dump, the call chain is: awg set peer remove → wg_set_device() [netlink handler] → set_peer() → wg_peer_remove() → peer_remove_after_dead() → __flush_workqueue(wg-kex-wg11) ← BLOCKS HERE At the time of the deadlock, the wg-kex-wg11 workqueue has wg_packet_handshake_send_worker in-flight: workqueue wg-kex-wg11: flags=0x6 pwq 2: cpus=0 flags=0x4 nice=0 active=1/256 refcnt=2 in-flight: 0:wg_packet_handshake_send_worker [amneziawg] peer_remove_after_dead calls flush_workqueue() synchronously, waiting for all in-flight work to drain. However, wg_packet_handshake_send_worker likely needs to acquire the peer lock or device lock already held by wg_peer_remove, creating a classic ABBA deadlock: wg_peer_remove holds device/peer lock → waits for workqueue to flush wg_packet_handshake_send_worker (in workqueue) → waits for device/peer lock Additional Findings 1. The deadlock blocks ALL interfaces, not just the affected one. Three processes entered D-state simultaneously: D awg 3614 awg set wg11 peer ... remove ← triggering process D wg 3616 wg show wg0 dump ← DIFFERENT interface (wg0)! D awg 3617 awg show wg11 dump wg show wg0 dump (a read-only operation on a different interface) also blocked. This suggests the deadlock affects the generic netlink family handler at a global level, not per-device. 2. sshd also entered D-state (PID 3610), indicating network layer disruption beyond just WireGuard netlink operations. 3. WireGuard 1.0 (upstream kernel module) does NOT reproduce this issue. The same add→remove pattern with wg set peer remove on a WireGuard 1.0 interface completes indefinitely without hanging. Kernel Stack Trace /proc/3614/stack (the hung awg set peer remove process): [<0>] __flush_workqueue+0x14a/0x3e0 [<0>] peer_remove_after_dead+0x88/0x100 [amneziawg] [<0>] wg_peer_remove+0x7f/0xb0 [amneziawg] [<0>] set_peer+0x46b/0x650 [amneziawg] [<0>] wg_set_device+0x68a/0x870 [amneziawg] [<0>] genl_family_rcv_msg_doit+0xfa/0x160 [<0>] genl_family_rcv_msg+0x189/0x260 [<0>] genl_rcv_msg+0x4c/0xb0 [<0>] netlink_rcv_skb+0x5d/0x110 [<0>] genl_rcv+0x28/0x50 [<0>] netlink_unicast+0x1f0/0x2e0 [<0>] netlink_sendmsg+0x214/0x470 [<0>] __sys_sendto+0x21e/0x230 [<0>] __x64_sys_sendto+0x24/0x40 [<0>] x64_sys_call+0x1f5c/0x2480 [<0>] do_syscall_64+0x81/0x170 [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 /proc/3614/wchan: __flush_workqueue Suggested Fix Direction peer_remove_after_dead should not call flush_workqueue() synchronously while holding locks that wg_packet_handshake_send_worker may need. Possible approaches: Use drain_workqueue() with a timeout instead of blocking flush_workqueue() Cancel pending/in-flight handshake work before flushing (cancel_work_sync) Release the device lock before flushing, re-acquire after The upstream WireGuard module (wireguard.ko) handles this correctly — comparing peer_remove_after_dead implementations between upstream WireGuard and AmneziaWG may reveal the regression point. Workaround Use awg-quick down <iface> && awg-quick up <iface> (full interface restart) instead of awg set peer remove. This avoids the deadlock but resets all peer statistics. --- Diagnostic file is attached [deadlock_awg.txt](https://github.com/user-attachments/files/25801309/deadlock_awg.txt)

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

amnezia-vpn/amneziawg-linux-kernel-module#90

No description provided.

Rows
Columns