amnezia-vpn/amnezia-client

Fork 0

mirror of https://github.com/amnezia-vpn/amnezia-client.git synced 2026-10-14 21:34:26 +00:00

[GH-ISSUE #3216] [Windows] Split tunnel driver leaks paged pool on every process start/exit when app split tunneling is disabled #1489

New issue

Open

opened 2026-10-13 20:43:44 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-13 20:43:44 +00:00

Owner

Originally created by @neonatoman on GitHub (Sep 25, 2026).
Original GitHub issue: https://github.com/amnezia-vpn/amnezia-client/issues/3216

Environment

AmneziaVPN 5.0.1.5, bundled mullvad-split-tunnel.sys 1.2.5.0 (service AmneziaVPNSplitTunnel)
Windows 11 Enterprise LTSC 2024 (build 26100), x64
Settings: app split tunneling disabled (appsSplitTunnelingEnabled=false), site split tunneling enabled

What happens

While AmneziaVPN-service is running, kernel paged pool grows without bound. The growth is under
the split tunnel driver's pool tag SPTU (ST_POOL_TAG 'UTPS'), about two allocations per process
lifetime (one on process start, one on exit), and nothing ever frees them until reboot or a restart of the
service.

Measured on 2026-09-25 (pool tags read with NtQuerySystemInformation(SystemPoolTagInformation), same data
as poolmon):

after ~6 days of uptime: SPTU = 6,759,412 live allocations, ~817 MB of paged pool;
the counter grows by the same amount as the number of process start + exit events (checked with batches of
300 cmd.exe /c exit); it never goes down;
Restart-Service AmneziaVPN-service → SPTU drops to 259 within 5 s (paged pool −0.54 GiB), then
immediately starts growing again;
a WPR Pool trace attributes all SPTU allocations that were never freed to mullvad-split-tunnel.sys.

On a developer machine that starts many short-lived processes (build tools, test runners), this is roughly
1 GB of kernel memory per week.

Root cause (from the sources)

The daemon creates the split tunnel manager at startup, whether or not app split tunneling is used:
windowsdaemon.cpp:40
→ initDriver() sends IOCTL_ST_INITIALIZE:
windowssplittunnel.cpp:269.
In the driver, IOCTL_ST_INITIALIZE initializes process management
(ioctl.cpp:914), which
registers a process notify routine. Every process start and exit then allocates a PagedPool record
(procmon.cpp:69,
procmon.cpp:104)
and appends it to EventQueue
(procmon.cpp:126).
The worker that drains and frees the queue is woken only when DispatchingEnabled is set
(procmon.cpp:128).
DispatchingEnabled is set only by IOCTL_ST_REGISTER_PROCESSES
(ioctl.cpp:1365 →
procmon.cpp:392).
Amnezia sends IOCTL_REGISTER_PROCESSES only from WindowsSplitTunnel::start()
(windowssplittunnel.cpp:347),
and start() is called only when m_vpnDisabledApps is not empty
(windowsdaemon.cpp:110).
Otherwise the daemon calls stop() (IOCTL_CLEAR_CONFIGURATION), which the driver rejects in the
INITIALIZED state, where only IOCTL_ST_REGISTER_PROCESSES is accepted
(driverentry.cpp:600). The service log repeats Stopping Split tunnel not successfull on every connect
(windowssplittunnel.cpp:386).

So with app split tunneling disabled, the driver stays INITIALIZED forever and its process-event queue is
never drained.

Steps to reproduce

Install AmneziaVPN 5.0.1.5 on Windows and leave app split tunneling disabled.
Start many short processes, e.g. for /L %i in (1,1,1000) do @cmd /c exit.
Watch the SPTU tag in poolmon (or with any pool tag reader): it grows by about 2 per process and never
shrinks.

Suggested fix

Do not send IOCTL_ST_INITIALIZE until app split tunneling is actually enabled (create the driver lazily in
activateSplitTunnel/run), or
when app split tunneling is disabled, send IOCTL_ST_RESET instead of IOCTL_CLEAR_CONFIGURATION. The
reset tears down the process monitor
(ioctl.cpp:813) and puts the
driver back into STARTED
(ioctl.cpp:843).

(Arguably the driver should also not queue events forever while dispatching is disabled, but the client-side
fix is enough to stop the leak.)

Originally created by @neonatoman on GitHub (Sep 25, 2026). Original GitHub issue: https://github.com/amnezia-vpn/amnezia-client/issues/3216 ### Environment - AmneziaVPN 5.0.1.5, bundled `mullvad-split-tunnel.sys` 1.2.5.0 (service `AmneziaVPNSplitTunnel`) - Windows 11 Enterprise LTSC 2024 (build 26100), x64 - Settings: **app split tunneling disabled** (`appsSplitTunnelingEnabled=false`), site split tunneling enabled ### What happens While `AmneziaVPN-service` is running, kernel paged pool grows without bound. The growth is under the split tunnel driver's pool tag `SPTU` (`ST_POOL_TAG 'UTPS'`), about **two allocations per process lifetime** (one on process start, one on exit), and nothing ever frees them until reboot or a restart of the service. Measured on 2026-09-25 (pool tags read with `NtQuerySystemInformation(SystemPoolTagInformation)`, same data as poolmon): - after ~6 days of uptime: `SPTU` = **6,759,412 live allocations, ~817 MB of paged pool**; - the counter grows by the same amount as the number of process start + exit events (checked with batches of 300 `cmd.exe /c exit`); it never goes down; - `Restart-Service AmneziaVPN-service` → `SPTU` drops to **259** within 5 s (paged pool −0.54 GiB), then immediately starts growing again; - a WPR `Pool` trace attributes all `SPTU` allocations that were never freed to `mullvad-split-tunnel.sys`. On a developer machine that starts many short-lived processes (build tools, test runners), this is roughly 1 GB of kernel memory per week. ### Root cause (from the sources) 1. The daemon creates the split tunnel manager at startup, whether or not app split tunneling is used: [`windowsdaemon.cpp:40`](https://github.com/amnezia-vpn/amnezia-client/blob/94b51df24790bf52427afe82d81c87a95460bdfd/client/platforms/windows/daemon/windowsdaemon.cpp#L40) → `initDriver()` sends `IOCTL_ST_INITIALIZE`: [`windowssplittunnel.cpp:269`](https://github.com/amnezia-vpn/amnezia-client/blob/94b51df24790bf52427afe82d81c87a95460bdfd/client/platforms/windows/daemon/windowssplittunnel.cpp#L269). 2. In the driver, `IOCTL_ST_INITIALIZE` initializes process management ([`ioctl.cpp:914`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/ioctl.cpp#L914)), which registers a process notify routine. Every process start and exit then allocates a `PagedPool` record ([`procmon.cpp:69`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/procmon/procmon.cpp#L69), [`procmon.cpp:104`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/procmon/procmon.cpp#L104)) and appends it to `EventQueue` ([`procmon.cpp:126`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/procmon/procmon.cpp#L126)). The worker that drains and frees the queue is woken only when `DispatchingEnabled` is set ([`procmon.cpp:128`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/procmon/procmon.cpp#L128)). 3. `DispatchingEnabled` is set only by `IOCTL_ST_REGISTER_PROCESSES` ([`ioctl.cpp:1365`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/ioctl.cpp#L1365) → [`procmon.cpp:392`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/procmon/procmon.cpp#L392)). 4. Amnezia sends `IOCTL_REGISTER_PROCESSES` only from `WindowsSplitTunnel::start()` ([`windowssplittunnel.cpp:347`](https://github.com/amnezia-vpn/amnezia-client/blob/94b51df24790bf52427afe82d81c87a95460bdfd/client/platforms/windows/daemon/windowssplittunnel.cpp#L347)), and `start()` is called only when `m_vpnDisabledApps` is not empty ([`windowsdaemon.cpp:110`](https://github.com/amnezia-vpn/amnezia-client/blob/94b51df24790bf52427afe82d81c87a95460bdfd/client/platforms/windows/daemon/windowsdaemon.cpp#L110)). Otherwise the daemon calls `stop()` (`IOCTL_CLEAR_CONFIGURATION`), which the driver rejects in the `INITIALIZED` state, where only `IOCTL_ST_REGISTER_PROCESSES` is accepted ([`driverentry.cpp:600`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/driverentry.cpp#L600)). The service log repeats `Stopping Split tunnel not successfull` on every connect ([`windowssplittunnel.cpp:386`](https://github.com/amnezia-vpn/amnezia-client/blob/94b51df24790bf52427afe82d81c87a95460bdfd/client/platforms/windows/daemon/windowssplittunnel.cpp#L386)). So with app split tunneling disabled, the driver stays `INITIALIZED` forever and its process-event queue is never drained. ### Steps to reproduce 1. Install AmneziaVPN 5.0.1.5 on Windows and leave app split tunneling disabled. 2. Start many short processes, e.g. `for /L %i in (1,1,1000) do @cmd /c exit`. 3. Watch the `SPTU` tag in poolmon (or with any pool tag reader): it grows by about 2 per process and never shrinks. ### Suggested fix - Do not send `IOCTL_ST_INITIALIZE` until app split tunneling is actually enabled (create the driver lazily in `activateSplitTunnel`/`run`), **or** - when app split tunneling is disabled, send `IOCTL_ST_RESET` instead of `IOCTL_CLEAR_CONFIGURATION`. The reset tears down the process monitor ([`ioctl.cpp:813`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/ioctl.cpp#L813)) and puts the driver back into `STARTED` ([`ioctl.cpp:843`](https://github.com/mullvad/win-split-tunnel/blob/v1.2.5.0/src/ioctl.cpp#L843)). (Arguably the driver should also not queue events forever while dispatching is disabled, but the client-side fix is enough to stop the leak.)

bsakaguchi commented

2026-10-13 20:43:45 +00:00

Author

Owner

@neonatoman commented on GitHub (Sep 25, 2026):

Checked the latest code as well: the behaviour is unchanged in 5.0.3.0 (latest release, 2026-09-18) and on dev. client/platforms/windows/daemon/ has no commits since 2026-08-01, and the 5.0.1.5...5.0.3.0 diff doesn't touch windowsdaemon.cpp or windowssplittunnel.cpp. None of the open PRs that modify the split tunnel code (#3099, #3112, #3007, #2598) change when the driver is initialized or how it's stopped, so upgrading won't fix this.

Workaround until it's fixed: restart AmneziaVPN-service. Measured: within 5 s of stopping the service, SPTU dropped from 6,759,412 to 259 live allocations. It starts growing again as soon as the service is back up.

@neonatoman commented on GitHub (Sep 25, 2026): Checked the latest code as well: the behaviour is unchanged in **5.0.3.0** (latest release, 2026-09-18) and on `dev`. `client/platforms/windows/daemon/` has no commits since 2026-08-01, and the 5.0.1.5...5.0.3.0 diff doesn't touch `windowsdaemon.cpp` or `windowssplittunnel.cpp`. None of the open PRs that modify the split tunnel code (#3099, #3112, #3007, #2598) change when the driver is initialized or how it's stopped, so upgrading won't fix this. Workaround until it's fixed: restart `AmneziaVPN-service`. Measured: within 5 s of stopping the service, `SPTU` dropped from 6,759,412 to 259 live allocations. It starts growing again as soon as the service is back up.

bsakaguchi commented

2026-10-13 20:43:46 +00:00

Author

Owner

@Art3mis0940 commented on GitHub (Sep 26, 2026):

@neonatoman Have u found stable version of the client? I have same problem and for me the only solution was reinstalling AVPN completely. It works for couple of days.

@Art3mis0940 commented on GitHub (Sep 26, 2026): @neonatoman Have u found stable version of the client? I have same problem and for me the only solution was reinstalling AVPN completely. It works for couple of days.

bsakaguchi commented

2026-10-13 20:43:47 +00:00

Author

Owner

@Art3mis0940 commented on GitHub (Sep 26, 2026):

AmneziaVPN.log
AmneziaVPN-service.log

There are log files. Maybe it will help to solve this issue.

@Art3mis0940 commented on GitHub (Sep 26, 2026): [AmneziaVPN.log](https://github.com/user-attachments/files/32687392/AmneziaVPN.log) [AmneziaVPN-service.log](https://github.com/user-attachments/files/32687395/AmneziaVPN-service.log) There are log files. Maybe it will help to solve this issue.

bsakaguchi commented

2026-10-13 20:43:47 +00:00

Author

Owner

@neonatoman commented on GitHub (Sep 26, 2026):

@Art3mis0940 No, I haven't found one: the split tunnel code is the same from 5.0.1.5 to 5.0.3.0 and on dev, so switching versions won't help.

Your logs, though, show a different failure from the one described here:

You have app split tunneling enabled (App split tunneling is enabled, route mode is 2), and the driver reached STATE_READY (Set Process Config ok). This issue covers the case where that never happens.
In AmneziaVPN-service.log the last line from the daemon during activation is Pushing new Ruleset for Split-Tunnel 3 (06:36:08). In excludeApps() that line is followed by IOCTL_SET_CONFIGURATION and then either New Configuration applied or Failed to set Config err code. Neither line appears in your log. After that the daemon doesn't handle a single command: 25 s later the client gives up (Deactivating, Failed to flush DNS, Failed to clear saved routes), and later connections end with QWindowsPipeWriter: write failed. So the daemon hangs during activation inside excludeApps(), most likely in the call to the split tunnel driver. That would also explain why only a reinstall helps: it removes the driver and loads it again.

Could you check:

Does it keep working with app split tunneling turned off?
When it's broken, what does sc query AmneziaVPNSplitTunnel show, and does Restart-Service AmneziaVPN-service fix it without a reinstall?

I'd suggest opening a separate issue for this hang with your logs and linking it here.

Side note: the service log contains your server's IP address; you may want to edit it out.

@neonatoman commented on GitHub (Sep 26, 2026): @Art3mis0940 No, I haven't found one: the split tunnel code is the same from 5.0.1.5 to 5.0.3.0 and on `dev`, so switching versions won't help. Your logs, though, show a different failure from the one described here: - You have app split tunneling **enabled** (`App split tunneling is enabled, route mode is 2`), and the driver reached `STATE_READY` (`Set Process Config ok`). This issue covers the case where that never happens. - In `AmneziaVPN-service.log` the last line from the daemon during activation is `Pushing new Ruleset for Split-Tunnel 3` (06:36:08). In `excludeApps()` that line is followed by `IOCTL_SET_CONFIGURATION` and then either `New Configuration applied` or `Failed to set Config err code`. Neither line appears in your log. After that the daemon doesn't handle a single command: 25 s later the client gives up (`Deactivating`, `Failed to flush DNS`, `Failed to clear saved routes`), and later connections end with `QWindowsPipeWriter: write failed`. So the daemon hangs during activation inside `excludeApps()`, most likely in the call to the split tunnel driver. That would also explain why only a reinstall helps: it removes the driver and loads it again. Could you check: 1. Does it keep working with **app** split tunneling turned off? 2. When it's broken, what does `sc query AmneziaVPNSplitTunnel` show, and does `Restart-Service AmneziaVPN-service` fix it without a reinstall? I'd suggest opening a separate issue for this hang with your logs and linking it here. Side note: the service log contains your server's IP address; you may want to edit it out.

bsakaguchi commented

2026-10-13 20:43:47 +00:00

Author

Owner

@Art3mis0940 commented on GitHub (Sep 26, 2026):

@neonatoman I have tried with turning off split tunneling. Same thing. And i could not restart the service because it is busy all time. Only rebooting solves my connection.

PS. in log i have replaced my IP - its randomized)))

@Art3mis0940 commented on GitHub (Sep 26, 2026): @neonatoman I have tried with turning off split tunneling. Same thing. And i could not restart the service because it is busy all time. Only rebooting solves my connection. PS. in log i have replaced my IP - its randomized)))

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

amnezia-vpn/amnezia-client#1489

No description provided.

Rows
Columns