morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8410] [Bug] extract-artifact takes 40-49 s for the Node.js release zip on Windows (plain extraction: 1.7 s) #6593

New issue

Open

opened 2026-11-01 21:12:31 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-11-01 21:12:31 +00:00

Owner

Originally created by @Plastic-time on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8410

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

I extracted the official Node.js Windows zip (2,459 entries, 1,994 files, 37.7 MB) with rea extract-artifact. It takes 40–49 s on Windows 11, although the output itself is correct. On the same machine, .NET's ZipFile.ExtractToDirectory extracts the zip in 1.7 s, and hashing every extracted file with SHA-256 takes another 1.5 s.

A CPU profile of the REA run (node --cpu-prof) shows the main thread idle for 81% of the time (34 s of 42 s). That is roughly 17 ms of waiting per file.

Reproduction

Download https://nodejs.org/dist/v24.22.1/node-v24.22.1-win-x64.zip.
Run rea extract-artifact node-v24.22.1-win-x64.zip --format json.

Wall times with the machine otherwise idle:

49.1 s
40.3 s, per --timings
42 s, under --cpu-prof

Every run produced the same result.

Expected behavior

I expected roughly the time of a plain extraction plus hashing, a few seconds. Failing that, a note in the docs that extraction on Windows is this slow.

Impact and scope

Any archive with many small files is affected. Extracting an npm-heavy Electron app or a release zip of this size costs most of a minute.

Measured only on Windows 11 with Microsoft Defender real-time protection on, which is the default. Not measured on Linux, on macOS, or with a Defender exclusion for the output folder, because I don't have admin rights on this machine.

REA version

6.8.0 (rea --version), source commit 6e9ee97a6 on main

Runtime and platform

Node 24.22.1; Windows 11 x64 (10.0.26200); local NTFS SSD; Defender real-time protection on

Target and artifact context

Official Node.js 24.22.1 win-x64 release zip

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Microbenchmark on the same machine and disk: 1,000 files of 16 KiB each, Node 24.22.1, measured as per-file averages.

write                      0.49 ms/file
write + sync               0.81 ms/file
write + sync + reopen/read 9.63 ms/file

fsync is cheap here. Opening the file again right after closing it is what costs.

Analysis or suspected cause

SafeOutputTree.write syncs and closes each file, then immediately reopens it to hash the bytes back (src/artifacts/SafeOutputTree.ts, around lines 323–334). With real-time scanning on Windows, that second open has to wait for the scan of the file that was just written. One option is to hash the bytes back through the original handle opened read/write, before closing it, which would avoid the second open. I have not tried that.

Originally created by @Plastic-time on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8410 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior I extracted the official Node.js Windows zip (2,459 entries, 1,994 files, 37.7 MB) with `rea extract-artifact`. It takes 40–49 s on Windows 11, although the output itself is correct. On the same machine, .NET's `ZipFile.ExtractToDirectory` extracts the zip in 1.7 s, and hashing every extracted file with SHA-256 takes another 1.5 s. A CPU profile of the REA run (`node --cpu-prof`) shows the main thread idle for 81% of the time (34 s of 42 s). That is roughly 17 ms of waiting per file. ### Reproduction 1. Download https://nodejs.org/dist/v24.22.1/node-v24.22.1-win-x64.zip. 2. Run `rea extract-artifact node-v24.22.1-win-x64.zip --format json`. Wall times with the machine otherwise idle: - 49.1 s - 40.3 s, per `--timings` - 42 s, under `--cpu-prof` Every run produced the same result. ### Expected behavior I expected roughly the time of a plain extraction plus hashing, a few seconds. Failing that, a note in the docs that extraction on Windows is this slow. ### Impact and scope Any archive with many small files is affected. Extracting an npm-heavy Electron app or a release zip of this size costs most of a minute. Measured only on Windows 11 with Microsoft Defender real-time protection on, which is the default. Not measured on Linux, on macOS, or with a Defender exclusion for the output folder, because I don't have admin rights on this machine. ### REA version 6.8.0 (`rea --version`), source commit 6e9ee97a6 on `main` ### Runtime and platform Node 24.22.1; Windows 11 x64 (10.0.26200); local NTFS SSD; Defender real-time protection on ### Target and artifact context Official Node.js 24.22.1 win-x64 release zip ### Interface CLI ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence Microbenchmark on the same machine and disk: 1,000 files of 16 KiB each, Node 24.22.1, measured as per-file averages. ```text write 0.49 ms/file write + sync 0.81 ms/file write + sync + reopen/read 9.63 ms/file ``` `fsync` is cheap here. Opening the file again right after closing it is what costs. ### Analysis or suspected cause `SafeOutputTree.write` syncs and closes each file, then immediately reopens it to hash the bytes back (`src/artifacts/SafeOutputTree.ts`, around lines 323–334). With real-time scanning on Windows, that second open has to wait for the scan of the file that was just written. One option is to hash the bytes back through the original handle opened read/write, before closing it, which would avoid the second open. I have not tried that.

bsakaguchi commented

2026-11-01 21:12:32 +00:00

Author

Owner

@kanchanbishtt commented on GitHub (Nov 1, 2026):

I'm looking into this on Windows 11 Pro (10.0.26200, Defender real-time protection on, Node 24.22.1, main at e18594b2). I reproduced it: 48.8 s for the Node.js 24.22.1 win-x64 zip.

The readback reopen does not appear to be the main cost here. A standalone benchmark of reopen vs same-handle readback over the same 1,994 files showed no consistent difference (4.5–5.2 ms/file either way). Timing every filesystem call during extract-artifact (one instrumented run, 30.7 s) shows the time going to per-file destination checks in SafeOutputTree: about 105,500 lstat calls (~53/file, 10.9 s), 11,900 mkdir calls rejected with EEXIST for directories this extraction already created (3.3 s), and 12,400 readdir calls from the case-collision check those rejections trigger (2.0 s). The readback open calls total about 0.6 s.

I plan to open a PR that skips mkdir and the case-collision scan when the exact path is a directory this tree already created, keeping every identity check. A differently-cased segment would still go through mkdir and the scan. Separately: most of the lstat calls re-verify the same ancestor directories several times per file (each level's parent is checked twice, then the lineage is checked twice more in write). Would reducing those be acceptable, or are they intentional for the pathname-race boundary?

@kanchanbishtt commented on GitHub (Nov 1, 2026): I'm looking into this on Windows 11 Pro (10.0.26200, Defender real-time protection on, Node 24.22.1, `main` at `e18594b2`). I reproduced it: 48.8 s for the Node.js 24.22.1 win-x64 zip. The readback reopen does not appear to be the main cost here. A standalone benchmark of reopen vs same-handle readback over the same 1,994 files showed no consistent difference (4.5–5.2 ms/file either way). Timing every filesystem call during `extract-artifact` (one instrumented run, 30.7 s) shows the time going to per-file destination checks in `SafeOutputTree`: about 105,500 `lstat` calls (~53/file, 10.9 s), 11,900 `mkdir` calls rejected with `EEXIST` for directories this extraction already created (3.3 s), and 12,400 `readdir` calls from the case-collision check those rejections trigger (2.0 s). The readback `open` calls total about 0.6 s. I plan to open a PR that skips `mkdir` and the case-collision scan when the exact path is a directory this tree already created, keeping every identity check. A differently-cased segment would still go through `mkdir` and the scan. Separately: most of the `lstat` calls re-verify the same ancestor directories several times per file (each level's parent is checked twice, then the lineage is checked twice more in `write`). Would reducing those be acceptable, or are they intentional for the pathname-race boundary?

bsakaguchi commented

2026-11-01 21:12:32 +00:00

Author

Owner

@Plastic-time commented on GitHub (Nov 1, 2026):

Thanks for digging into this. Your per-call timing is more direct than my benchmark. An isolated close-and-reopen of a just-written file does cost about 8 ms more per file on this machine than reading it back through the same handle (8.8 vs 0.8 ms/file, 1,000 files of 16 KiB, two rounds). But your instrumented run shows that the reopen isn't where REA spends its time, so I'm treating the readback guess in the issue as superseded.

I also tried #8458 here (Windows 11, Defender real-time protection on, Node 24.22.1). I alternated its base a6caa79bf and head bd78275f1 on the same Node.js zip, after one warm-up round each:

build	runs (s)	median
base `a6caa79bf`	42.2, 35.9, 39.4, 38.3, 38.4	38.4
#8458 `bd78275f1`	33.4, 34.0, 36.6, 35.6, 35.4	35.4

That is about 3 s (~8%) here, in line with the mkdir/readdir share you measured. Both builds extracted identical files, with the same 1,994 digests and extraction_id.

@Plastic-time commented on GitHub (Nov 1, 2026): Thanks for digging into this. Your per-call timing is more direct than my benchmark. An isolated close-and-reopen of a just-written file does cost about 8 ms more per file on this machine than reading it back through the same handle (8.8 vs 0.8 ms/file, 1,000 files of 16 KiB, two rounds). But your instrumented run shows that the reopen isn't where REA spends its time, so I'm treating the readback guess in the issue as superseded. I also tried #8458 here (Windows 11, Defender real-time protection on, Node 24.22.1). I alternated its base `a6caa79bf` and head `bd78275f1` on the same Node.js zip, after one warm-up round each: | build | runs (s) | median | | --- | --- | --- | | base `a6caa79bf` | 42.2, 35.9, 39.4, 38.3, 38.4 | 38.4 | | #8458 `bd78275f1` | 33.4, 34.0, 36.6, 35.6, 35.4 | 35.4 | That is about 3 s (~8%) here, in line with the `mkdir`/`readdir` share you measured. Both builds extracted identical files, with the same 1,994 digests and `extraction_id`.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6593

No description provided.

Rows
Columns