[GH-ISSUE #6947] perf: decouple byte entropy reads from output block size #4245

New issue

Closed

opened 2026-10-31 00:48:02 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:48:02 +00:00

Owner

Originally created by @gensokyo-dev on GitHub (Oct 24, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6947

Before submitting

Searched open and closed issues for entropy read; existing reports concern other behavior.
Removed credentials and sensitive target data.

Area / interface / earliest stage

Evidence, artifacts, or reconstruction; shared CLI/MCP service; file reading during native analysis.

Observed behavior

At 1d896e03b, inspecting a 65,539-byte window of the installed LLVM 16 library with block_size: 256 makes 481,955 physical reads to cover its 123,379,936 bytes. One guarded production-service run took 17.09 seconds, 15.10 seconds process-family CPU, and 162.7 MiB kernel family peak. Payload exit and cleanup verified; no guard stop/OOM. This is one boundary measurement, not a public-workflow speedup claim.

Reproduction

Built the pinned checkout, then ran ByteEntropyService.inspect({path: "/usr/lib/x86_64-linux-gnu/libLLVM-16.so.1", block_size: 256, offset: 3, length: 65539}). An injected filesystem delegates unchanged to real open/stat/read, counting reads and bytes. Maximum requested read was 256 bytes; all file bytes were read. Receipt: /tmp/rea-perf-byte-entropy-20261024/baseline-profile.json and guarded log on the investigation host.

Expected behavior

Preserve every selected block, file-wide entropy and SHA-256, Evidence, cancellation and stable-file checks while reading in bounded chunks independent of output resolution. Source: src/application/binaryDiagnostics/ByteEntropyService.ts currently allocates and requests input.block_size bytes even outside the selected window.

Impact and scope

Observed once on this real library. Full-file hashing remains required; excessive small asynchronous reads are avoidable. Other operating systems and disks are unmeasured. A small-file control and repeated production CLI/MCP tasks are proposed before integration. No output/token reduction is expected.

REA version / runtime / target

1d896e03b, Node 24.22.1 / npm 11.16.0, Linux x64. Installed ELF LLVM 16 shared library; static byte inspection only, no target execution or external analysis provider.

Proposed correction and verification

Separate a bounded 64 KiB read buffer from the caller's power-of-two 256..65,536-byte report blocks at their shared owner. Independently calculate real-file histograms/blocks/runs/digests, compare CLI/MCP results and retained Evidence, test short reads, unaligned windows and boundary tails, and retain existing cancellation/integrity/cleanup coverage. Measure repeated whole tasks, CPU and family memory; report catalog and token estimates without implying a token saving.

Originally created by @gensokyo-dev on GitHub (Oct 24, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6947 ### Before submitting - [x] Searched open and closed issues for `entropy read`; existing reports concern other behavior. - [x] Removed credentials and sensitive target data. ### Area / interface / earliest stage Evidence, artifacts, or reconstruction; shared CLI/MCP service; file reading during native analysis. ### Observed behavior At `1d896e03b`, inspecting a 65,539-byte window of the installed LLVM 16 library with `block_size: 256` makes **481,955 physical reads** to cover its 123,379,936 bytes. One guarded production-service run took 17.09 seconds, 15.10 seconds process-family CPU, and 162.7 MiB kernel family peak. Payload exit and cleanup verified; no guard stop/OOM. This is one boundary measurement, not a public-workflow speedup claim. ### Reproduction Built the pinned checkout, then ran `ByteEntropyService.inspect({path: "/usr/lib/x86_64-linux-gnu/libLLVM-16.so.1", block_size: 256, offset: 3, length: 65539})`. An injected filesystem delegates unchanged to real `open/stat/read`, counting reads and bytes. Maximum requested read was 256 bytes; all file bytes were read. Receipt: `/tmp/rea-perf-byte-entropy-20261024/baseline-profile.json` and guarded log on the investigation host. ### Expected behavior Preserve every selected block, file-wide entropy and SHA-256, Evidence, cancellation and stable-file checks while reading in bounded chunks independent of output resolution. Source: `src/application/binaryDiagnostics/ByteEntropyService.ts` currently allocates and requests `input.block_size` bytes even outside the selected window. ### Impact and scope Observed once on this real library. Full-file hashing remains required; excessive small asynchronous reads are avoidable. Other operating systems and disks are unmeasured. A small-file control and repeated production CLI/MCP tasks are proposed before integration. No output/token reduction is expected. ### REA version / runtime / target `1d896e03b`, Node 24.22.1 / npm 11.16.0, Linux x64. Installed ELF LLVM 16 shared library; static byte inspection only, no target execution or external analysis provider. ### Proposed correction and verification Separate a bounded 64 KiB read buffer from the caller's power-of-two 256..65,536-byte report blocks at their shared owner. Independently calculate real-file histograms/blocks/runs/digests, compare CLI/MCP results and retained Evidence, test short reads, unaligned windows and boundary tails, and retain existing cancellation/integrity/cleanup coverage. Measure repeated whole tasks, CPU and family memory; report catalog and token estimates without implying a token saving.

bsakaguchi

2026-10-31 00:48:02 +00:00

closed this issue
added the
bug

triage
labels

bsakaguchi commented

2026-10-31 00:48:03 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 24, 2026):

This is a real and easily reproducible problem: with block_size 256 the code issues one read per 256-byte window, so syscall count scales with file size rather than the selected window, and 480k reads dominate the 17s runtime. Decoupling a bounded read buffer (64 KiB) from the caller's report block is the right direction — it keeps the selected-window semantics and file-wide digests intact while collapsing thousands of short reads into sequential buffered reads. One suggestion: since the injected filesystem counts reads/bytes, it would be valuable to also report syscall count before/after in the verification step, so the reduction is measurable independently of wall-clock (which can be noisy on cached files).

@XinwangTao commented on GitHub (Oct 24, 2026): This is a real and easily reproducible problem: with block_size 256 the code issues one read per 256-byte window, so syscall count scales with file size rather than the selected window, and 480k reads dominate the 17s runtime. Decoupling a bounded read buffer (64 KiB) from the caller's report block is the right direction — it keeps the selected-window semantics and file-wide digests intact while collapsing thousands of short reads into sequential buffered reads. One suggestion: since the injected filesystem counts reads/bytes, it would be valuable to also report syscall count before/after in the verification step, so the reduction is measurable independently of wall-clock (which can be noisy on cached files).

bsakaguchi commented

2026-10-31 00:48:03 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 24, 2026):

Good find — coupling the physical read buffer to the caller block_size is exactly the kind of thing that explodes on large binaries: 123 MB with 256-byte blocks means hundreds of thousands of tiny reads, most of them outside the selected window. Separating the bounded 64 KiB read buffer from the power-of-two report resolution (PR #6950) looks like the right fix: it preserves the selected-block semantics and file-wide entropy/SHA-256 while removing the read amplification. Agree that short-read, unaligned-window and boundary-tail cases deserve explicit tests, and keeping cancellation and stable-file checks on the read loop is important so the optimization never weakens integrity guarantees.

@XinwangTao commented on GitHub (Oct 24, 2026): Good find — coupling the physical read buffer to the caller block_size is exactly the kind of thing that explodes on large binaries: 123 MB with 256-byte blocks means hundreds of thousands of tiny reads, most of them outside the selected window. Separating the bounded 64 KiB read buffer from the power-of-two report resolution (PR #6950) looks like the right fix: it preserves the selected-block semantics and file-wide entropy/SHA-256 while removing the read amplification. Agree that short-read, unaligned-window and boundary-tail cases deserve explicit tests, and keeping cancellation and stable-file checks on the read loop is important so the optimization never weakens integrity guarantees.

bsakaguchi referenced this issue

2026-10-31 00:48:27 +00:00

[GH-ISSUE #7129] [Feature] Static readers for firmware containers: Intel HEX/S-record regions, U-Boot uImage headers, and FIT images #4280

bsakaguchi referenced this issue

2026-10-31 00:52:01 +00:00

[GH-ISSUE #7955] [Roadmap] Embedded firmware: make REA work on the devices around us #4654

bsakaguchi referenced this issue

2026-11-01 21:10:58 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-11-05 01:48:02 +00:00

[PR #7368] [CLOSED] feat(artifacts): inspect and extract Intel HEX and S-record firmware #7837