[GH-ISSUE #6954] perf: avoid duplicating stable artifact buffers

bsakaguchi commented

2026-10-31 00:48:04 +00:00

Owner

Originally created by @gensokyo-dev on GitHub (Oct 24, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6954

Before submitting

Searched open and closed issues for readStableArtifact memory; no matching report found.
Removed credentials and sensitive target data.

Area / interface / earliest stage

Evidence/artifacts; shared CLI/MCP file producer; whole-file buffering before format interpretation.

Observed behavior

At b5c42dfc2a8eddeffb8d37c669178fa04169f050, readStableArtifact retains all input chunks and then allocates a second complete file buffer with Buffer.concat. It has 37 production call sites spanning native diagnostics, Python/BEAM, browser, Go/Rust, Wasm, SQLite and other static workflows.

One real LLVM 16 read (123,379,936 bytes) reported 247,034,863 bytes of Node ArrayBuffers after completion, process max RSS 302,988 KiB, and a 387 ms read phase. An independent sha256sum and a separate SHA-256 over returned bytes both matched. A small GNU ls control (151,344 bytes) also passed. These are single baseline observations, not a measured before/after improvement.

The actual public CLI inspect-elf-notes on LLVM returned 3,573 output bytes, exactly equal to the shared service's complete Evidence. Its guarded process family peaked at 585,539,584 bytes, elapsed 3.749 s, CPU 3.321 s. GNU readelf independently agrees on build ID 649be1a7cae92cf064a666f2a64ececdf264cb7b and gold version gold 1.16. No OOM occurred; output size alone is not the memory measurement.

Reproduction

Built the pinned checkout with the locked toolchain, then ran:

node scripts/rea.mjs inspect-elf-notes /usr/lib/x86_64-linux-gnu/libLLVM-16.so.1 \
  --max-file-bytes 268435456 --json

The direct producer probe calls readStableArtifact(path, 268435456), verifies both the returned digest and a fresh SHA-256 over its bytes against independently measured source identity, and records Node memory/max RSS. Local receipts and probe source: /tmp/rea-perf-stable-read-20261024/. Linux x64, Node 24.22.1, npm 11.16.0; one-CPU quota, 3 GiB early stop, 4 GiB hard/no swap. Payload exit/cleanup verified, no limit events; final whole-scope counters unavailable.

Expected behavior

Produce the exact same complete stable-file snapshot and digest without retaining a second full-file representation. Preserve selected byte limits, malformed-stream errors, physical read failures, file-change detection, cancellation, completed observations through failed finalization, and retryable descriptor ownership.

Scope and proposed checks

Keep the correction at the shared reader; no cache, target execution, provider authority, public schema or output change is proposed. A pre-sized bounded destination may remove the retained chunk array/concatenation; do not allocate from a raced descriptor size above the selected byte budget, and preserve error ordering when data grows or shrinks. This is a proposed implementation, not a verified benefit.

Run focused producer/lifecycle/consumer regressions and real public tasks on large/small inputs before integration. Measure repeated equivalent CLI/MCP tasks, whole-task token estimates, elapsed/CPU and process-family memory; retain complete Evidence and report any tradeoff. Other operating systems, cold-disk behavior and actual model usage are unverified.

Artifact context

LLVM SHA-256 f62d254b7f2bf42df8c8b07d46ee3bb4c2cafeca436b2e6bc6ccbe4581f58f40; GNU ls SHA-256 cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4. Both are installed public system files read without executing them. This report does not claim a runtime correctness failure or OOM reproduction.

Originally created by @gensokyo-dev on GitHub (Oct 24, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6954 ### Before submitting - [x] Searched open and closed issues for `readStableArtifact memory`; no matching report found. - [x] Removed credentials and sensitive target data. ### Area / interface / earliest stage Evidence/artifacts; shared CLI/MCP file producer; whole-file buffering before format interpretation. ### Observed behavior At `b5c42dfc2a8eddeffb8d37c669178fa04169f050`, `readStableArtifact` retains all input chunks and then allocates a second complete file buffer with `Buffer.concat`. It has 37 production call sites spanning native diagnostics, Python/BEAM, browser, Go/Rust, Wasm, SQLite and other static workflows. One real LLVM 16 read (123,379,936 bytes) reported **247,034,863 bytes of Node ArrayBuffers** after completion, process max RSS **302,988 KiB**, and a 387 ms read phase. An independent `sha256sum` and a separate SHA-256 over returned bytes both matched. A small GNU ls control (151,344 bytes) also passed. These are single baseline observations, not a measured before/after improvement. The actual public CLI `inspect-elf-notes` on LLVM returned 3,573 output bytes, exactly equal to the shared service's complete Evidence. Its guarded process family peaked at **585,539,584 bytes**, elapsed 3.749 s, CPU 3.321 s. GNU readelf independently agrees on build ID `649be1a7cae92cf064a666f2a64ececdf264cb7b` and gold version `gold 1.16`. No OOM occurred; output size alone is not the memory measurement. ### Reproduction Built the pinned checkout with the locked toolchain, then ran: ```sh node scripts/rea.mjs inspect-elf-notes /usr/lib/x86_64-linux-gnu/libLLVM-16.so.1 \ --max-file-bytes 268435456 --json ``` The direct producer probe calls `readStableArtifact(path, 268435456)`, verifies both the returned digest and a fresh SHA-256 over its bytes against independently measured source identity, and records Node memory/max RSS. Local receipts and probe source: `/tmp/rea-perf-stable-read-20261024/`. Linux x64, Node 24.22.1, npm 11.16.0; one-CPU quota, 3 GiB early stop, 4 GiB hard/no swap. Payload exit/cleanup verified, no limit events; final whole-scope counters unavailable. ### Expected behavior Produce the exact same complete stable-file snapshot and digest without retaining a second full-file representation. Preserve selected byte limits, malformed-stream errors, physical read failures, file-change detection, cancellation, completed observations through failed finalization, and retryable descriptor ownership. ### Scope and proposed checks Keep the correction at the shared reader; no cache, target execution, provider authority, public schema or output change is proposed. A pre-sized bounded destination may remove the retained chunk array/concatenation; do not allocate from a raced descriptor size above the selected byte budget, and preserve error ordering when data grows or shrinks. This is a proposed implementation, not a verified benefit. Run focused producer/lifecycle/consumer regressions and real public tasks on large/small inputs before integration. Measure repeated equivalent CLI/MCP tasks, whole-task token estimates, elapsed/CPU and process-family memory; retain complete Evidence and report any tradeoff. Other operating systems, cold-disk behavior and actual model usage are unverified. ### Artifact context LLVM SHA-256 `f62d254b7f2bf42df8c8b07d46ee3bb4c2cafeca436b2e6bc6ccbe4581f58f40`; GNU ls SHA-256 `cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4`. Both are installed public system files read without executing them. This report does not claim a runtime correctness failure or OOM reproduction.

bsakaguchi

2026-10-31 00:48:04 +00:00

closed this issue
added the
bug

triage
labels

bsakaguchi commented

2026-10-31 00:48:05 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 24, 2026):

The doubling pattern is easy to confirm from the numbers: ~123 MB input produced ~247 MB of retained ArrayBuffers, i.e. the chunk list plus the Buffer.concat result coexist. Since max-file-bytes is known before reading, you can preallocate a single Buffer up to that cap and copy chunks into it, dropping the separate chunk array entirely; if you want to avoid preallocating the cap on small files, a growth strategy (e.g. doubling buffer) keeps peak usage near one copy instead of two. Digest verification can still run over the single buffer before returning. That keeps byte limits and malformed-stream errors on the same path without changing the public result. Worth measuring before/after with the same LLVM read as the baseline.

@XinwangTao commented on GitHub (Oct 24, 2026): The doubling pattern is easy to confirm from the numbers: ~123 MB input produced ~247 MB of retained ArrayBuffers, i.e. the chunk list plus the Buffer.concat result coexist. Since max-file-bytes is known before reading, you can preallocate a single Buffer up to that cap and copy chunks into it, dropping the separate chunk array entirely; if you want to avoid preallocating the cap on small files, a growth strategy (e.g. doubling buffer) keeps peak usage near one copy instead of two. Digest verification can still run over the single buffer before returning. That keeps byte limits and malformed-stream errors on the same path without changing the public result. Worth measuring before/after with the same LLVM read as the baseline.

bsakaguchi commented

2026-10-31 00:48:05 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 24, 2026):

PR #6966 合并后的实测数字符合预期：producer 侧约 121 MiB 的 RSS 下降正好对应去掉的那份重复 buffer，LLVM 整任务峰值 756.5→725.6 MiB、耗时 15.848→12.209 s，方向完全正确。64 KiB EOF/growth scratch buffer 也避免了小文件按 cap 预分配的浪费。想确认一个边界场景：当 --max-file-bytes 大于实际文件大小时，返回的仍是完整文件内容而非截断到预算值，byte-limit 失败只在文件真实超过上限时触发，对吧？

@XinwangTao commented on GitHub (Oct 24, 2026): PR #6966 合并后的实测数字符合预期：producer 侧约 121 MiB 的 RSS 下降正好对应去掉的那份重复 buffer，LLVM 整任务峰值 756.5→725.6 MiB、耗时 15.848→12.209 s，方向完全正确。64 KiB EOF/growth scratch buffer 也避免了小文件按 cap 预分配的浪费。想确认一个边界场景：当 --max-file-bytes 大于实际文件大小时，返回的仍是完整文件内容而非截断到预算值，byte-limit 失败只在文件真实超过上限时触发，对吧？

bsakaguchi referenced this issue

2026-10-31 00:48:27 +00:00

[GH-ISSUE #7129] [Feature] Static readers for firmware containers: Intel HEX/S-record regions, U-Boot uImage headers, and FIT images #4280

bsakaguchi referenced this issue

2026-10-31 00:48:28 +00:00

[GH-ISSUE #7129] [Feature] Static readers for firmware containers: Intel HEX/S-record regions, U-Boot uImage headers, and FIT images #4280

bsakaguchi referenced this issue

2026-10-31 00:48:29 +00:00

[GH-ISSUE #7129] [Feature] Static readers for firmware containers: Intel HEX/S-record regions, U-Boot uImage headers, and FIT images #4280

bsakaguchi referenced this issue

2026-10-31 00:52:01 +00:00

[GH-ISSUE #7955] [Roadmap] Embedded firmware: make REA work on the devices around us #4654