[PR #6966] [MERGED] perf(artifacts): bound stable reads to one snapshot buffer #7479

New issue

Closed

opened 2026-11-01 21:48:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:48:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6966
Author: @gensokyo-dev
Created: 10/24/2026
Status: ✅ Merged
Merged: 10/24/2026
Merged by: @gensokyo-dev

Base: main ← Head: perf/6954-stable-read-buffer-20261024

📝 Commits (7)

cb73647 perf(artifacts): read stable files into one bounded buffer
45307fb perf(artifacts): fill stable snapshots through direct descriptor reads
41adc62 docs(perf): qualify stable snapshot memory with real ELF and BEAM tasks
3476cb8 test(artifacts): count descriptor close retries from admission
6b07a51 Merge remote-tracking branch 'origin/main' into perf/6954-stable-read-buffer-20261024
8e5bf78 test(artifacts): capture the retained descriptor after asynchronous admission
cc92aeb docs(perf): record integrated qualification and cleanup fixture correction

📊 Changes

9 files changed (+2413 additions, -16 deletions)

View changed files

➕ experiments/analysis-efficiency/stable-artifact-costs.py (+99 -0)
➕ experiments/analysis-efficiency/stable-artifact-elf-oracle.py (+108 -0)
➕ experiments/analysis-efficiency/stable-artifact-profile.mjs (+81 -0)
➕ experiments/analysis-efficiency/stable-artifact-reads.json (+1430 -0)
➕ experiments/analysis-efficiency/stable-artifact-reads.md (+212 -0)
➕ experiments/analysis-efficiency/stable-artifact-reads.mjs (+156 -0)
📝 src/artifacts/readStableArtifact.ts (+32 -14)
📝 tests/boundary/filesystem/stableArtifact.test.ts (+282 -0)
📝 tests/boundary/providers/artifactReadCleanupRetry.test.ts (+13 -2)

📄 Description

Summary

Reading a large stable artifact retained every chunk and then concatenated a second full-file copy. Fill one private, budget-capped snapshot directly through the admitted descriptor, hashing incrementally; keep a reusable 64 KiB EOF/growth scratch buffer. Fixes #6954.

The shared owner serves CLI/MCP consumers without changing schemas, caller options, Evidence, file-state validation or cleanup ownership. Incomplete bytes stay private. Continued growth preserves physical I/O and byte-limit failures before final integrity refusal. Scope decision: not applicable; ordinary optimization of existing read-only file inspection.

Evidence and validation

153 exact owner/consumer tests passed, including short reads, pending-read cancellation, error precedence, growth, raced sizes, incomplete reads and failed-close retry.
After integrating main 98ec00f40, all 153 tests, six serial static tasks (typecheck/lint/format/knip/metadata/compilation), architecture guards, three fresh real MCP/source-export tasks and three CLI calls passed again. Complete Evidence matches the frozen corpus; seven guarded groups verified cleanup/no OOM, static peak 1.59 GiB.
Real LLVM 16, GNU ls and OTP 27 BEAM: six CLI calls and 18 alternating fresh production stdio MCP tasks passed independent byte-level oracle facts, complete source export and identical full report/Evidence identities. No target execution.
LLVM whole-task kernel family peak median 756.5 → 725.6 MiB; observed elapsed 15.848 → 12.209 s, family CPU 13.393 → 11.161 s. Separate producer probes locate about 121 MiB less process RSS from removing the duplicate buffer. Tokens unchanged. Small/BEAM controls establish no general memory/speed benefit; discovery/host variance limits latency attribution.
One-CPU quota, 3 GiB early stop/4 GiB hard/no swap, serial tasks. Measured payload exits/cleanup/events verified; no guard stop/OOM. Final whole-scope counters unavailable. Cached preparation witness failures and initial harness failures are retained and described.

Fixed pins, corpus hashes, ranges, methods and reproduction; per-task measurements. Runtime experiments cover these Linux files; Windows/macOS, unusual filesystems and all 37 caller workflows remain unverified by real-provider runs.

Initial CI passed source tests (17,305), static/package/Windows checks and selected real-workflow/SQLite lanes. Two cleanup fixture cases failed because the fixture omitted its first close from its counter; it now captures the descriptor at admission and verifies the same descriptor stays open after failure and closes after one retry. Four built shards passed; three were cancelled and remain unknown coverage. The corrected 153-case local gate passed. Updated head cc92aebd1: Codex review completed with no findings; source tests (17,336), all eight built shards, static/package/native baseline/Windows checks and 11 real-workflow/SQLite jobs passed. The source-map lane initially hit an unchanged Chrome launch deadline before analysis, then passed the unchanged-job rerun. CI run 40074023785, attempt 2, completed successfully; all eight shards and all 12 selected real-workflow/SQLite lanes are green. The original failed attempt is retained. Local same-browser launch/capture controls completed on fixed main and current head; this is not an identical CI-timeout reproduction.

Author readiness

Human owner: @morluto

The requester explicitly authorized opening this PR ready for review and continuing through merge without additional confirmation, and waived agent-side readiness gating. The declarations below remain unchecked because no human review of this final diff is being attested on their behalf.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6966 **Author:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Created:** 10/24/2026 **Status:** ✅ Merged **Merged:** 10/24/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `perf/6954-stable-read-buffer-20261024` --- ### 📝 Commits (7) - [`cb73647`](https://github.com/morluto/rea/commit/cb7364732c116d86c732a010ac74972a76c76ba0) perf(artifacts): read stable files into one bounded buffer - [`45307fb`](https://github.com/morluto/rea/commit/45307fb9534f464eee094ccddc4e295d1c0dd0fb) perf(artifacts): fill stable snapshots through direct descriptor reads - [`41adc62`](https://github.com/morluto/rea/commit/41adc629fe804170be4b76996efb5ae4166c8029) docs(perf): qualify stable snapshot memory with real ELF and BEAM tasks - [`3476cb8`](https://github.com/morluto/rea/commit/3476cb88fe02e257e8cc7a85087cb72a614e60da) test(artifacts): count descriptor close retries from admission - [`6b07a51`](https://github.com/morluto/rea/commit/6b07a513b2e15ce8dfff5974ac6291ea07329c40) Merge remote-tracking branch 'origin/main' into perf/6954-stable-read-buffer-20261024 - [`8e5bf78`](https://github.com/morluto/rea/commit/8e5bf787650e147aa1927c920501665df45dfe3a) test(artifacts): capture the retained descriptor after asynchronous admission - [`cc92aeb`](https://github.com/morluto/rea/commit/cc92aebd19401d057441e85e4bffe762ddccdd52) docs(perf): record integrated qualification and cleanup fixture correction ### 📊 Changes **9 files changed** (+2413 additions, -16 deletions) <details> <summary>View changed files</summary> ➕ `experiments/analysis-efficiency/stable-artifact-costs.py` (+99 -0) ➕ `experiments/analysis-efficiency/stable-artifact-elf-oracle.py` (+108 -0) ➕ `experiments/analysis-efficiency/stable-artifact-profile.mjs` (+81 -0) ➕ `experiments/analysis-efficiency/stable-artifact-reads.json` (+1430 -0) ➕ `experiments/analysis-efficiency/stable-artifact-reads.md` (+212 -0) ➕ `experiments/analysis-efficiency/stable-artifact-reads.mjs` (+156 -0) 📝 `src/artifacts/readStableArtifact.ts` (+32 -14) 📝 `tests/boundary/filesystem/stableArtifact.test.ts` (+282 -0) 📝 `tests/boundary/providers/artifactReadCleanupRetry.test.ts` (+13 -2) </details> ### 📄 Description ## Summary Reading a large stable artifact retained every chunk and then concatenated a second full-file copy. Fill one private, budget-capped snapshot directly through the admitted descriptor, hashing incrementally; keep a reusable 64 KiB EOF/growth scratch buffer. Fixes #6954. The shared owner serves CLI/MCP consumers without changing schemas, caller options, Evidence, file-state validation or cleanup ownership. Incomplete bytes stay private. Continued growth preserves physical I/O and byte-limit failures before final integrity refusal. Scope decision: not applicable; ordinary optimization of existing read-only file inspection. ## Evidence and validation - 153 exact owner/consumer tests passed, including short reads, pending-read cancellation, error precedence, growth, raced sizes, incomplete reads and failed-close retry. - After integrating main `98ec00f40`, all 153 tests, six serial static tasks (typecheck/lint/format/knip/metadata/compilation), architecture guards, three fresh real MCP/source-export tasks and three CLI calls passed again. Complete Evidence matches the frozen corpus; seven guarded groups verified cleanup/no OOM, static peak 1.59 GiB. - Real LLVM 16, GNU ls and OTP 27 BEAM: six CLI calls and 18 alternating fresh production stdio MCP tasks passed independent byte-level oracle facts, complete source export and identical full report/Evidence identities. No target execution. - LLVM whole-task kernel family peak median **756.5 → 725.6 MiB**; observed elapsed **15.848 → 12.209 s**, family CPU **13.393 → 11.161 s**. Separate producer probes locate about **121 MiB** less process RSS from removing the duplicate buffer. Tokens unchanged. Small/BEAM controls establish no general memory/speed benefit; discovery/host variance limits latency attribution. - One-CPU quota, 3 GiB early stop/4 GiB hard/no swap, serial tasks. Measured payload exits/cleanup/events verified; no guard stop/OOM. Final whole-scope counters unavailable. Cached preparation witness failures and initial harness failures are retained and described. [Fixed pins, corpus hashes, ranges, methods and reproduction](experiments/analysis-efficiency/stable-artifact-reads.md); [per-task measurements](experiments/analysis-efficiency/stable-artifact-reads.json). Runtime experiments cover these Linux files; Windows/macOS, unusual filesystems and all 37 caller workflows remain unverified by real-provider runs. Initial CI passed source tests (17,305), static/package/Windows checks and selected real-workflow/SQLite lanes. Two cleanup fixture cases failed because the fixture omitted its first close from its counter; it now captures the descriptor at admission and verifies the same descriptor stays open after failure and closes after one retry. Four built shards passed; three were cancelled and remain unknown coverage. The corrected 153-case local gate passed. Updated head `cc92aebd1`: Codex review completed with no findings; source tests (17,336), all eight built shards, static/package/native baseline/Windows checks and 11 real-workflow/SQLite jobs passed. The source-map lane initially hit an unchanged Chrome launch deadline before analysis, then passed the unchanged-job rerun. CI run `40074023785`, attempt 2, completed successfully; all eight shards and all 12 selected real-workflow/SQLite lanes are green. The original failed attempt is retained. Local same-browser launch/capture controls completed on fixed main and current head; this is not an identical CI-timeout reproduction. ## Author readiness Human owner: @morluto The requester explicitly authorized opening this PR ready for review and continuing through merge without additional confirmation, and waived agent-side readiness gating. The declarations below remain unchecked because no human review of this final diff is being attested on their behalf. - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:48:07 +00:00