[GH-ISSUE #1251] feat(process): explain omitted file digests and separate hashing from retained output #449

New issue

Closed

opened 2026-10-15 18:25:09 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:25:09 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1251

Current limitation

limits.file_bytes admits whole files to filesystem SHA-256 work. An oversized file remains listed with sha256:null and snapshot truncation, but lacks a per-file reason distinguishing budget exhaustion from a changed/unstable file or another hashing limitation. Analysts can mistake a bounded file listing for a complete report binding.

Reproduction

Capture an owned Node process that writes report.json containing 8,192 X bytes. Select its directory for filesystem observation and disable text normalization. With limits.file_bytes:128, the file is listed at its full size with a null digest; with 16,384, its complete SHA-256 is returned and matches independent hashing. Both children exit0.

Required behavior

Explain per-file digest status/reason and applicable cumulative I/O budget. Offer an explicit full-report digest workflow whose hashing budget is independent of terminal or inline retained content, without unbounded I/O or a misleading prefix digest. Caller-selected periodic/final artifacts should identify complete versus partial/running content.

The implementation already hashes admitted files in 64 KiB chunks: this is not a request to replace nonexistent whole-file buffering. Clarify the existing option and evaluate a separate explicit hashing budget/manifest only where needed. Preserve file identity/state checks and cancellation; a changed file must not gain a supposedly stable full digest.

Test exact-size and cumulative budgets, unstable/replaced files, cancellation, complete digest controls and CLI/MCP parity. Keep an independent external sealer's digest distinct from REA-observed facts.

Source and relationship

src/process/capture/FilesystemSnapshot.ts:47 and src/process/capture/FilesystemSnapshot.ts:132. Related #59/#48; per-dimension truncation reporting does not itself supply a complete artifact digest.

Verification and scope

Whole-file admission and the sufficient-budget digest control reproduced through real MCP on both versions. Existing bounded behavior is a contract limitation, not a hashing algorithm defect.

Audit date: 2026-10-09. Latest npm dist-tag verified as rea-agents@6.1.0; current main tested at f4c99ab8a53806dd57dc24b3d5a600905bcb4686. Linux x64, Node 24.18.0, MCP client SDK 2.3.1. Native observations use unmodified Ghidra 12.1.4 and JDK 21, two CPUs and a 512 MiB Java heap. No target execution or executable mutation was used for the static native checks.

Originally created by @N0zoM1z0 on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1251 ### Current limitation `limits.file_bytes` admits whole files to filesystem SHA-256 work. An oversized file remains listed with `sha256:null` and snapshot truncation, but lacks a per-file reason distinguishing budget exhaustion from a changed/unstable file or another hashing limitation. Analysts can mistake a bounded file listing for a complete report binding. ### Reproduction Capture an owned Node process that writes `report.json` containing 8,192 `X` bytes. Select its directory for filesystem observation and disable text normalization. With `limits.file_bytes:128`, the file is listed at its full size with a null digest; with 16,384, its complete SHA-256 is returned and matches independent hashing. Both children exit0. ### Required behavior Explain per-file digest status/reason and applicable cumulative I/O budget. Offer an explicit full-report digest workflow whose hashing budget is independent of terminal or inline retained content, without unbounded I/O or a misleading prefix digest. Caller-selected periodic/final artifacts should identify complete versus partial/running content. The implementation already hashes admitted files in 64 KiB chunks: this is not a request to replace nonexistent whole-file buffering. Clarify the existing option and evaluate a separate explicit hashing budget/manifest only where needed. Preserve file identity/state checks and cancellation; a changed file must not gain a supposedly stable full digest. Test exact-size and cumulative budgets, unstable/replaced files, cancellation, complete digest controls and CLI/MCP parity. Keep an independent external sealer's digest distinct from REA-observed facts. ### Source and relationship [src/process/capture/FilesystemSnapshot.ts:47](https://github.com/morluto/rea/blob/f4c99ab8a53806dd57dc24b3d5a600905bcb4686/src/process/capture/FilesystemSnapshot.ts#L47) and [src/process/capture/FilesystemSnapshot.ts:132](https://github.com/morluto/rea/blob/f4c99ab8a53806dd57dc24b3d5a600905bcb4686/src/process/capture/FilesystemSnapshot.ts#L132). Related #59/#48; per-dimension truncation reporting does not itself supply a complete artifact digest. ### Verification and scope Whole-file admission and the sufficient-budget digest control reproduced through real MCP on both versions. Existing bounded behavior is a contract limitation, not a hashing algorithm defect. Audit date: 2026-10-09. Latest npm dist-tag verified as `rea-agents@6.1.0`; current main tested at `f4c99ab8a53806dd57dc24b3d5a600905bcb4686`. Linux x64, Node 24.18.0, MCP client SDK 2.3.1. Native observations use unmodified Ghidra 12.1.4 and JDK 21, two CPUs and a 512 MiB Java heap. No target execution or executable mutation was used for the static native checks.

bsakaguchi

2026-10-15 18:25:09 +00:00

closed this issue
added the
triage

area: domain

enhancement

P2: medium
labels

bsakaguchi commented

2026-10-15 18:25:10 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 9, 2026):

The omission-diagnostics portion is addressed by #1284: each filesystem checkpoint now reports the existing cumulative file_bytes budget, hashed bytes, and a per-file omission with its size, remaining budget, and reason (file_bytes_budget or file_changed_or_short_read). Whole-file admission remains unchanged; a smaller later file can still receive a complete digest after an oversized file is skipped.

Verified through actual CLI and SDK MCP captures, including the later smaller-file control and comparison that preserves unaffected terminal/exit observations. This does not add an independent hashing budget, digest tool, or final-artifact sealing workflow; those remaining requirements keep this issue open.

@N0zoM1z0 commented on GitHub (Oct 9, 2026): The omission-diagnostics portion is addressed by #1284: each filesystem checkpoint now reports the existing cumulative `file_bytes` budget, hashed bytes, and a per-file omission with its size, remaining budget, and reason (`file_bytes_budget` or `file_changed_or_short_read`). Whole-file admission remains unchanged; a smaller later file can still receive a complete digest after an oversized file is skipped. Verified through actual CLI and SDK MCP captures, including the later smaller-file control and comparison that preserves unaffected terminal/exit observations. This does not add an independent hashing budget, digest tool, or final-artifact sealing workflow; those remaining requirements keep this issue open.

bsakaguchi commented

2026-10-15 18:25:10 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 10, 2026):

Follow-up source tracing corrects my earlier status note: limits.file_bytes already controls only cumulative whole-file hash I/O, independently of terminal output_bytes; lack of a new knob does not mean the budgets are coupled. Each checkpoint resets that existing budget.

#1440 documents this and runs the actual example through CLI/MCP. A 131,103-byte report receives its independently verified full SHA-256 with only a 1 KiB terminal budget, even though verbose terminal chunks are omitted. Exact/cumulative budget controls and before/after accounting pass; the existing identity-change/replacement/cancellation tests remain green (35 tests total). A fresh public stdio capture also verifies text/structured parity, owned cleanup and subsequent ping.

The documented workflow distinguishes complete checkpoint bytes from the producer's completion state, including stable running reports and external-sealer claims. No duplicate hash-budget option or new manifest protocol is needed for this scope. The PR is awaiting final-head CI; this issue stays open until that gate and merge.

@N0zoM1z0 commented on GitHub (Oct 10, 2026): Follow-up source tracing corrects my earlier status note: `limits.file_bytes` already controls only cumulative whole-file hash I/O, independently of terminal `output_bytes`; lack of a new knob does not mean the budgets are coupled. Each checkpoint resets that existing budget. #1440 documents this and runs the actual example through CLI/MCP. A 131,103-byte report receives its independently verified full SHA-256 with only a 1 KiB terminal budget, even though verbose terminal chunks are omitted. Exact/cumulative budget controls and before/after accounting pass; the existing identity-change/replacement/cancellation tests remain green (35 tests total). A fresh public stdio capture also verifies text/structured parity, owned cleanup and subsequent ping. The documented workflow distinguishes complete checkpoint bytes from the producer's completion state, including stable running reports and external-sealer claims. No duplicate hash-budget option or new manifest protocol is needed for this scope. The PR is awaiting final-head CI; this issue stays open until that gate and merge.

bsakaguchi referenced this issue

2026-10-15 18:28:03 +00:00

[GH-ISSUE #2296] [Feature] Decompile static Python bytecode (.pyc) files into labeled source inference #758

bsakaguchi referenced this issue

2026-10-15 18:28:29 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-16 21:13:12 +00:00

[PR #449] [MERGED] feat(bytecode): add JVM and Python bytecode providers #1996

bsakaguchi added

enhancement

and removed