morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #2331] [MERGED] feat(diagnostics): report a byte entropy profile for one local file #5167

New issue

Closed

opened 2026-10-31 01:04:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:04:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2331
Author: @mvanhorn
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/22/2026
Merged by: @morluto

Base: main ← Head: fix/byte-entropy-profile

📝 Commits (10+)

519ebaa feat(diagnostics): report a byte entropy profile for one local file
b82e190 fix(diagnostics): flag zero-size files as not shown empty
b87f511 fix(diagnostics): merge main and preserve entropy boundary failures
7a67707 fix(mcp): reject unsupported diagnostic summaries before inspection
eccc741 Merge remote-tracking branch 'origin/main' into fix/byte-entropy-profile
a156b1d fix(ci): repair integrated checks and Hopper test baseline
a2f6136 test(entropy): use shared MCP schema verification after merge
4df3682 chore: merge current main into PR #2331
f54b2ae Merge branch 'main' into fix/byte-entropy-profile
3a794e9 Merge branch 'main' into fix/byte-entropy-profile

📊 Changes

29 files changed (+1510 additions, -3 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+2 -0)
📝 docs/binary-diagnostics.md (+84 -0)
📝 docs/user-visible-errors.csv (+2 -0)
📝 scripts/lib/catalog-builders.mjs (+10 -1)
📝 src/application/CapabilityInventory.ts (+1 -0)
📝 src/application/InvestigationProviders.ts (+7 -0)
➕ src/application/binaryDiagnostics/ByteEntropyService.test.ts (+373 -0)
➕ src/application/binaryDiagnostics/ByteEntropyService.ts (+355 -0)
📝 src/cli/binaryDiagnosticsCommands.ts (+40 -0)
📝 src/cliCommandNames.ts (+1 -0)
📝 src/composition/binaryDiagnostics.ts (+7 -0)
➕ src/contracts/native/binaryDiagnosticsToolContracts.test.ts (+33 -0)
📝 src/contracts/native/binaryDiagnosticsToolContracts.ts (+36 -0)
📝 src/contracts/toolContractExamples.ts (+1 -0)
📝 src/contracts/toolEffects.test.ts (+1 -0)
📝 src/contracts/toolEffects.ts (+1 -0)
📝 src/contracts/toolOutputSchemaGroups.ts (+9 -0)
📝 src/domain/analysisView/analysisView.ts (+29 -0)
➕ src/domain/analysisView/byteEntropyView.ts (+66 -0)
➕ src/domain/native/byteEntropy.test.ts (+53 -0)

...and 9 more files

📄 Description

Summary

REA can now report a byte entropy profile for one local file: inspect_byte_entropy (CLI inspect-byte-entropy) reads an absolute path offline and returns file-wide SHA-256 and Shannon entropy, per-block entropy for an optional byte window, and adjacent high and low runs with exact file offsets. It runs no analysis engine, executes nothing, extracts nothing and makes no network requests.

No issue is linked. This is a new diagnostics tool, proposed here with the implementation.

Demo

Full-quality MP4

The sample is a generated 32 KB file (8 KB zeros, 8 KB text, 12 KB random bytes, 4 KB zeros), and every number in the video is copied from the CLI output on this branch.

Problem and expected behavior

Before opening an unknown blob in a disassembler or trying to extract it, a useful first question is where the padding, the plain data and the packed or compressed regions sit. inspect_binary_layout, inspect_pe_resources and inspect_recorded_crash report structure, but none of them reports entropy, so on upstream/main (2e45b4d) there is no file-wide entropy, no block series and no high or low runs to cite as Evidence.

With this change, one call on a regular file returns observed Evidence with artifact (path, byte size, SHA-256), block_size, window, file_entropy, thresholds, blocks (offset, length, entropy) and runs (offset, length, classification). On the demo sample that is file_entropy: 5.4872, two text blocks at 4.4006 and 4.4011 left unclassified, and three runs: 0+8192 low, 16384+12288 high, 28672+4096 low.

Change and scope

ByteEntropyService (src/application/binaryDiagnostics/ByteEntropyService.ts) backs both CLI and MCP. It streams the file once with one block-sized buffer and a 256-entry histogram, hashing and counting every byte so SHA-256 and file_entropy always cover the whole file, while blocks are recorded only inside the selected window. Blocks start at offset (default 0); an omitted length runs to EOF, and the final block may be shorter. block_size must be a power of two from 256 through 65536, default 4096.

Entropy is bits per byte from 0 to 8, rounded to four places. Adjacent blocks with reported entropy >= 7.2 merge into high runs and blocks <= 1.0 into low runs; anything in between breaks a run. Classification uses the rounded value that is reported, so a reader can recompute it from the output. The thresholds are returned in the result, and three limitations go with every result: entropy cannot tell compressed from encrypted bytes, small blocks can underestimate entropy, and file-wide fields cover the whole file while blocks and runs cover only the window. Empty files and empty windows are accepted with entropy 0 by convention.

Errors are typed rather than clipped or guessed: a relative path, a missing path, a non-regular file, a window past EOF or a file larger than Number.MAX_SAFE_INTEGER is an input error; EACCES/EPERM is access denied; a change in device, inode, size, mtime or ctime between the first stat and the end of the read is an artifact-changed error; cancellation is checked before and during the read.

detail: "summary" retains the complete Evidence in the session and returns file entropy, block and run counts, thresholds, the window and the ten longest runs (ties by offset), with parent_evidence_id for the full series. inspect_analysis_view gains a summary projection for this parent operation; any other view kind is rejected with a pointer to the parent Evidence. Summary detail requires a session that can retain Evidence.

Memory: the read itself is bounded by one block buffer, but the returned block list grows with the number of selected blocks, and summary detail still retains that full series. The docs say so and point to a narrower window or a larger block size. A local run on a 100,000,000-byte cached file with 4096-byte blocks, recorded in docs/binary-diagnostics.md, took about 1.1 s and 145 MiB peak RSS for 24,415 blocks; complete Evidence serialized to 1,122,303 bytes and the summary to 2,705 bytes.

Out of scope: extraction or carving of high-entropy regions, compression or encryption classification, configurable thresholds, and any other summary view kinds for this parent.

Contract and boundary impact

Semantic owner and earliest changed stage: domain schemas and math in src/domain/native/byteEntropy.ts (inspectByteEntropyInputSchema, byteEntropySchema, entropyFromCounts, mergeEntropyRuns); I/O in src/application/binaryDiagnostics/ByteEntropyService.ts; summary projection in src/domain/analysisView/byteEntropyView.ts, dispatched from projectAnalysisView in src/domain/analysisView/analysisView.ts.
Contribution scope fit and linked maintainer decision: format inspection that only reads one caller-selected local file, which CONTRIBUTING lists as supported; no target execution, injection, mutation or protection recovery, so no maintainer scope decision is required. Not applicable.
CLI and MCP/tool-catalog contract: new MCP tool inspect_byte_entropy (src/contracts/native/binaryDiagnosticsToolContracts.ts, output schema in src/contracts/toolOutputSchemaGroups.ts, example in src/contracts/toolContractExamples.ts, effect evidence in src/contracts/toolEffects.ts, availability in src/application/CapabilityInventory.ts) registered in src/server/registerBinaryDiagnosticsTools.ts and src/server/createServer.ts; new CLI command inspect-byte-entropy <path> --block-size --offset --length in src/cli/binaryDiagnosticsCommands.ts and src/cliCommandNames.ts. inspect_analysis_view parent_operation adds inspect_byte_entropy. Existing tools are unchanged; inspect_binary_layout registration now shares one runner with the new tool.
Provider, bridge, target-format, or platform compatibility: none. Provider-free (src/composition/binaryDiagnostics.ts), works on any regular file, no Hopper, Ghidra or external binary needed.
Evidence, artifact, provenance, or reconstruction contract: confidence: "observed", provider byte-entropy v1, artifact identity is the file SHA-256, parameters record path, block size, offset and length (when given), and limitations are carried into the summary view.
Target selection, authority, upstream behavior, and cleanup details (if applicable): the caller names one absolute path; the file is opened O_RDONLY | O_NONBLOCK and closed in finally. No processes, no writes, no temp files.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: authored skill line in .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md ("Use inspect_byte_entropy to triage high- and low-entropy ranges before extraction."); docs in docs/binary-diagnostics.md and two rows (91, 92) in docs/user-visible-errors.csv. metadata:check and docs:check pass; no build outputs committed.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated: src/domain/native/byteEntropy.test.ts (known Shannon answers including partial and empty blocks; threshold-edge merging that skips unclassified blocks and gaps; rejection of unsafe windows, unknown fields and non-power-of-two or out-of-range block sizes); src/application/binaryDiagnostics/ByteEntropyService.test.ts (whole-file identity and entropy with an unaligned window and partial tail; missing files, non-files and windows past EOF rejected while empty windows are accepted; cancellation before and during streaming); tests/boundary/mcp/binaryDiagnosticsMcp.test.ts (real bytes at exact offsets, advertised schemas valid, read-only effects); tests/boundary/mcp/analysisSummaryDetailMcp.test.ts (complete Evidence retained, runs summarized, CLI JSON matches MCP); src/contracts/toolEffects.test.ts (new tool registered as Evidence-only).
Base reproduction or other evidence: source-derived. On 2e45b4d there is no entropy computation anywhere under src/ or docs/ (git grep -il entropy returns nothing), so none of these outputs can be produced on the base.
User-visible CLI/MCP output (if applicable): rea inspect-byte-entropy sample.bin on the 32,768-byte demo file returns file_entropy: 5.4872, thresholds: {high: 7.2, low: 1}, eight 4096-byte blocks (0, 0, 4.4006, 4.4011, 7.9591, 7.9597, 7.9501, 0) and runs 0+8192 low, 16384+12288 high, 28672+4096 low, with confidence: observed and SHA-256 cd7a1e58...675f.
Remaining proof gaps: macOS only. The 100 MB timing in the docs is a single warm-cache local measurement; cold-disk I/O was not measured.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

All on macOS arm64 with Node 26.5.0 and npm 12.2.0 (.nvmrc pins 24.18.0, which is not installed on this host; engines allows >=26.0.0).

npm run test:focused -- src/domain/native/byteEntropy.test.ts \
  src/application/binaryDiagnostics/ByteEntropyService.test.ts \
  tests/boundary/mcp/binaryDiagnosticsMcp.test.ts \
  tests/boundary/mcp/analysisSummaryDetailMcp.test.ts \
  src/contracts/toolEffects.test.ts          # pass: 5 files, 17 tests
npm run check:fast                           # pass (typecheck, lint)
npm run check                                # pass (typecheck, lint:built 0 errors, format:check, knip, metadata:check, architecture guards)
npm run docs:check                           # pass
npm run verify:test-discovery                # pass: 952 test files discovered once
npm run metadata:check                       # pass
rea inspect-byte-entropy sample.bin          # demo sample, output as above

npm run check:pr was not run for this change.

Compatibility, safety, and release

Breaking changes or migration steps: none. Additive tool and CLI command; inspect_binary_layout behavior and schema are unchanged.
Real Hopper/Ghidra, browser, or OS coverage: not applicable to this tool (no engine involved); exercised on macOS only.
Package or release metadata impact: new tool and command appear in the generated catalog; authored skill reference line only. No dependency changes.
Security, privacy, process, or containment review: reads one caller-named absolute path read-only, no execution, child processes, writes or network. Output contains the caller's path, as with other file diagnostics. Retained memory scales with selected blocks, which is documented.

Author readiness

This PR was prepared with AI assistance (Claude Code and Grok). The declarations below are left for the human owner to complete.

Human owner: @mvanhorn

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

Summary by CodeRabbit

New Features
- Added byte-entropy analysis for local files, reporting whole-file entropy and SHA-256 alongside entropy by block and high- or low-entropy runs.
- Supports analysis of a selected byte range, with configurable block sizes, through the CLI and MCP.
- Added concise summary views for byte-entropy results, including retained summaries where available.
Documentation
- Added guidance on using entropy profiles to triage files before extraction.

Maintainer blocker verification

The artifact descriptor now has one retryable cleanup owner shared by CLI/MCP. Failed close retains that exact owner; subsequent access retries cleanup before admission, preserving the primary failure and secondary cleanup causes. Cancellation from open or final close maps to AnalysisCancelledError with the AbortSignal reason. If final close also fails, public projection retains cancellation as the execution failure and exposes incomplete cleanup until retry succeeds.

The producer rejects more than 32,768 selected rows before scanning or allocating retained results. This is an explicit output-capacity budget, not a file-format limit. A cap-sized sparse-file task retained 1,514,877 serialized bytes and showed approximately +21.2 MiB V8 heap / +18.1 MiB RSS on this macOS/Node 24 host; these are host-specific observations, not portable peak-memory guarantees. The refusal is retryable: false and directs callers to a smaller window or larger block size.

Latest focused ByteEntropy/MCP regressions: 12 passed, 1 skipped; npm run check:fast passed. They cover open cancellation, finalization cancellation plus failed close, cause ordering, retry ownership and public capacity projection. The /proc case was skipped on this host. The earlier public-input-helper finding cites morluto/preference; REA has no publicInputStrictObject policy/helper, so importing that unrelated wrapper is not applicable.

Restack

Local merge of this tip with origin/main (c537c52c9) is clean. This branch does not edit .github/workflows; main's workflow files are taken when the pull request merges. inspect_byte_entropy is registered from src/server/registerByteEntropyTool.ts, and inspect_binary_layout refuses an unretained summary before inspection.

npm run typecheck — pass on this tip and on the merged tree
npm run test:local — byte entropy, service, contracts, tool effects, and public input: 23 passed
npm run test:focused — binaryDiagnosticsMcp, analysisSummaryDetailMcp, and productCatalog on the merged tree: 16 passed

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2331 **Author:** [@mvanhorn](https://github.com/mvanhorn) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/22/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/byte-entropy-profile` --- ### 📝 Commits (10+) - [`519ebaa`](https://github.com/morluto/rea/commit/519ebaac4c68f6fecffd573ee1f96640462b5cc1) feat(diagnostics): report a byte entropy profile for one local file - [`b82e190`](https://github.com/morluto/rea/commit/b82e190c3ed73eb29ad56a1ea4c66a19e89144ab) fix(diagnostics): flag zero-size files as not shown empty - [`b87f511`](https://github.com/morluto/rea/commit/b87f51184c6898e00552ca5c61ad1007e58d0020) fix(diagnostics): merge main and preserve entropy boundary failures - [`7a67707`](https://github.com/morluto/rea/commit/7a677078cffd1e1cdba47e6dcb9fe70fa74601df) fix(mcp): reject unsupported diagnostic summaries before inspection - [`eccc741`](https://github.com/morluto/rea/commit/eccc741a4477c556e685f63bb4a4e9214b8081ef) Merge remote-tracking branch 'origin/main' into fix/byte-entropy-profile - [`a156b1d`](https://github.com/morluto/rea/commit/a156b1dd30c8cc32cd693703c08bea9f864c124e) fix(ci): repair integrated checks and Hopper test baseline - [`a2f6136`](https://github.com/morluto/rea/commit/a2f61367161173997ec18a6380e58f33ccf89c5f) test(entropy): use shared MCP schema verification after merge - [`4df3682`](https://github.com/morluto/rea/commit/4df3682a053046df98ec7ad3d645a48eb5f4dc4a) chore: merge current main into PR #2331 - [`f54b2ae`](https://github.com/morluto/rea/commit/f54b2ae186a8cd3fabee6d44e9247b6560a30381) Merge branch 'main' into fix/byte-entropy-profile - [`3a794e9`](https://github.com/morluto/rea/commit/3a794e9d75ddb138a1b92c26299d8f63911f2fc9) Merge branch 'main' into fix/byte-entropy-profile ### 📊 Changes **29 files changed** (+1510 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+2 -0) 📝 `docs/binary-diagnostics.md` (+84 -0) 📝 `docs/user-visible-errors.csv` (+2 -0) 📝 `scripts/lib/catalog-builders.mjs` (+10 -1) 📝 `src/application/CapabilityInventory.ts` (+1 -0) 📝 `src/application/InvestigationProviders.ts` (+7 -0) ➕ `src/application/binaryDiagnostics/ByteEntropyService.test.ts` (+373 -0) ➕ `src/application/binaryDiagnostics/ByteEntropyService.ts` (+355 -0) 📝 `src/cli/binaryDiagnosticsCommands.ts` (+40 -0) 📝 `src/cliCommandNames.ts` (+1 -0) 📝 `src/composition/binaryDiagnostics.ts` (+7 -0) ➕ `src/contracts/native/binaryDiagnosticsToolContracts.test.ts` (+33 -0) 📝 `src/contracts/native/binaryDiagnosticsToolContracts.ts` (+36 -0) 📝 `src/contracts/toolContractExamples.ts` (+1 -0) 📝 `src/contracts/toolEffects.test.ts` (+1 -0) 📝 `src/contracts/toolEffects.ts` (+1 -0) 📝 `src/contracts/toolOutputSchemaGroups.ts` (+9 -0) 📝 `src/domain/analysisView/analysisView.ts` (+29 -0) ➕ `src/domain/analysisView/byteEntropyView.ts` (+66 -0) ➕ `src/domain/native/byteEntropy.test.ts` (+53 -0) _...and 9 more files_ </details> ### 📄 Description ## Summary REA can now report a byte entropy profile for one local file: `inspect_byte_entropy` (CLI `inspect-byte-entropy`) reads an absolute path offline and returns file-wide SHA-256 and Shannon entropy, per-block entropy for an optional byte window, and adjacent `high` and `low` runs with exact file offsets. It runs no analysis engine, executes nothing, extracts nothing and makes no network requests. No issue is linked. This is a new diagnostics tool, proposed here with the implementation. ## Demo ![Byte entropy profile demo](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/2026-10-12-020-feat-byte-entropy-profile/preview.gif) [Full-quality MP4](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/2026-10-12-020-feat-byte-entropy-profile/out.mp4) The sample is a generated 32 KB file (8 KB zeros, 8 KB text, 12 KB random bytes, 4 KB zeros), and every number in the video is copied from the CLI output on this branch. ## Problem and expected behavior Before opening an unknown blob in a disassembler or trying to extract it, a useful first question is where the padding, the plain data and the packed or compressed regions sit. `inspect_binary_layout`, `inspect_pe_resources` and `inspect_recorded_crash` report structure, but none of them reports entropy, so on `upstream/main` (`2e45b4d`) there is no file-wide entropy, no block series and no high or low runs to cite as Evidence. With this change, one call on a regular file returns observed Evidence with `artifact` (path, byte size, SHA-256), `block_size`, `window`, `file_entropy`, `thresholds`, `blocks` (`offset`, `length`, `entropy`) and `runs` (`offset`, `length`, `classification`). On the demo sample that is `file_entropy: 5.4872`, two text blocks at 4.4006 and 4.4011 left unclassified, and three runs: `0+8192 low`, `16384+12288 high`, `28672+4096 low`. ## Change and scope `ByteEntropyService` (`src/application/binaryDiagnostics/ByteEntropyService.ts`) backs both CLI and MCP. It streams the file once with one block-sized buffer and a 256-entry histogram, hashing and counting every byte so SHA-256 and `file_entropy` always cover the whole file, while blocks are recorded only inside the selected window. Blocks start at `offset` (default 0); an omitted `length` runs to EOF, and the final block may be shorter. `block_size` must be a power of two from 256 through 65536, default 4096. Entropy is bits per byte from 0 to 8, rounded to four places. Adjacent blocks with reported entropy >= 7.2 merge into `high` runs and blocks <= 1.0 into `low` runs; anything in between breaks a run. Classification uses the rounded value that is reported, so a reader can recompute it from the output. The thresholds are returned in the result, and three limitations go with every result: entropy cannot tell compressed from encrypted bytes, small blocks can underestimate entropy, and file-wide fields cover the whole file while blocks and runs cover only the window. Empty files and empty windows are accepted with entropy 0 by convention. Errors are typed rather than clipped or guessed: a relative path, a missing path, a non-regular file, a window past EOF or a file larger than `Number.MAX_SAFE_INTEGER` is an input error; `EACCES`/`EPERM` is access denied; a change in device, inode, size, mtime or ctime between the first `stat` and the end of the read is an artifact-changed error; cancellation is checked before and during the read. `detail: "summary"` retains the complete Evidence in the session and returns file entropy, block and run counts, thresholds, the window and the ten longest runs (ties by offset), with `parent_evidence_id` for the full series. `inspect_analysis_view` gains a summary projection for this parent operation; any other view kind is rejected with a pointer to the parent Evidence. Summary detail requires a session that can retain Evidence. Memory: the read itself is bounded by one block buffer, but the returned block list grows with the number of selected blocks, and summary detail still retains that full series. The docs say so and point to a narrower window or a larger block size. A local run on a 100,000,000-byte cached file with 4096-byte blocks, recorded in `docs/binary-diagnostics.md`, took about 1.1 s and 145 MiB peak RSS for 24,415 blocks; complete Evidence serialized to 1,122,303 bytes and the summary to 2,705 bytes. Out of scope: extraction or carving of high-entropy regions, compression or encryption classification, configurable thresholds, and any other summary view kinds for this parent. ## Contract and boundary impact - Semantic owner and earliest changed stage: domain schemas and math in `src/domain/native/byteEntropy.ts` (`inspectByteEntropyInputSchema`, `byteEntropySchema`, `entropyFromCounts`, `mergeEntropyRuns`); I/O in `src/application/binaryDiagnostics/ByteEntropyService.ts`; summary projection in `src/domain/analysisView/byteEntropyView.ts`, dispatched from `projectAnalysisView` in `src/domain/analysisView/analysisView.ts`. - Contribution scope fit and linked maintainer decision: format inspection that only reads one caller-selected local file, which CONTRIBUTING lists as supported; no target execution, injection, mutation or protection recovery, so no maintainer scope decision is required. Not applicable. - CLI and MCP/tool-catalog contract: new MCP tool `inspect_byte_entropy` (`src/contracts/native/binaryDiagnosticsToolContracts.ts`, output schema in `src/contracts/toolOutputSchemaGroups.ts`, example in `src/contracts/toolContractExamples.ts`, effect `evidence` in `src/contracts/toolEffects.ts`, availability in `src/application/CapabilityInventory.ts`) registered in `src/server/registerBinaryDiagnosticsTools.ts` and `src/server/createServer.ts`; new CLI command `inspect-byte-entropy <path> --block-size --offset --length` in `src/cli/binaryDiagnosticsCommands.ts` and `src/cliCommandNames.ts`. `inspect_analysis_view` `parent_operation` adds `inspect_byte_entropy`. Existing tools are unchanged; `inspect_binary_layout` registration now shares one runner with the new tool. - Provider, bridge, target-format, or platform compatibility: none. Provider-free (`src/composition/binaryDiagnostics.ts`), works on any regular file, no Hopper, Ghidra or external binary needed. - Evidence, artifact, provenance, or reconstruction contract: `confidence: "observed"`, provider `byte-entropy` v1, artifact identity is the file SHA-256, parameters record path, block size, offset and length (when given), and limitations are carried into the summary view. - Target selection, authority, upstream behavior, and cleanup details (if applicable): the caller names one absolute path; the file is opened `O_RDONLY | O_NONBLOCK` and closed in `finally`. No processes, no writes, no temp files. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: authored skill line in `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` ("Use `inspect_byte_entropy` to triage high- and low-entropy ranges before extraction."); docs in `docs/binary-diagnostics.md` and two rows (91, 92) in `docs/user-visible-errors.csv`. `metadata:check` and `docs:check` pass; no build outputs committed. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: `src/domain/native/byteEntropy.test.ts` (known Shannon answers including partial and empty blocks; threshold-edge merging that skips unclassified blocks and gaps; rejection of unsafe windows, unknown fields and non-power-of-two or out-of-range block sizes); `src/application/binaryDiagnostics/ByteEntropyService.test.ts` (whole-file identity and entropy with an unaligned window and partial tail; missing files, non-files and windows past EOF rejected while empty windows are accepted; cancellation before and during streaming); `tests/boundary/mcp/binaryDiagnosticsMcp.test.ts` (real bytes at exact offsets, advertised schemas valid, read-only effects); `tests/boundary/mcp/analysisSummaryDetailMcp.test.ts` (complete Evidence retained, runs summarized, CLI JSON matches MCP); `src/contracts/toolEffects.test.ts` (new tool registered as Evidence-only). - Base reproduction or other evidence: source-derived. On `2e45b4d` there is no entropy computation anywhere under `src/` or `docs/` (`git grep -il entropy` returns nothing), so none of these outputs can be produced on the base. - User-visible CLI/MCP output (if applicable): `rea inspect-byte-entropy sample.bin` on the 32,768-byte demo file returns `file_entropy: 5.4872`, `thresholds: {high: 7.2, low: 1}`, eight 4096-byte blocks (`0, 0, 4.4006, 4.4011, 7.9591, 7.9597, 7.9501, 0`) and runs `0+8192 low`, `16384+12288 high`, `28672+4096 low`, with `confidence: observed` and SHA-256 `cd7a1e58...675f`. - Remaining proof gaps: macOS only. The 100 MB timing in the docs is a single warm-cache local measurement; cold-disk I/O was not measured. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed All on macOS arm64 with Node 26.5.0 and npm 12.2.0 (`.nvmrc` pins 24.18.0, which is not installed on this host; `engines` allows `>=26.0.0`). ``` npm run test:focused -- src/domain/native/byteEntropy.test.ts \ src/application/binaryDiagnostics/ByteEntropyService.test.ts \ tests/boundary/mcp/binaryDiagnosticsMcp.test.ts \ tests/boundary/mcp/analysisSummaryDetailMcp.test.ts \ src/contracts/toolEffects.test.ts # pass: 5 files, 17 tests npm run check:fast # pass (typecheck, lint) npm run check # pass (typecheck, lint:built 0 errors, format:check, knip, metadata:check, architecture guards) npm run docs:check # pass npm run verify:test-discovery # pass: 952 test files discovered once npm run metadata:check # pass rea inspect-byte-entropy sample.bin # demo sample, output as above ``` `npm run check:pr` was not run for this change. ## Compatibility, safety, and release - Breaking changes or migration steps: none. Additive tool and CLI command; `inspect_binary_layout` behavior and schema are unchanged. - Real Hopper/Ghidra, browser, or OS coverage: not applicable to this tool (no engine involved); exercised on macOS only. - Package or release metadata impact: new tool and command appear in the generated catalog; authored skill reference line only. No dependency changes. - Security, privacy, process, or containment review: reads one caller-named absolute path read-only, no execution, child processes, writes or network. Output contains the caller's path, as with other file diagnostics. Retained memory scales with selected blocks, which is documented. ## Author readiness This PR was prepared with AI assistance (Claude Code and Grok). The declarations below are left for the human owner to complete. Human owner: @mvanhorn - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims.  ## Summary by CodeRabbit * **New Features** * Added byte-entropy analysis for local files, reporting whole-file entropy and SHA-256 alongside entropy by block and high- or low-entropy runs. * Supports analysis of a selected byte range, with configurable block sizes, through the CLI and MCP. * Added concise summary views for byte-entropy results, including retained summaries where available. * **Documentation** * Added guidance on using entropy profiles to triage files before extraction.   ### Maintainer blocker verification The artifact descriptor now has one retryable cleanup owner shared by CLI/MCP. Failed close retains that exact owner; subsequent access retries cleanup before admission, preserving the primary failure and secondary cleanup causes. Cancellation from open or final close maps to `AnalysisCancelledError` with the AbortSignal reason. If final close also fails, public projection retains cancellation as the execution failure and exposes incomplete cleanup until retry succeeds. The producer rejects more than 32,768 selected rows before scanning or allocating retained results. This is an explicit output-capacity budget, not a file-format limit. A cap-sized sparse-file task retained 1,514,877 serialized bytes and showed approximately +21.2 MiB V8 heap / +18.1 MiB RSS on this macOS/Node 24 host; these are host-specific observations, not portable peak-memory guarantees. The refusal is `retryable: false` and directs callers to a smaller window or larger block size. Latest focused ByteEntropy/MCP regressions: 12 passed, 1 skipped; `npm run check:fast` passed. They cover open cancellation, finalization cancellation plus failed close, cause ordering, retry ownership and public capacity projection. The `/proc` case was skipped on this host. The earlier public-input-helper finding cites `morluto/preference`; REA has no `publicInputStrictObject` policy/helper, so importing that unrelated wrapper is not applicable.  ## Restack Local merge of this tip with origin/main (`c537c52c9`) is clean. This branch does not edit `.github/workflows`; main's workflow files are taken when the pull request merges. `inspect_byte_entropy` is registered from `src/server/registerByteEntropyTool.ts`, and `inspect_binary_layout` refuses an unretained summary before inspection. - `npm run typecheck` — pass on this tip and on the merged tree - `npm run test:local` — byte entropy, service, contracts, tool effects, and public input: 23 passed - `npm run test:focused` — `binaryDiagnosticsMcp`, `analysisSummaryDetailMcp`, and `productCatalog` on the merged tree: 16 passed  <div><a href="https://cursor.com/agents/bc-bcce92c8-f0b5-5ff3-8efa-652bc77de903?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-bcce92c8-f0b5-5ff3-8efa-652bc77de903&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:04:30 +00:00