morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3634] [MERGED] feat(artifacts): inspect and extract Godot PCK with retained facts #5859

New issue

Closed

opened 2026-10-31 01:09:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:09:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3634
Author: @0xthearchitect
Created: 10/14/2026
Status: ✅ Merged
Merged: 10/20/2026
Merged by: @gensokyo-dev

Base: main ← Head: feat/godot-pck-reader

📝 Commits (6)

6edfe0a feat(artifacts): read Godot PCK packs
d8d5e86 fix(artifacts): bound Godot PCK metadata and own member streams
00bedf6 chore: merge latest main into Godot PCK follow-up
047fe3a feat(artifacts): complete Godot PCK facts, integrity and selected views
f1535a5 fix(artifacts): preserve atomic-root inventory Evidence
0f818b8 fix(artifacts): bound aggregate PCK member expansion

📊 Changes

58 files changed (+3298 additions, -361 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+19 -0)
📝 README.md (+28 -27)
📝 docs/cli.md (+11 -0)
📝 docs/error-contract.schema.json (+47 -17)
➕ docs/godot-pck.md (+108 -0)
📝 docs/index.md (+13 -13)
📝 src/application/BinaryTargetResolver.ts (+5 -1)
📝 src/artifacts/ArtifactHash.ts (+10 -1)
📝 src/artifacts/ArtifactReader.ts (+54 -9)
📝 src/artifacts/AsarArtifactReader.ts (+10 -14)
📝 src/artifacts/DirectoryArtifactReader.ts (+1 -1)
➕ src/artifacts/GodotPckArtifactReader.test.ts (+315 -0)
➕ src/artifacts/GodotPckArtifactReader.ts (+564 -0)
➕ src/artifacts/GodotPckLocation.ts (+87 -0)
📝 src/artifacts/MachOSliceArtifactReader.ts (+1 -1)
📝 src/artifacts/ZipArtifactReader.test.ts (+4 -23)
📝 src/artifacts/ZipArtifactReader.ts (+7 -7)
📝 src/artifacts/extraction/ArtifactExtraction.ts (+45 -9)
📝 src/artifacts/inventory/ArtifactGraphConstruction.test.ts (+1 -1)
📝 src/artifacts/inventory/ArtifactGraphConstruction.ts (+22 -3)

...and 38 more files

📄 Description

Summary

Complete the original author's static Godot PCK reader with shared CLI/MCP inventory and extraction. Ordinary formats 2, 3 and 4 now retain engine/header/directory facts, verify declared MD5 through the shared integrity policy, and expose stable occurrence/content identities through selected inventory views. Refs #3633 and #3631; the broader resource/GDScript/signal work in #3631 remains open.

Use inspect_artifact({"detail":"summary"}) for a standalone active pack, then inspect_analysis_view with exact paths or occurrence IDs, batch content IDs and selected fields. Complete Evidence remains retained; unknown encrypted coverage never becomes a claim of absence. CLI supports inspect-artifact PATH --detail summary --evidence_output BUNDLE --json; export never overwrites. Existing complete delivery and native target defaults remain compatible. CLI artifact inspection/extraction also locate exact embedded EOF trailers.

Approach and boundaries

The PCK reader owns header/directory parsing, positional reads and descriptor/stream cleanup. Every readable logical occurrence, including repeated ranges and zero-MD5 claims, consumes the shared ZIP/PCK expansion budget: max(256 MiB, 500 × encoded PCK span), with exact-number accounting. Carrier bytes do not grant budget. The entire PCK directory passes this check before any member is yielded or hashed; extraction uses the same owner. Bounded aliases remain valid and encrypted content is not read. Directory metadata has a 64 MiB encoded retention budget with expansion checked before allocation; parsing uses a 64 KiB read window. Unfinished reads drain before owned descriptors close. Borrowed descriptors remain caller-owned.
Atomic-root inventories retain their existing limitation and Evidence semantics; Godot-specific unknown coverage stays in the PCK producer.
One canonical integrity claim represents SHA-256 or MD5; shared streaming verification implements strict versus record-and-continue. Observed SHA-256 remains content identity. Legacy SHA contradiction identities/fields remain compatible. Extraction verifies the complete carrier before creating output, including header-only replacements with identical member bytes.
Shared inventory views validate authenticated facts once, borrow sealed snapshots and lazily index IDs/paths. Cache lifetime follows the immutable result; every projection retains current parent identity and subject checks. Filters intersect, pagination follows selected canonical order, and identity links survive field selection. Producer coverage is separate from retained graph/page completeness.
No target execution, decryption, key recovery, resource decoding or GDScript decompilation. Extraction writes only a fresh REA-owned output tree; optional CLI export writes the caller-selected bundle. No dependencies, installers, package metadata or deep-provider changes. Maintainer scope covers the ordinary archive slice.

Executed validation

npm run check, npm run docs:check, npm run verify:test-discovery — passed on the final source tree, including generated error-schema consistency and authored skill/docs.
Source regressions: 111 tests across 10 files passed, including reader bounds/cancellation, ownership, inventory identity, cached context binding and Evidence authentication/comparison.
Final focused CLI/MCP/filesystem/provider suite: 61 tests passed. Actual compiled CLI covers formats 2/3/4, embedded carriers, integrity policies, encrypted unknowns, no-overwrite export and historical reads. The SDK validates named advertised JSON Schemas and exact batch/field responses. ASAR/ZIP integrity, extraction identity and cleanup regressions remain covered.
Atomic-root compatibility and Godot public workflows: 15 tests across 4 files passed, including the exact CLI/MCP comparison regression caught by CI. Tests retain their original assertions. Full check, docs and discovery checks passed again after the producing-boundary correction.
Aggregate expansion regression: four new metadata-admission controls fail on f1535a575 (formats 2/3/4 and embedded-carrier budget), using a roughly 0.32 MiB fixture declaring 275 MiB across 1,100 aliases; no amplified payload hashing/writes were run on the affected revision. After correction, all 35 reader/budget tests and 35 public CLI/MCP/ZIP/coverage tests passed. Both integrity policies reject the over-budget pack with typed truncation before extraction output exists. All 13 real corpus packs were verified again against the independent oracle.
Shared CLI delivery regression: all 5 Unity CLI/MCP acceptance cases passed, including export and caller-selected symlink-parent paths.
Real compiled CLI: 13 packs checked against an independent Python struct/hashlib oracle, including official Godot 4.4.1 (PCK 2), 4.6.1 (PCK 3), 4.7.1 (PCK 4) SDK outputs and Dodge the Creeps. All 15,063 plaintext member hashes/bytes agreed; plain small packs and the public game's 48 members extracted byte-for-byte. Encrypted samples retain metadata only. The public pack SHA-256 is 989666e8c4619bc4022903cfb58fd29d88b881a136452952a3df06b8cf99ad70.
Two constructed official-editor ELF carriers containing official PCK 3/4 packs verified whole-carrier SHA, native content classification, exact bounds and member hashes. These are constructed carriers, not shipped embedded games; no target was executed by REA.

Measured completed task

The actual stdio SDK finds three exact paths in an official 5,000-unique-member PCK 3, joins their observed SHA-256/size, unlinks the owned input copy and retrieves their stable identities again. Both implementations use summary delivery and the best available batch path query; the original reader integrated with current main must read all 5,001 content nodes for the join. The follow-up selects three nodes. Both complete in 6 calls including open/close, with independent oracle, schema and provenance assertions and zero retries.

Two alternating pairs on pinned revisions, in baseline/follow-up/follow-up/baseline order:

Cost	Baseline `00bedf665`	Follow-up `047fe3a84`
Tool calls	6	6
Selected input declarations + all calls/replies, tokenx estimate	832,709	23,492
All input declarations + calls/replies, tokenx estimate	1,050,177	240,960
Entire recorded protocol including output schemas, tokenx estimate	1,811,194	1,006,151
Whole task elapsed, seconds	28.60–31.94	28.30–29.95
Whole process-family CPU, seconds	31.63–36.90	29.01–33.81
Sampled peak process-family RSS, MiB	1042–1110	1005–1019

Includes staging/input hashing, discovery, advertised-schema validation, protocol envelopes, calls and teardown; build/install excluded. Actual model/host-rendered tokens are unmeasured. Fresh processes and analysis caches, two CPUs, bounded heaps, disabled Node compile caches, 20 ms process-family memory sampling; OS cache and host load uncontrolled. Runtime and RSS varied between earlier probes, so no general speed or memory improvement is claimed. Cold analysis, first retained view and warm retained queries are recorded separately; warm reads after source removal are historical facts, not current filesystem claims.

The final resource-bound tree passed the complete SDK task again: 23,492 / 240,960 / 1,006,151 tokenx estimates respectively, 18.07 s elapsed, 19.74 s CPU and 1081 MiB sampled peak RSS. The paired table pins its earlier revisions; this final probe covers the atomic-root compatibility and aggregate expansion corrections.

Remaining verification limits

Real corpus/SDK verification ran on Linux. PCK 1, signed/section-only embedded discovery, patch/removal entries and encrypted contents are unsupported. Sparse-bundle unknown coverage has synthetic tests; real sparse producers and shipped PE/Mach-O embedded games were not verified. Complete Evidence preserves the observed header/directory facts and limitations; it is not a retained copy of every source payload.

Author readiness

Human owner: @gensokyo-dev

The repository operator explicitly delegated these declarations and authorized ready review, CI follow-up and normal merge in this session. Codex reviewed the final diff and executed evidence; independent human review was not observed. Original author commits remain in this PR's history.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3634 **Author:** [@0xthearchitect](https://github.com/0xthearchitect) **Created:** 10/14/2026 **Status:** ✅ Merged **Merged:** 10/20/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `feat/godot-pck-reader` --- ### 📝 Commits (6) - [`6edfe0a`](https://github.com/morluto/rea/commit/6edfe0a1b9d07a7646e2bbdb78691dd7b09558a8) feat(artifacts): read Godot PCK packs - [`d8d5e86`](https://github.com/morluto/rea/commit/d8d5e86b0de2aa34a44696504419e445957776e5) fix(artifacts): bound Godot PCK metadata and own member streams - [`00bedf6`](https://github.com/morluto/rea/commit/00bedf66588941a888276f63774efa7248e38581) chore: merge latest main into Godot PCK follow-up - [`047fe3a`](https://github.com/morluto/rea/commit/047fe3a84c76297e6163b3d229ea49878c2a12a3) feat(artifacts): complete Godot PCK facts, integrity and selected views - [`f1535a5`](https://github.com/morluto/rea/commit/f1535a575eb3866e4b1d559215bf9b9e69094f22) fix(artifacts): preserve atomic-root inventory Evidence - [`0f818b8`](https://github.com/morluto/rea/commit/0f818b8beaf3d7299d33d0c50e9b7f0722d4ecdc) fix(artifacts): bound aggregate PCK member expansion ### 📊 Changes **58 files changed** (+3298 additions, -361 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+19 -0) 📝 `README.md` (+28 -27) 📝 `docs/cli.md` (+11 -0) 📝 `docs/error-contract.schema.json` (+47 -17) ➕ `docs/godot-pck.md` (+108 -0) 📝 `docs/index.md` (+13 -13) 📝 `src/application/BinaryTargetResolver.ts` (+5 -1) 📝 `src/artifacts/ArtifactHash.ts` (+10 -1) 📝 `src/artifacts/ArtifactReader.ts` (+54 -9) 📝 `src/artifacts/AsarArtifactReader.ts` (+10 -14) 📝 `src/artifacts/DirectoryArtifactReader.ts` (+1 -1) ➕ `src/artifacts/GodotPckArtifactReader.test.ts` (+315 -0) ➕ `src/artifacts/GodotPckArtifactReader.ts` (+564 -0) ➕ `src/artifacts/GodotPckLocation.ts` (+87 -0) 📝 `src/artifacts/MachOSliceArtifactReader.ts` (+1 -1) 📝 `src/artifacts/ZipArtifactReader.test.ts` (+4 -23) 📝 `src/artifacts/ZipArtifactReader.ts` (+7 -7) 📝 `src/artifacts/extraction/ArtifactExtraction.ts` (+45 -9) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.test.ts` (+1 -1) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.ts` (+22 -3) _...and 38 more files_ </details> ### 📄 Description ## Summary Complete the original author's static Godot PCK reader with shared CLI/MCP inventory and extraction. Ordinary formats 2, 3 and 4 now retain engine/header/directory facts, verify declared MD5 through the shared integrity policy, and expose stable occurrence/content identities through selected inventory views. Refs #3633 and #3631; the broader resource/GDScript/signal work in #3631 remains open. Use `inspect_artifact({"detail":"summary"})` for a standalone active pack, then `inspect_analysis_view` with exact paths or occurrence IDs, batch content IDs and selected fields. Complete Evidence remains retained; unknown encrypted coverage never becomes a claim of absence. CLI supports `inspect-artifact PATH --detail summary --evidence_output BUNDLE --json`; export never overwrites. Existing complete delivery and native target defaults remain compatible. CLI artifact inspection/extraction also locate exact embedded EOF trailers. ## Approach and boundaries - The PCK reader owns header/directory parsing, positional reads and descriptor/stream cleanup. Every readable logical occurrence, including repeated ranges and zero-MD5 claims, consumes the shared ZIP/PCK expansion budget: max(256 MiB, 500 × encoded PCK span), with exact-number accounting. Carrier bytes do not grant budget. The entire PCK directory passes this check before any member is yielded or hashed; extraction uses the same owner. Bounded aliases remain valid and encrypted content is not read. Directory metadata has a 64 MiB encoded retention budget with expansion checked before allocation; parsing uses a 64 KiB read window. Unfinished reads drain before owned descriptors close. Borrowed descriptors remain caller-owned. - Atomic-root inventories retain their existing limitation and Evidence semantics; Godot-specific unknown coverage stays in the PCK producer. - One canonical integrity claim represents SHA-256 or MD5; shared streaming verification implements strict versus record-and-continue. Observed SHA-256 remains content identity. Legacy SHA contradiction identities/fields remain compatible. Extraction verifies the complete carrier before creating output, including header-only replacements with identical member bytes. - Shared inventory views validate authenticated facts once, borrow sealed snapshots and lazily index IDs/paths. Cache lifetime follows the immutable result; every projection retains current parent identity and subject checks. Filters intersect, pagination follows selected canonical order, and identity links survive field selection. Producer coverage is separate from retained graph/page completeness. - No target execution, decryption, key recovery, resource decoding or GDScript decompilation. Extraction writes only a fresh REA-owned output tree; optional CLI export writes the caller-selected bundle. No dependencies, installers, package metadata or deep-provider changes. [Maintainer scope](https://github.com/morluto/rea/issues/3633#issuecomment-6176900760) covers the ordinary archive slice. ## Executed validation - `npm run check`, `npm run docs:check`, `npm run verify:test-discovery` — passed on the final source tree, including generated error-schema consistency and authored skill/docs. - Source regressions: 111 tests across 10 files passed, including reader bounds/cancellation, ownership, inventory identity, cached context binding and Evidence authentication/comparison. - Final focused CLI/MCP/filesystem/provider suite: 61 tests passed. Actual compiled CLI covers formats 2/3/4, embedded carriers, integrity policies, encrypted unknowns, no-overwrite export and historical reads. The SDK validates named advertised JSON Schemas and exact batch/field responses. ASAR/ZIP integrity, extraction identity and cleanup regressions remain covered. - Atomic-root compatibility and Godot public workflows: 15 tests across 4 files passed, including the exact CLI/MCP comparison regression caught by CI. Tests retain their original assertions. Full check, docs and discovery checks passed again after the producing-boundary correction. - Aggregate expansion regression: four new metadata-admission controls fail on `f1535a575` (formats 2/3/4 and embedded-carrier budget), using a roughly 0.32 MiB fixture declaring 275 MiB across 1,100 aliases; no amplified payload hashing/writes were run on the affected revision. After correction, all 35 reader/budget tests and 35 public CLI/MCP/ZIP/coverage tests passed. Both integrity policies reject the over-budget pack with typed truncation before extraction output exists. All 13 real corpus packs were verified again against the independent oracle. - Shared CLI delivery regression: all 5 Unity CLI/MCP acceptance cases passed, including export and caller-selected symlink-parent paths. - Real compiled CLI: 13 packs checked against an independent Python struct/hashlib oracle, including official Godot 4.4.1 (PCK 2), 4.6.1 (PCK 3), 4.7.1 (PCK 4) SDK outputs and [Dodge the Creeps](https://godotengine.github.io/godot-demo-projects/2d/dodge_the_creeps/index.pck). All 15,063 plaintext member hashes/bytes agreed; plain small packs and the public game's 48 members extracted byte-for-byte. Encrypted samples retain metadata only. The public pack SHA-256 is `989666e8c4619bc4022903cfb58fd29d88b881a136452952a3df06b8cf99ad70`. - Two constructed official-editor ELF carriers containing official PCK 3/4 packs verified whole-carrier SHA, native content classification, exact bounds and member hashes. These are constructed carriers, not shipped embedded games; no target was executed by REA. ## Measured completed task The actual stdio SDK finds three exact paths in an official 5,000-unique-member PCK 3, joins their observed SHA-256/size, unlinks the owned input copy and retrieves their stable identities again. Both implementations use summary delivery and the best available batch path query; the original reader integrated with current main must read all 5,001 content nodes for the join. The follow-up selects three nodes. Both complete in 6 calls including open/close, with independent oracle, schema and provenance assertions and zero retries. Two alternating pairs on pinned revisions, in baseline/follow-up/follow-up/baseline order: | Cost | Baseline `00bedf665` | Follow-up `047fe3a84` | | --- | ---: | ---: | | Tool calls | 6 | 6 | | Selected input declarations + all calls/replies, tokenx estimate | 832,709 | 23,492 | | All input declarations + calls/replies, tokenx estimate | 1,050,177 | 240,960 | | Entire recorded protocol including output schemas, tokenx estimate | 1,811,194 | 1,006,151 | | Whole task elapsed, seconds | 28.60–31.94 | 28.30–29.95 | | Whole process-family CPU, seconds | 31.63–36.90 | 29.01–33.81 | | Sampled peak process-family RSS, MiB | 1042–1110 | 1005–1019 | Includes staging/input hashing, discovery, advertised-schema validation, protocol envelopes, calls and teardown; build/install excluded. Actual model/host-rendered tokens are unmeasured. Fresh processes and analysis caches, two CPUs, bounded heaps, disabled Node compile caches, 20 ms process-family memory sampling; OS cache and host load uncontrolled. Runtime and RSS varied between earlier probes, so no general speed or memory improvement is claimed. Cold analysis, first retained view and warm retained queries are recorded separately; warm reads after source removal are historical facts, not current filesystem claims. The final resource-bound tree passed the complete SDK task again: 23,492 / 240,960 / 1,006,151 tokenx estimates respectively, 18.07 s elapsed, 19.74 s CPU and 1081 MiB sampled peak RSS. The paired table pins its earlier revisions; this final probe covers the atomic-root compatibility and aggregate expansion corrections. ## Remaining verification limits Real corpus/SDK verification ran on Linux. PCK 1, signed/section-only embedded discovery, patch/removal entries and encrypted contents are unsupported. Sparse-bundle unknown coverage has synthetic tests; real sparse producers and shipped PE/Mach-O embedded games were not verified. Complete Evidence preserves the observed header/directory facts and limitations; it is not a retained copy of every source payload. ## Author readiness Human owner: @gensokyo-dev The repository operator explicitly delegated these declarations and authorized ready review, CI follow-up and normal merge in this session. Codex reviewed the final diff and executed evidence; independent human review was not observed. Original author commits remain in this PR's history. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:09:58 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:47:38 +00:00

[PR #6748] [MERGED] fix(ghidra): report kernel OOM kills as a memory constraint #7405

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5859

No description provided.

Rows
Columns