[PR #4362] feat(unreal): inspect Unreal pak footers and indexes #6204

Open
opened 2026-10-31 01:12:39 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4362
Author: @8rulerstar
Created: 10/16/2026
Status: 🔄 Open

Base: main ← Head: feat/unreal-pak


📝 Commits (10+)

  • 256c3c1 feat(unreal): decode pak footers and indexes for versions 1 to 12
  • 6e03715 feat(unreal): read pak indexes from stable file ranges through a provider and service
  • 411fed9 feat(unreal): expose inspect_unreal_pak through CLI, MCP and analysis views
  • cb42c6b test(unreal): cover pak versions, index forms, failures and CLI/MCP delivery
  • 486d430 docs(unreal): add the pak index guide, CLI reference, skill notes and a real-file verifier
  • 587cbf5 refactor(unreal): drop an unused index type export
  • e1e3864 fix(unreal): reject inconsistent counts, leftover listing bytes and unknown compression slots
  • 52ab2fb docs(unreal): document the pruned directory index and the new consistency failures
  • 77ceb30 refactor(artifacts): share admission and change checks between whole-file and ranged stable reads
  • 539e754 fix(unreal): check index SHA-1s, entry placement and built path size

📊 Changes

43 files changed (+4727 additions, -49 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+5 -0)
📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+10 -0)
📝 README.md (+1 -0)
📝 docs/.vitepress/config.ts (+1 -0)
📝 docs/cli.md (+14 -0)
➕ docs/unreal-pak.md (+211 -0)
📝 package.json (+1 -0)
📝 scripts/lib/module-boundaries.mjs (+1 -0)
➕ scripts/verify-unreal-pak.mjs (+145 -0)
📝 src/application/CapabilityInventory.ts (+1 -0)
➕ src/application/unreal/UnrealPakPort.ts (+20 -0)
➕ src/application/unreal/UnrealPakService.ts (+134 -0)
📝 src/artifacts/readStableArtifact.ts (+169 -37)
📝 src/cli.ts (+2 -0)
➕ src/cli/unrealCommands.ts (+55 -0)
📝 src/cliCommandNames.ts (+1 -0)
📝 src/composition/capabilityDeclarations.ts (+8 -0)
📝 src/composition/capabilityServices.ts (+6 -0)
➕ src/composition/unreal.ts (+6 -0)
📝 src/contracts/analysisViewToolContracts.ts (+1 -1)

...and 23 more files

📄 Description

Thank you for merging main into this branch to refresh CI. I merged the latest main again and fixed five reader boundaries found in a self-review; details are under Executed validation.

Summary

Adds inspect_unreal_pak (CLI: inspect-unreal-pak): static inspection of one local Unreal .pak (versions 1 to 12). It reads the footer and the index regions the footer names and returns each entry's path, offset, sizes, compression method, block facts, encryption and deletion flags and stored SHA-1. File data is never decoded. Refs #3625, narrower: no extraction, decompression, decryption or keys, no IoStore or .uasset.

Behavior and boundaries

  • src/unreal/UnrealPakFile.ts owns format interpretation; src/unreal/UnrealPakPathHash.ts owns the path hash rule. The service projects the decoded regions into Evidence locations, and the analysis view only projects retained Evidence (kind unreal-pak, collections entries by indices/paths and issues by indices/kinds, with fields; every row keeps its source index).
  • MCP defaults to a summary; the CLI exports complete Evidence with --detail summary --evidence_output FILE, matching inspect_il2cpp_metadata.
  • Redundant facts that disagree are recorded as offset-backed issues and decoding continues: region SHA-1, repeated path or path hash, entry counts between listings, an empty compression slot, a directory name without its closing /, and (version 11+, both indexes present) a directory index path whose FPakFile::HashPath value the path hash index does not map to the same record. An entry-count mismatch nulls index.entry_count; an empty slot nulls that entry's compression_name; the last two null the affected path, so no path is joined from a malformed directory name. A SHA-1 mismatch or repeated path is an issue only. Compression slot issues come only from records a listing references. Untrustworthy structure (out-of-file offsets or sizes, truncated records, dangling references) is refused with one of 12 named reasons.
  • Path hashes follow UE 5.8 PakFile.cpp and Misc/Fnv.cpp: ASCII lowercase, UTF-16LE code units, FNV-64 with the seed added to the offset basis. Version 10 is not checked: UE uses legacy constants there, repak 0.2.3 writes the version 11 hash, and no UE-written version 10 pak was available.
  • An encrypted index is not read: index.state: "encrypted", and empty entries/issues there mean "not read", not "none".
  • Admission before expansion: the index, path hash index and full directory index share one 256 MiB budget, entries are limited to 1,000,000 and the report to 64 MiB, refused from the declared entry count before entries are built; path text is counted in UTF-16 code units against the same 64 Mi budget before each join. paths-too-large reports resource transport with maximum_output_bytes and maximum_path_utf16_units. Size uses the shared jsonValueByteLength. Long decodes yield every 4,096 records, directories included. Peak memory is not measured.
  • Failure keeps facts that were complete: once the whole file was read and hashed, cancellation, close failure, schema failure or a finished report above the output budget keep partial Evidence with the hashed source identity only. The early output refusal from the declared entry count comes before hashing and keeps no partial Evidence. A cancellation observed after the reader returned keeps the decoded report.
  • Reads go through one owned descriptor (explicit absolute path, symlinks rejected, identity rechecked after hashing); src/artifacts/readStableArtifact.ts gains positioned reads with existing tests unchanged.
  • Scope: read-only inspection of a caller-selected file. No execution, injection, mutation or protection recovery, no new external toolchain, dependency or installer change. A new provider still needs the maintainer scope decision in #3625.

Executed validation

At 7f194e500 (Greptile fixes on merge 1c4870b94 of main 8413ec08f; reader fixes in b7486a740):

  • Regressions written before the fix: 5 of 68 Unreal boundary tests failed on the previous reader (a directory name without / produced Game/ConfigDefaultGame.ini with no issue; a wrong path hash went unnoticed; 50,000 unreferenced records produced 50,000 issues; three regions each under the limit were accepted together; paths-too-large reported file-size without the path limit). After the fix, 68/68.
  • Greptile review of 1c4870b94, regressions first: a version 1 timestamp of real FDateTime ticks (about 6.4e17) failed region-out-of-bounds (58 passed / 1 failed in unrealPakFile.test.ts) and now reads (69/69 Unreal boundary tests). A 9,600,347-byte pak of 600,000 unreferenced records naming an empty slot ran out of a 128 MiB heap and now finishes (0 entries, 0 issues); structure-only walks keep no issues.
  • npm run check (typecheck, lint with 0 errors, format, knip, metadata, architecture guards), docs:check.
  • Focused: tests/boundary/unreal 69, unrealPakCli acceptance 4, stableArtifactConsumers 39.
  • Full suite at 7f194e500: 24,535 passed, 13 failed, 402 skipped; all 13 are outside Unreal (inspector, process, pcap, gdb, two acceptance files).
  • verify:unreal:pak on the UE 5.8.3 Blank58-Mac.pak: CLI/MCP parity, layout 12, 1,722 entries, 0 issues; all 1,722 directory index paths match their path hashes and the 1,722 paths repak list 0.2.3 prints. Replacing the seed by seed + 1 in a local experiment produced a path-hash-mismatch for the paths, so the comparison runs.
  • Not rerun at this head, 0 issues at b7486a740: 13 UE 5.8 paks from the CUE4Parse test fixtures (11 with both indexes, 2 with encrypted indexes) and 24 repak 0.2.3 paks (versions 1 to 11, stored and Zlib).

Contributor-reported at 162a90e8f (not rerun at the head above): 47 self-built paks matched repak 0.2.3 and CUE4Parse entry for entry, except CUE4Parse's single-block size convention; ten UE 5.8.3 UnrealPak variants also read correctly, including path-hash-only paks that both readers list as empty.

Tests: a byte-level fixture builder drives one boundary test per failure reason and issue kind and layouts v1, v3, v4 and v7 to v12, plus CLI/MCP parity. No threshold or exclusion changes.

Coverage limits

Not covered: Windows or console paks, real version 9 frozen indexes (refused), version 10 path hashes (unchecked), and layouts v2, v5 and v6 in the test matrix. Compression names identify methods only. Entry and hash agreement is structural evidence, not proof of a correct cook. Cost (time, memory) on large paks is unmeasured. No game was executed and no file was modified.

Contract and boundary impact

  • Semantic owner and earliest changed stage: src/unreal/UnrealPakFile.ts.
  • Contribution scope fit and linked maintainer decision: static format inspection; new provider, decision pending in #3625.
  • CLI and MCP/tool-catalog contract: one new tool and CLI command; inspect_analysis_view gains kind unreal-pak, an Unreal pak summary and the inspect_unreal_pak parent operation, additively.
  • Provider, bridge, target-format, or platform compatibility: bundled, no external toolchain; pak versions 1 to 12.
  • Evidence, artifact, provenance, or reconstruction contract: observed Evidence with artifact SHA-256 and file-offset ranges of the footer and index regions read.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): see Behavior and boundaries.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: catalog entries regenerate; authored skill references updated; one verify:unreal:pak script; no package or installation change.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Author readiness

Human owner: @8rulerstar

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4362 **Author:** [@8rulerstar](https://github.com/8rulerstar) **Created:** 10/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/unreal-pak` --- ### 📝 Commits (10+) - [`256c3c1`](https://github.com/morluto/rea/commit/256c3c130f4634e45430da154f85a31d23e1ef65) feat(unreal): decode pak footers and indexes for versions 1 to 12 - [`6e03715`](https://github.com/morluto/rea/commit/6e0371583c572b0cd753ccc2362ca156ab678cb9) feat(unreal): read pak indexes from stable file ranges through a provider and service - [`411fed9`](https://github.com/morluto/rea/commit/411fed9144cf25eefaa40cb6b280b8c3df8e9266) feat(unreal): expose inspect_unreal_pak through CLI, MCP and analysis views - [`cb42c6b`](https://github.com/morluto/rea/commit/cb42c6bc279304a13fd7a520f813083875438bb8) test(unreal): cover pak versions, index forms, failures and CLI/MCP delivery - [`486d430`](https://github.com/morluto/rea/commit/486d43040a2066310234426ae3e97046a476fab7) docs(unreal): add the pak index guide, CLI reference, skill notes and a real-file verifier - [`587cbf5`](https://github.com/morluto/rea/commit/587cbf544029e4c84f4c672b7b9f1fbaf9ab0d43) refactor(unreal): drop an unused index type export - [`e1e3864`](https://github.com/morluto/rea/commit/e1e38649d3e6b555f07dce336808b7e5cfc30478) fix(unreal): reject inconsistent counts, leftover listing bytes and unknown compression slots - [`52ab2fb`](https://github.com/morluto/rea/commit/52ab2fbc78353da1ced7c82f9c0d3de843f49428) docs(unreal): document the pruned directory index and the new consistency failures - [`77ceb30`](https://github.com/morluto/rea/commit/77ceb302a4ff992afb0833a20b41d1ceeb1c60a1) refactor(artifacts): share admission and change checks between whole-file and ranged stable reads - [`539e754`](https://github.com/morluto/rea/commit/539e7545f09e5ede0a9e9b0ce697ed39266de890) fix(unreal): check index SHA-1s, entry placement and built path size ### 📊 Changes **43 files changed** (+4727 additions, -49 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+5 -0) 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+10 -0) 📝 `README.md` (+1 -0) 📝 `docs/.vitepress/config.ts` (+1 -0) 📝 `docs/cli.md` (+14 -0) ➕ `docs/unreal-pak.md` (+211 -0) 📝 `package.json` (+1 -0) 📝 `scripts/lib/module-boundaries.mjs` (+1 -0) ➕ `scripts/verify-unreal-pak.mjs` (+145 -0) 📝 `src/application/CapabilityInventory.ts` (+1 -0) ➕ `src/application/unreal/UnrealPakPort.ts` (+20 -0) ➕ `src/application/unreal/UnrealPakService.ts` (+134 -0) 📝 `src/artifacts/readStableArtifact.ts` (+169 -37) 📝 `src/cli.ts` (+2 -0) ➕ `src/cli/unrealCommands.ts` (+55 -0) 📝 `src/cliCommandNames.ts` (+1 -0) 📝 `src/composition/capabilityDeclarations.ts` (+8 -0) 📝 `src/composition/capabilityServices.ts` (+6 -0) ➕ `src/composition/unreal.ts` (+6 -0) 📝 `src/contracts/analysisViewToolContracts.ts` (+1 -1) _...and 23 more files_ </details> ### 📄 Description Thank you for merging main into this branch to refresh CI. I merged the latest main again and fixed five reader boundaries found in a self-review; details are under Executed validation. ## Summary Adds `inspect_unreal_pak` (CLI: `inspect-unreal-pak`): static inspection of one local Unreal `.pak` (versions 1 to 12). It reads the footer and the index regions the footer names and returns each entry's path, offset, sizes, compression method, block facts, encryption and deletion flags and stored SHA-1. File data is never decoded. Refs #3625, narrower: no extraction, decompression, decryption or keys, no IoStore or `.uasset`. ## Behavior and boundaries - `src/unreal/UnrealPakFile.ts` owns format interpretation; `src/unreal/UnrealPakPathHash.ts` owns the path hash rule. The service projects the decoded regions into Evidence locations, and the analysis view only projects retained Evidence (kind `unreal-pak`, collections `entries` by indices/paths and `issues` by indices/kinds, with `fields`; every row keeps its source `index`). - MCP defaults to a summary; the CLI exports complete Evidence with `--detail summary --evidence_output FILE`, matching `inspect_il2cpp_metadata`. - Redundant facts that disagree are recorded as offset-backed `issues` and decoding continues: region SHA-1, repeated path or path hash, entry counts between listings, an empty compression slot, a directory name without its closing `/`, and (version 11+, both indexes present) a directory index path whose `FPakFile::HashPath` value the path hash index does not map to the same record. An entry-count mismatch nulls `index.entry_count`; an empty slot nulls that entry's `compression_name`; the last two null the affected `path`, so no path is joined from a malformed directory name. A SHA-1 mismatch or repeated path is an issue only. Compression slot issues come only from records a listing references. Untrustworthy structure (out-of-file offsets or sizes, truncated records, dangling references) is refused with one of 12 named reasons. - Path hashes follow UE 5.8 `PakFile.cpp` and `Misc/Fnv.cpp`: ASCII lowercase, UTF-16LE code units, FNV-64 with the seed added to the offset basis. Version 10 is not checked: UE uses legacy constants there, repak 0.2.3 writes the version 11 hash, and no UE-written version 10 pak was available. - An encrypted index is not read: `index.state: "encrypted"`, and empty `entries`/`issues` there mean "not read", not "none". - Admission before expansion: the index, path hash index and full directory index share one 256 MiB budget, entries are limited to 1,000,000 and the report to 64 MiB, refused from the declared entry count before entries are built; path text is counted in UTF-16 code units against the same 64 Mi budget before each join. `paths-too-large` reports resource `transport` with `maximum_output_bytes` and `maximum_path_utf16_units`. Size uses the shared `jsonValueByteLength`. Long decodes yield every 4,096 records, directories included. Peak memory is not measured. - Failure keeps facts that were complete: once the whole file was read and hashed, cancellation, close failure, schema failure or a finished report above the output budget keep partial Evidence with the hashed source identity only. The early output refusal from the declared entry count comes before hashing and keeps no partial Evidence. A cancellation observed after the reader returned keeps the decoded report. - Reads go through one owned descriptor (explicit absolute path, symlinks rejected, identity rechecked after hashing); `src/artifacts/readStableArtifact.ts` gains positioned reads with existing tests unchanged. - Scope: read-only inspection of a caller-selected file. No execution, injection, mutation or protection recovery, no new external toolchain, dependency or installer change. A new provider still needs the maintainer scope decision in #3625. ## Executed validation At `7f194e500` (Greptile fixes on merge `1c4870b94` of main `8413ec08f`; reader fixes in `b7486a740`): - Regressions written before the fix: 5 of 68 Unreal boundary tests failed on the previous reader (a directory name without `/` produced `Game/ConfigDefaultGame.ini` with no issue; a wrong path hash went unnoticed; 50,000 unreferenced records produced 50,000 issues; three regions each under the limit were accepted together; `paths-too-large` reported `file-size` without the path limit). After the fix, 68/68. - Greptile review of `1c4870b94`, regressions first: a version 1 timestamp of real `FDateTime` ticks (about 6.4e17) failed `region-out-of-bounds` (58 passed / 1 failed in `unrealPakFile.test.ts`) and now reads (69/69 Unreal boundary tests). A 9,600,347-byte pak of 600,000 unreferenced records naming an empty slot ran out of a 128 MiB heap and now finishes (0 entries, 0 issues); structure-only walks keep no issues. - `npm run check` (typecheck, lint with 0 errors, format, knip, metadata, architecture guards), `docs:check`. - Focused: `tests/boundary/unreal` 69, `unrealPakCli` acceptance 4, `stableArtifactConsumers` 39. - Full suite at `7f194e500`: 24,535 passed, 13 failed, 402 skipped; all 13 are outside Unreal (inspector, process, pcap, gdb, two acceptance files). - `verify:unreal:pak` on the UE 5.8.3 `Blank58-Mac.pak`: CLI/MCP parity, layout 12, 1,722 entries, 0 issues; all 1,722 directory index paths match their path hashes and the 1,722 paths `repak list` 0.2.3 prints. Replacing the seed by seed + 1 in a local experiment produced a `path-hash-mismatch` for the paths, so the comparison runs. - Not rerun at this head, 0 issues at `b7486a740`: 13 UE 5.8 paks from the CUE4Parse test fixtures (11 with both indexes, 2 with encrypted indexes) and 24 repak 0.2.3 paks (versions 1 to 11, stored and Zlib). Contributor-reported at `162a90e8f` (not rerun at the head above): 47 self-built paks matched repak 0.2.3 and CUE4Parse entry for entry, except CUE4Parse's single-block size convention; ten UE 5.8.3 UnrealPak variants also read correctly, including path-hash-only paks that both readers list as empty. Tests: a byte-level fixture builder drives one boundary test per failure reason and issue kind and layouts v1, v3, v4 and v7 to v12, plus CLI/MCP parity. No threshold or exclusion changes. ## Coverage limits Not covered: Windows or console paks, real version 9 frozen indexes (refused), version 10 path hashes (unchecked), and layouts v2, v5 and v6 in the test matrix. Compression names identify methods only. Entry and hash agreement is structural evidence, not proof of a correct cook. Cost (time, memory) on large paks is unmeasured. No game was executed and no file was modified. ## Contract and boundary impact - Semantic owner and earliest changed stage: `src/unreal/UnrealPakFile.ts`. - Contribution scope fit and linked maintainer decision: static format inspection; new provider, decision pending in #3625. - CLI and MCP/tool-catalog contract: one new tool and CLI command; `inspect_analysis_view` gains kind `unreal-pak`, an Unreal pak summary and the `inspect_unreal_pak` parent operation, additively. - Provider, bridge, target-format, or platform compatibility: bundled, no external toolchain; pak versions 1 to 12. - Evidence, artifact, provenance, or reconstruction contract: `observed` Evidence with artifact SHA-256 and file-offset ranges of the footer and index regions read. - Target selection, authority, upstream behavior, and cleanup details (if applicable): see Behavior and boundaries. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: catalog entries regenerate; authored skill references updated; one `verify:unreal:pak` script; no package or installation change. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Author readiness Human owner: @8rulerstar - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#6204
No description provided.