[PR #936] [CLOSED] feat(artifacts): inventory flat installer packages #2513

New issue

Closed

opened 2026-10-17 21:07:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:07:01 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/936
Author: @kaoru0822-kitauji
Created: 10/7/2026
Status: ❌ Closed

Base: main ← Head: feat/pkg-inventory

📝 Commits (10+)

54dd1a7 feat(artifacts): inventory flat installer packages
daf504b fix(artifacts): harden xar and cpio readers for hostile packages
667bf16 Merge remote-tracking branch 'origin/main' into feat/pkg-inventory
7857d2d fix(artifacts): apply the integrity policy to xar checksums and bound symlinks
bb58eee Merge origin/main into feat/pkg-inventory
ce40fe6 Merge remote-tracking branch 'origin/main' into feat/pkg-inventory
f3e41f1 Merge remote-tracking branch 'origin/main' into feat/pkg-inventory
79e686f chore: merge main into feat/pkg-inventory
5389b23 chore: merge main into feat/pkg-inventory
31a642f chore: merge main into feat/pkg-inventory

📊 Changes

32 files changed (+4326 additions, -161 deletions)

View changed files

📝 .github/workflows/ci.yml (+3 -2)
📝 docs/error-contract.schema.json (+23 -1)
📝 docs/mcp-contracts.md (+6 -0)
📝 docs/testing.md (+8 -0)
📝 package-lock.json (+1 -0)
📝 package.json (+4 -1)
➕ scripts/verify/apple/installer-package.mjs (+174 -0)
📝 skill-src/reverse-engineer-anything/references/native-and-artifacts.md (+6 -1)
➕ src/artifacts/ArtifactDecodedBudget.ts (+78 -0)
📝 src/artifacts/ArtifactHash.ts (+12 -1)
📝 src/artifacts/ArtifactProviderMetadata.ts (+1 -1)
📝 src/artifacts/ArtifactReader.ts (+29 -7)
➕ src/artifacts/ArtifactStreamPipeline.test.ts (+52 -0)
➕ src/artifacts/ArtifactStreamPipeline.ts (+60 -0)
➕ src/artifacts/ArtifactTraversal.ts (+94 -0)
➕ src/artifacts/CpioArtifactReader.ts (+796 -0)
➕ src/artifacts/InstallerPackage.fixture.ts (+167 -0)
➕ src/artifacts/InstallerPackageLimits.test.ts (+604 -0)
➕ src/artifacts/InstallerPackageReaders.test.ts (+413 -0)
➕ src/artifacts/XarArtifactReader.ts (+875 -0)

...and 12 more files

📄 Description

Summary

inspect_artifact on a .pkg recorded only the root hash, with the limitation "PKG root hash is observed; child inventory requires a native macOS adapter". Flat installer packages are xar archives, and an analyst reviewing an installer mostly needs what is inside: Distribution, PackageInfo, the preinstall/postinstall scripts, and the payload that will land on disk.

The inventory now covers those members, with exact content digests. The reader is pure TypeScript, so it also works on Linux.

Change and scope

XarArtifactReader (src/artifacts/XarArtifactReader.ts):
- Validates the header and reads the zlib TOC (bounded at 64 MiB), then verifies the TOC checksum stored in the heap.
- Walks nested <file> elements, so a product package lists Component.pkg/….
- Decodes raw (application/octet-stream) and zlib (application/x-gzip) members, verifying each member's extracted SHA-1/MD5 checksum as its bytes stream.
- Rejects unsafe member names. Unsupported encodings such as bzip2 fail with format.
CpioArtifactReader (src/artifacts/CpioArtifactReader.ts):
- Streams odc and newc gzip-cpio archives, reading members once, in archive order, and buffering at most one chunk, so large payloads stay bounded.
- Reports symlink targets.
- Skips device nodes, FIFOs and sockets.
- Rejects traversal and absolute names (path) and truncation (format).
Scan integration:
- A member the xar reader flags as nestedArchive: "gzip-cpio", i.e. Scripts or Payload starting with gzip magic, is expanded as a nested container, generalizing the existing ASAR expansion. Children are parented to the container occurrence.
- pbzx (xz) payloads are recorded as files with a limitation pointing to pkgutil --expand-full. Node has no xz decoder, so no dependency is added for it.
Dependency: @xmldom/xmldom 0.9.12, already installed through plist, becomes a direct pinned dependency for TOC parsing. The lockfile change is one line.

Not included: extract_artifact for PKG, Bom parsing, pbzx decompression, and projecting PKG payload apps through project_apple_application_graph.

Contract and boundary impact

CLI and MCP: no schema change. PKG inventories now contain child occurrences, and the inspect_artifact description mentions PKG.
Integrity: xar declares SHA-1/MD5 rather than SHA-256, so declaredSha256 stays null. A mismatch fails with integrity when the member is read, whatever the integrity policy. The commit message and docs say so.
Effects: none new. The scan is read-only and in-process.
Generated metadata: catalog (description), skill (version 28) and conformance digests are regenerated.

Real inventory of a productbuild package, trimmed:

directory Component.pkg
file      Component.pkg/Bom
file      Component.pkg/PackageInfo
file      Component.pkg/Payload
directory Component.pkg/Payload/usr/local/rea
file      Component.pkg/Payload/usr/local/rea/hello.txt
file      Component.pkg/Scripts
file      Component.pkg/Scripts/postinstall
file      Component.pkg/Scripts/preinstall
file      Distribution

Evidence and regression coverage

src/artifacts/InstallerPackageReaders.test.ts:
- nested xar members, and zlib and raw decoding;
- gzip-cpio flagging, the pbzx limitation, symlinks, and nested cpio expansion;
- a tampered member checksum, a corrupt TOC checksum, bzip2, and non-xar input;
- odc and newc archives with in-order reads and skipping of unopened data, and special files;
- traversal, absolute names and truncation.
tests/boundary/filesystem/installerPackageInventory.test.ts runs a full inventory_artifact scan of a synthetic product package:
- payload digests, executable scripts, and container parent links;
- strict failure on a member checksum mismatch.
Real lane npm run verify:installer-package, added to the macOS CI job. It needs only pkgbuild, productbuild and pkgutil.
- It builds a component package with scripts, a product package around it, and a --compression latest (pbzx) package.
- The expanded Payload members must equal Apple's pkgutil --payload-files listing.
- Payload and script digests must equal the sources, and scripts must be executable.
- The product package must nest its component.
- The pbzx payload must be recorded and not expanded.
- CLI and stdio MCP results must agree.

Remaining proof gaps

No real Apple-distributed pkg was available on this machine. Pbzx handling is checked with a real pkgbuild pbzx package.

Validation performed

npm run check:changed: 293 files passed.
npm run test:focused -- tests/boundary/filesystem/installerPackageInventory.test.ts tests/boundary/filesystem/asarInventoryContainment.test.ts tests/boundary/filesystem/artifactInventorySnapshot.test.ts tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/cli/productCatalog.test.ts tests/acceptance/contractPresentation.test.ts tests/boundary/providers/auxiliaryDeclarations.test.ts: 48 passed.
npm run docs:generate && npm run docs:check, npx oxfmt --check ., npm run lint: no new errors.
npm run verify:installer-package: {"ok":true,"mocked":false,"cli":true,"stdio_mcp":true,"payload_members":8,"matches_pkgutil_bom":true,"product_nested":true,"pbzx_payload":"recorded-not-expanded"}.

Compatibility, safety, and release

Breaking changes: none. PKG inventories gain children.
Real Hopper/Ghidra coverage: not applicable.
Real OS coverage: macOS 26.6.2 arm64 locally, and the macOS 14 CI job. The unit and boundary tests run on every host.
Security: hostile archives are bounded. The TOC size is capped, names are limited to 64 KiB, members stream, traversal is rejected, and checksums are verified.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/936 **Author:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Created:** 10/7/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/pkg-inventory` --- ### 📝 Commits (10+) - [`54dd1a7`](https://github.com/morluto/rea/commit/54dd1a7604a1f629b341c3586efa8823088d82e4) feat(artifacts): inventory flat installer packages - [`daf504b`](https://github.com/morluto/rea/commit/daf504b62f1257de5c96c5318ef889e665ab237b) fix(artifacts): harden xar and cpio readers for hostile packages - [`667bf16`](https://github.com/morluto/rea/commit/667bf164f9040c8f56cf795bbed23875ab4fe35e) Merge remote-tracking branch 'origin/main' into feat/pkg-inventory - [`7857d2d`](https://github.com/morluto/rea/commit/7857d2d32960424512f238236d340de2c503bb99) fix(artifacts): apply the integrity policy to xar checksums and bound symlinks - [`bb58eee`](https://github.com/morluto/rea/commit/bb58eeef27b15032268f1baa5ba5c5e7aecf25da) Merge origin/main into feat/pkg-inventory - [`ce40fe6`](https://github.com/morluto/rea/commit/ce40fe6ded8cd5694f764d3836ed2ff31a0c563e) Merge remote-tracking branch 'origin/main' into feat/pkg-inventory - [`f3e41f1`](https://github.com/morluto/rea/commit/f3e41f1e0c3056a90b00359459d8dbf24e24b979) Merge remote-tracking branch 'origin/main' into feat/pkg-inventory - [`79e686f`](https://github.com/morluto/rea/commit/79e686f6a2288bb5e3f6d049140debf96d99c834) chore: merge main into feat/pkg-inventory - [`5389b23`](https://github.com/morluto/rea/commit/5389b233b8d758f07d74d16948fa45f9f4282057) chore: merge main into feat/pkg-inventory - [`31a642f`](https://github.com/morluto/rea/commit/31a642f73de3aa1dc6aeef90769d5d7c200dcb3b) chore: merge main into feat/pkg-inventory ### 📊 Changes **32 files changed** (+4326 additions, -161 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+3 -2) 📝 `docs/error-contract.schema.json` (+23 -1) 📝 `docs/mcp-contracts.md` (+6 -0) 📝 `docs/testing.md` (+8 -0) 📝 `package-lock.json` (+1 -0) 📝 `package.json` (+4 -1) ➕ `scripts/verify/apple/installer-package.mjs` (+174 -0) 📝 `skill-src/reverse-engineer-anything/references/native-and-artifacts.md` (+6 -1) ➕ `src/artifacts/ArtifactDecodedBudget.ts` (+78 -0) 📝 `src/artifacts/ArtifactHash.ts` (+12 -1) 📝 `src/artifacts/ArtifactProviderMetadata.ts` (+1 -1) 📝 `src/artifacts/ArtifactReader.ts` (+29 -7) ➕ `src/artifacts/ArtifactStreamPipeline.test.ts` (+52 -0) ➕ `src/artifacts/ArtifactStreamPipeline.ts` (+60 -0) ➕ `src/artifacts/ArtifactTraversal.ts` (+94 -0) ➕ `src/artifacts/CpioArtifactReader.ts` (+796 -0) ➕ `src/artifacts/InstallerPackage.fixture.ts` (+167 -0) ➕ `src/artifacts/InstallerPackageLimits.test.ts` (+604 -0) ➕ `src/artifacts/InstallerPackageReaders.test.ts` (+413 -0) ➕ `src/artifacts/XarArtifactReader.ts` (+875 -0) _...and 12 more files_ </details> ### 📄 Description ## Summary `inspect_artifact` on a `.pkg` recorded only the root hash, with the limitation "PKG root hash is observed; child inventory requires a native macOS adapter". Flat installer packages are xar archives, and an analyst reviewing an installer mostly needs what is inside: Distribution, PackageInfo, the `preinstall`/`postinstall` scripts, and the payload that will land on disk. The inventory now covers those members, with exact content digests. The reader is pure TypeScript, so it also works on Linux. ## Change and scope - **`XarArtifactReader`** (`src/artifacts/XarArtifactReader.ts`): - Validates the header and reads the zlib TOC (bounded at 64 MiB), then verifies the TOC checksum stored in the heap. - Walks nested `<file>` elements, so a product package lists `Component.pkg/…`. - Decodes raw (`application/octet-stream`) and zlib (`application/x-gzip`) members, verifying each member's extracted SHA-1/MD5 checksum as its bytes stream. - Rejects unsafe member names. Unsupported encodings such as bzip2 fail with `format`. - **`CpioArtifactReader`** (`src/artifacts/CpioArtifactReader.ts`): - Streams odc and newc gzip-cpio archives, reading members once, in archive order, and buffering at most one chunk, so large payloads stay bounded. - Reports symlink targets. - Skips device nodes, FIFOs and sockets. - Rejects traversal and absolute names (`path`) and truncation (`format`). - **Scan integration:** - A member the xar reader flags as `nestedArchive: "gzip-cpio"`, i.e. `Scripts` or `Payload` starting with gzip magic, is expanded as a nested container, generalizing the existing ASAR expansion. Children are parented to the container occurrence. - pbzx (xz) payloads are recorded as files with a limitation pointing to `pkgutil --expand-full`. Node has no xz decoder, so no dependency is added for it. - **Dependency:** `@xmldom/xmldom` 0.9.12, already installed through `plist`, becomes a direct pinned dependency for TOC parsing. The lockfile change is one line. Not included: `extract_artifact` for PKG, Bom parsing, pbzx decompression, and projecting PKG payload apps through `project_apple_application_graph`. ## Contract and boundary impact - **CLI and MCP:** no schema change. PKG inventories now contain child occurrences, and the `inspect_artifact` description mentions PKG. - **Integrity:** xar declares SHA-1/MD5 rather than SHA-256, so `declaredSha256` stays `null`. A mismatch fails with `integrity` when the member is read, whatever the integrity policy. The commit message and docs say so. - **Effects:** none new. The scan is read-only and in-process. - **Generated metadata:** catalog (description), skill (version 28) and conformance digests are regenerated. Real inventory of a `productbuild` package, trimmed: ``` directory Component.pkg file Component.pkg/Bom file Component.pkg/PackageInfo file Component.pkg/Payload directory Component.pkg/Payload/usr/local/rea file Component.pkg/Payload/usr/local/rea/hello.txt file Component.pkg/Scripts file Component.pkg/Scripts/postinstall file Component.pkg/Scripts/preinstall file Distribution ``` ## Evidence and regression coverage - `src/artifacts/InstallerPackageReaders.test.ts`: - nested xar members, and zlib and raw decoding; - gzip-cpio flagging, the pbzx limitation, symlinks, and nested cpio expansion; - a tampered member checksum, a corrupt TOC checksum, bzip2, and non-xar input; - odc and newc archives with in-order reads and skipping of unopened data, and special files; - traversal, absolute names and truncation. - `tests/boundary/filesystem/installerPackageInventory.test.ts` runs a full `inventory_artifact` scan of a synthetic product package: - payload digests, executable scripts, and container parent links; - strict failure on a member checksum mismatch. - **Real lane `npm run verify:installer-package`**, added to the macOS CI job. It needs only `pkgbuild`, `productbuild` and `pkgutil`. - It builds a component package with scripts, a product package around it, and a `--compression latest` (pbzx) package. - The expanded Payload members must equal Apple's `pkgutil --payload-files` listing. - Payload and script digests must equal the sources, and scripts must be executable. - The product package must nest its component. - The pbzx payload must be recorded and not expanded. - CLI and stdio MCP results must agree. **Remaining proof gaps** - No real Apple-distributed pkg was available on this machine. Pbzx handling is checked with a real `pkgbuild` pbzx package. ## Validation performed - `npm run check:changed`: 293 files passed. - `npm run test:focused -- tests/boundary/filesystem/installerPackageInventory.test.ts tests/boundary/filesystem/asarInventoryContainment.test.ts tests/boundary/filesystem/artifactInventorySnapshot.test.ts tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/cli/productCatalog.test.ts tests/acceptance/contractPresentation.test.ts tests/boundary/providers/auxiliaryDeclarations.test.ts`: 48 passed. - `npm run docs:generate && npm run docs:check`, `npx oxfmt --check .`, `npm run lint`: no new errors. - `npm run verify:installer-package`: `{"ok":true,"mocked":false,"cli":true,"stdio_mcp":true,"payload_members":8,"matches_pkgutil_bom":true,"product_nested":true,"pbzx_payload":"recorded-not-expanded"}`. ## Compatibility, safety, and release - Breaking changes: none. PKG inventories gain children. - Real Hopper/Ghidra coverage: not applicable. - Real OS coverage: macOS 26.6.2 arm64 locally, and the macOS 14 CI job. The unit and boundary tests run on every host. - Security: hostile archives are bounded. The TOC size is capped, names are limited to 64 KiB, members stream, traversal is rejected, and checksums are verified. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:07:01 +00:00