[GH-ISSUE #2173] fix(artifacts): identical complete plist inventories compare as unknown #718

New issue

Closed

opened 2026-10-15 18:27:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:27:40 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2173

Summary

Comparing two complete inventories of identical valid plist files returns unknown with zero unchanged occurrences and one unknown occurrence. Both inventories contain the same fully observed root, and the only inventory notice says the artifact has no child container entries. The comparison introduces an unresolved artifact-coverage finding and an open high-severity residual unknown even though this inventory has no missing members. This was reproduced through the production CLI and stdio MCP interfaces on the source build and published package.

Affected area

Evidence, artifacts, or reconstruction. Inventory capture succeeds; the first incorrect result appears during artifact comparison derivation. The demonstrated target is a standalone XML plist file on macOS, and no native analysis engine is needed. The report covers the inventory comparison and its downstream artifact-coverage finding and residual unknown as one failure.

Runtime or environment

The source build was tested at revision c22111eca2. The separately installed published rea-agents package was version 6.4.0; the source is 259 commits after that release tag despite carrying the same package version metadata. Both ran on darwin arm64 with Node.js 24.21.0, which meets documented installed-runtime support. This is not a claim that the contributor's pinned Node.js runtime was used. The source runtime was already built, and the release was already installed before verification.

Steps to reproduce

Use either tested runtime as rea. Run these commands in a disposable directory; Python, native analysis engines, credentials, and external targets are unnecessary. jq only extracts the returned nested inventory and does not modify it.

work_root=$(mktemp -d)
printf '%s' '<?xml version="1.0"?><plist version="1.0"><dict><key>label</key><string>hello</string></dict></plist>' > "$work_root/left.plist"
cp "$work_root/left.plist" "$work_root/right.plist"
chmod 0644 "$work_root/left.plist" "$work_root/right.plist"
rea inspect-artifact "$work_root/left.plist" --json > "$work_root/left-inspection.json"
jq '.normalized_result.substeps[] | select(.evidence.operation == "inventory_artifact") | .evidence' "$work_root/left-inspection.json" > "$work_root/left-inventory.json"
printf '%s\n' "$work_root/right.plist"

Connect an MCP client to rea mcp. Send these public tool calls in order on the same connection:

Call open_binary with path set to the absolute right-plist path printed above.
Call inspect_artifact with {}. Retain the full Evidence record nested in the returned normalized_result.substeps entry whose evidence.operation is inventory_artifact.
Call compare_artifacts with left set to the complete JSON record in left-inventory.json and right set to the nested Evidence from step 2. Do not pass only their normalized results or edit their Evidence IDs.
Retain the full comparison Evidence, then call find_changed_behavior with comparisons set to an array containing that record.
Call list_unknowns with {"domain":"artifact-comparison"}.

This CLI-left/MCP-right path was executed on both tested runtimes. Both inventories have root_format: "plist", one node, one occurrence at ., zero edges, and returned array lengths equal to their manifest counts. Their root hashes, graph hashes, and manifest IDs match. The nested inventory's only limitation is Artifact has no child container entries. Close the MCP connection and remove the disposable directory after inspection.

Expected behavior

The public comparison contract compares complete artifact inventories by occurrence path, content identity, metadata, and graph relations; gaps yield truncated or unknown. These equal, fully returned plist inventories have no missing occurrence or failed child-container read. Their notice describes a root with no child entries in the inventory's modeled scope. The comparison should return status: "unchanged", summary.unchanged: 1, summary.unknown: 0, and no changes.

The unchanged comparison should introduce no artifact-coverage unresolved finding or artifact-comparison residual unknown. find_changed_behavior may still report an overall unknown behavior status because no process comparison was supplied; it must not attribute that lack of runtime evidence to a nonexistent artifact-inventory gap.

Actual behavior

Both tested runtimes return the following comparison fields:

{
  "status": "unknown",
  "summary": {
    "unchanged": 0,
    "added": 0,
    "removed": 0,
    "changed": 0,
    "unknown": 1,
    "contradiction": 0
  },
  "changes": [],
  "limitations": [
    "Left: Artifact has no child container entries.",
    "Right: Artifact has no child container entries."
  ]
}

The actual public find_changed_behavior result contains one artifact:coverage finding with classification unresolved_branch, status unknown, and summary.unresolved: 1. The actual public list_unknowns result contains an open artifact-comparison unknown with severity high and a recommendation to inspect both artifacts and compare their complete graphs, which the reproduction has already done.

A real content change from hello to world still produces a root change with dimension content, summary.changed: 1, and summary.unknown: 0, but the overall comparison status remains unknown. Changing only the right file's mode from 0644 to 0755 produces the same overall status with a root executable change. These controls show that the bug adds false uncertainty while preserving detected deltas.

Evidence

Expected source: src/contracts/sessionToolContracts.ts:107 defines the complete-inventory comparison contract; src/artifacts/inventory/reader.ts:14 and :39 establish the plist root's lack of a child reader and the informational notice. tests/boundary/filesystem/artifactComparison.test.ts:173 separately asserts unchanged accounting for an identical complete directory inventory.
Failure source: Executed production CLI-left/MCP-right comparison, followed by public find_changed_behavior and list_unknowns, on both tested runtimes. src/domain/artifactComparison.ts:107, :136, :420, and :436 trace the informational notice through the coverage gate, overall status, and summary; src/domain/changedBehavior.ts:310 and src/server/registerArtifactComparisonTool.ts:86 trace the downstream finding and residual unknown.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The source path accepts .plist during target resolution in src/application/BinaryTargetResolver.ts:259 and :324. The production tool accepted the exact returned Evidence, including its authenticated ID, subject/root facts, unique members, complete graph commitment, and manifest commitment. The public downstream call also passed the comparison Evidence closure check. The release result comes from the packaged dispatcher and runtime, rather than an assumption based on source or package metadata.

Discriminating controls were executed on both runtimes. Identical one-entry ZIPs return unchanged, with two unchanged occurrences and zero unknowns, despite their ZIP scope notice. For semantic isolation, a correctly authenticated derived inventory made from the observed plist inventory with only its informational notice removed returns unchanged, with one unchanged occurrence and zero unknowns. Derived inventory controls with omitted nodes or an explicit omitted-child coverage limitation return truncated and unknown, respectively. These derived controls are not claims of real producer omissions or recommended user workarounds. No broad test suite was run.

Restoration check

Replay the public plist fixture through both inventory producers and the comparison tool. The equal complete pair must report unchanged, one unchanged occurrence, zero unknowns, and no changes. Its downstream aggregation must contain no artifact-coverage unresolved branch, and comparison must create no artifact-comparison residual unknown for the equal pair.

Preserve the real content and executable-bit changes as root deltas with the relevant dimensions and citations; fully covered changed inventories should report a changed comparison rather than become equivalent. Preserve the identical ZIP control and preserve truncated for authenticated partial inventories and unknown for genuine coverage gaps. These checks restore the same inventory contract without requiring any specific representation or implementation.

Additional context

Public reach was verified for the plist fixture only. An ordinary short text file was rejected by production target admission, so module-level acceptance of arbitrary files does not establish their public reachability. No claim is made that every raw-binary, native-executable, archive, or other root format follows the same producer and admission path.

Root-digest comparison is not a full workaround. The real executable-bit control preserves manifest.root_sha256 while changing required metadata. manifest.manifest_id does distinguish that control because it commits to the graph, but checking equality manually does not provide the tool's per-path change results or remove its false downstream finding and residual unknown. A full-result workaround was not established. Open and closed issue/PR records and live searches were checked; no matching report was found. Related ZIP coverage and partial-inventory summary reports concern different failing contracts.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The demonstrated complete plist comparison reports false uncertainty and incorrect unchanged accounting, then creates consequential unresolved work in its consumers. Complete inventories and true content/executable deltas remain available, so the observed impact is bounded result accuracy rather than data loss or total workflow failure.
Workaround status: unverified
Workaround basis: Root hashes miss the demonstrated executable metadata change. Manifest-ID equality preserves more of the equality check but does not restore the comparison results or downstream state. No practical alternative that preserves the full affected result was verified.

Originally created by @coygeek on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2173 ## Summary Comparing two complete inventories of identical valid plist files returns `unknown` with zero unchanged occurrences and one unknown occurrence. Both inventories contain the same fully observed root, and the only inventory notice says the artifact has no child container entries. The comparison introduces an unresolved artifact-coverage finding and an open high-severity residual unknown even though this inventory has no missing members. This was reproduced through the production CLI and stdio MCP interfaces on the source build and published package. ## Affected area Evidence, artifacts, or reconstruction. Inventory capture succeeds; the first incorrect result appears during artifact comparison derivation. The demonstrated target is a standalone XML plist file on macOS, and no native analysis engine is needed. The report covers the inventory comparison and its downstream artifact-coverage finding and residual unknown as one failure. ## Runtime or environment The source build was tested at revision c22111eca2c4bc84a084f12711f405b3e7422a40. The separately installed published `rea-agents` package was version 6.4.0; the source is 259 commits after that release tag despite carrying the same package version metadata. Both ran on darwin arm64 with Node.js 24.21.0, which meets documented installed-runtime support. This is not a claim that the contributor's pinned Node.js runtime was used. The source runtime was already built, and the release was already installed before verification. ## Steps to reproduce Use either tested runtime as `rea`. Run these commands in a disposable directory; Python, native analysis engines, credentials, and external targets are unnecessary. `jq` only extracts the returned nested inventory and does not modify it. ```sh work_root=$(mktemp -d) printf '%s' '<?xml version="1.0"?><plist version="1.0"><dict><key>label</key><string>hello</string></dict></plist>' > "$work_root/left.plist" cp "$work_root/left.plist" "$work_root/right.plist" chmod 0644 "$work_root/left.plist" "$work_root/right.plist" rea inspect-artifact "$work_root/left.plist" --json > "$work_root/left-inspection.json" jq '.normalized_result.substeps[] | select(.evidence.operation == "inventory_artifact") | .evidence' "$work_root/left-inspection.json" > "$work_root/left-inventory.json" printf '%s\n' "$work_root/right.plist" ``` Connect an MCP client to `rea mcp`. Send these public tool calls in order on the same connection: 1. Call `open_binary` with `path` set to the absolute right-plist path printed above. 2. Call `inspect_artifact` with `{}`. Retain the full Evidence record nested in the returned `normalized_result.substeps` entry whose `evidence.operation` is `inventory_artifact`. 3. Call `compare_artifacts` with `left` set to the complete JSON record in `left-inventory.json` and `right` set to the nested Evidence from step 2. Do not pass only their normalized results or edit their Evidence IDs. 4. Retain the full comparison Evidence, then call `find_changed_behavior` with `comparisons` set to an array containing that record. 5. Call `list_unknowns` with `{"domain":"artifact-comparison"}`. This CLI-left/MCP-right path was executed on both tested runtimes. Both inventories have `root_format: "plist"`, one node, one occurrence at `.`, zero edges, and returned array lengths equal to their manifest counts. Their root hashes, graph hashes, and manifest IDs match. The nested inventory's only limitation is `Artifact has no child container entries.` Close the MCP connection and remove the disposable directory after inspection. ## Expected behavior The public comparison contract compares complete artifact inventories by occurrence path, content identity, metadata, and graph relations; gaps yield `truncated` or `unknown`. These equal, fully returned plist inventories have no missing occurrence or failed child-container read. Their notice describes a root with no child entries in the inventory's modeled scope. The comparison should return `status: "unchanged"`, `summary.unchanged: 1`, `summary.unknown: 0`, and no changes. The unchanged comparison should introduce no artifact-coverage unresolved finding or artifact-comparison residual unknown. `find_changed_behavior` may still report an overall unknown behavior status because no process comparison was supplied; it must not attribute that lack of runtime evidence to a nonexistent artifact-inventory gap. ## Actual behavior Both tested runtimes return the following comparison fields: ```json { "status": "unknown", "summary": { "unchanged": 0, "added": 0, "removed": 0, "changed": 0, "unknown": 1, "contradiction": 0 }, "changes": [], "limitations": [ "Left: Artifact has no child container entries.", "Right: Artifact has no child container entries." ] } ``` The actual public `find_changed_behavior` result contains one `artifact:coverage` finding with classification `unresolved_branch`, status `unknown`, and `summary.unresolved: 1`. The actual public `list_unknowns` result contains an open artifact-comparison unknown with severity `high` and a recommendation to inspect both artifacts and compare their complete graphs, which the reproduction has already done. A real content change from `hello` to `world` still produces a root change with dimension `content`, `summary.changed: 1`, and `summary.unknown: 0`, but the overall comparison status remains `unknown`. Changing only the right file's mode from 0644 to 0755 produces the same overall status with a root `executable` change. These controls show that the bug adds false uncertainty while preserving detected deltas. ## Evidence - Expected source: `src/contracts/sessionToolContracts.ts:107` defines the complete-inventory comparison contract; `src/artifacts/inventory/reader.ts:14` and `:39` establish the plist root's lack of a child reader and the informational notice. `tests/boundary/filesystem/artifactComparison.test.ts:173` separately asserts unchanged accounting for an identical complete directory inventory. - Failure source: Executed production CLI-left/MCP-right comparison, followed by public `find_changed_behavior` and `list_unknowns`, on both tested runtimes. `src/domain/artifactComparison.ts:107`, `:136`, `:420`, and `:436` trace the informational notice through the coverage gate, overall status, and summary; `src/domain/changedBehavior.ts:310` and `src/server/registerArtifactComparisonTool.ts:86` trace the downstream finding and residual unknown. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The source path accepts `.plist` during target resolution in `src/application/BinaryTargetResolver.ts:259` and `:324`. The production tool accepted the exact returned Evidence, including its authenticated ID, subject/root facts, unique members, complete graph commitment, and manifest commitment. The public downstream call also passed the comparison Evidence closure check. The release result comes from the packaged dispatcher and runtime, rather than an assumption based on source or package metadata. Discriminating controls were executed on both runtimes. Identical one-entry ZIPs return `unchanged`, with two unchanged occurrences and zero unknowns, despite their ZIP scope notice. For semantic isolation, a correctly authenticated derived inventory made from the observed plist inventory with only its informational notice removed returns `unchanged`, with one unchanged occurrence and zero unknowns. Derived inventory controls with omitted nodes or an explicit omitted-child coverage limitation return `truncated` and `unknown`, respectively. These derived controls are not claims of real producer omissions or recommended user workarounds. No broad test suite was run. ## Restoration check Replay the public plist fixture through both inventory producers and the comparison tool. The equal complete pair must report `unchanged`, one unchanged occurrence, zero unknowns, and no changes. Its downstream aggregation must contain no artifact-coverage unresolved branch, and comparison must create no artifact-comparison residual unknown for the equal pair. Preserve the real content and executable-bit changes as root deltas with the relevant dimensions and citations; fully covered changed inventories should report a changed comparison rather than become equivalent. Preserve the identical ZIP control and preserve `truncated` for authenticated partial inventories and `unknown` for genuine coverage gaps. These checks restore the same inventory contract without requiring any specific representation or implementation. ## Additional context Public reach was verified for the plist fixture only. An ordinary short text file was rejected by production target admission, so module-level acceptance of arbitrary files does not establish their public reachability. No claim is made that every raw-binary, native-executable, archive, or other root format follows the same producer and admission path. Root-digest comparison is not a full workaround. The real executable-bit control preserves `manifest.root_sha256` while changing required metadata. `manifest.manifest_id` does distinguish that control because it commits to the graph, but checking equality manually does not provide the tool's per-path change results or remove its false downstream finding and residual unknown. A full-result workaround was not established. Open and closed issue/PR records and live searches were checked; no matching report was found. Related ZIP coverage and partial-inventory summary reports concern different failing contracts. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The demonstrated complete plist comparison reports false uncertainty and incorrect unchanged accounting, then creates consequential unresolved work in its consumers. Complete inventories and true content/executable deltas remain available, so the observed impact is bounded result accuracy rather than data loss or total workflow failure. - Workaround status: unverified - Workaround basis: Root hashes miss the demonstrated executable metadata change. Manifest-ID equality preserves more of the equality check but does not restore the comparison results or downstream state. No practical alternative that preserves the full affected result was verified.

bsakaguchi

2026-10-15 18:27:40 +00:00