[PR #7085] test(contracts): pin public machine-readable analysis output contracts #7630

Open
opened 2026-11-03 13:39:13 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/7085
Author: @Tapraj-Pathak
Created: 10/25/2026
Status: 🔄 Open

Base: main ← Head: test/public-output-contracts


📝 Commits (2)

  • 94015da test(contracts): pin public machine-readable analysis output contracts
  • 3dd8721 test(contracts): pin Evidence vocabularies against the runtime schema

📊 Changes

5 files changed (+685 additions, -3 deletions)

View changed files

📝 docs/testing.md (+38 -0)
📝 src/domain/evidence.ts (+9 -3)
➕ tests/acceptance/publicOutputContractCli.test.ts (+127 -0)
➕ tests/boundary/mcp/publicOutputContractMcp.test.ts (+329 -0)
➕ tests/fixtures/publicOutputContract.ts (+182 -0)

📄 Description

Summary

Adds the focused contract-test suite requested in #6339: regression tests that
pin the public machine-readable output contracts REA exposes to scripts and
agent integrations, so an unintended change to result fields, documented enum
values, or error representations fails CI instead of integrations.

Fixes #6339

Problem and expected behavior

Internal tests can pass while a public result field, documented enum, or error
representation changes and breaks external consumers. The declared contracts —
the canonical Evidence record, the stable error projection shared by CLI and
MCP, and result-embedded coverage enums — had no dedicated regression
coverage. Expected behavior: representative successful, partial, unsupported,
and failed outputs from more than one analysis category are pinned through
public CLI and MCP workflows, and the suite fails when one of those documented
surfaces drifts.

Change and scope

  • tests/fixtures/publicOutputContract.ts — shared assertions: the documented
    confidence, authority, and error-category vocabularies; canonical
    Evidence record keys, field types, and analysis_profile presence; error
    projections validated against both the runtime zod schema
    (src/contracts/errorSchemas.ts) and the committed generated
    docs/error-contract.schema.json.

  • tests/boundary/mcp/publicOutputContractMcp.test.ts — seven MCP tests over
    production servers (in-memory transport) across three analysis categories
    (byte entropy, native exports, artifact inspection plus its summary view):
    success, documented partial delivery, in-result coverage.status: "unsupported", unsupported_target, invalid_request, advertised
    output-schema validation (ajv), and a drift test asserting the runtime
    error-code set equals the generated document's.

  • tests/acceptance/publicOutputContractCli.test.ts — the same contracts
    through the compiled CLI (--json, exit status 0/1).

  • docs/testing.md — new "Public output contract regressions" section naming
    what the suite protects and how intentional contract changes are reviewed
    and versioned: change the owning schema first, regenerate the generated
    error contract, update the contract fixtures in the same change with an
    explanation, and mark breaking changes with !/BREAKING CHANGE: plus a
    migration note.

  • src/domain/evidence.ts — extracts the inline confidence enum as
    evidenceConfidenceSchema, mirroring evidenceAuthoritySchema, so the contract
    fixture can compare both full runtime vocabularies (review fix; no behavior
    change).

Intentionally not included: new contracts or production behavior changes;
freezing internal implementation details or incidental output fields;
real-provider lanes (the suites use portable fixtures and built-in parsers
only).

Contribution scope: test-only plus documentation within existing boundaries;
no new providers or capabilities involving target execution, injection,
mutation, or protection recovery — no maintainer scope decision required.

Contract and boundary impact

  • Semantic owner and earliest changed stage: no ownership changes; the tests
    consume the existing owners' facts (src/contracts/errorSchemas.ts,
    src/domain/evidence.ts, src/domain/native/nativeExports.ts,
    src/domain/artifactGraph.ts, src/domain/artifactInspection.ts). No
    superseded paths removed.
  • Contribution scope fit and linked maintainer decision: not applicable — test
    coverage and docs within existing boundaries.
  • CLI and MCP/tool-catalog contract: none changed; the suite asserts the
    existing documented contracts, including advertised (post-SDK-conversion)
    output schemas resolved by operation name.
  • Provider, bridge, target-format, or platform compatibility: none changed;
    fixtures are synthetic ELF images and directories built by existing test
    helpers.
  • Evidence, artifact, provenance, or reconstruction contract: none changed;
    the suite asserts Evidence identity/provenance fields and coverage enums.
  • Target selection, authority, upstream behavior, and cleanup details: not
    applicable — no provider or target execution; test-owned resources close via
    onTestFinished.
  • Generated metadata (docs/public/product-catalog.json), package, or
    installation impact: none. docs/error-contract.schema.json is committed
    and unchanged; the tests read it. No catalog or skills/ outputs touched.

Expected effects: the change is tests and documentation; the tests read
locally generated fixture files and exercise REA commands on synthetic local
fixtures, with no analyst-target interaction.

  • Read existing files or captured data. (tests read locally generated fixture files)
  • Documentation or usability only; no target interaction.

Evidence and regression coverage

  • Tests added or updated: 7 MCP boundary tests + 3 CLI acceptance tests + 1
    shared fixture module (executed; results under Validation).
  • Base reproduction or other evidence: not a behavior fix; the gap was the
    missing protection. Drift-gate negative probe (executed): temporarily added
    an undeclared contract_mutation_probe code to
    analysisErrorProjectionSchema; the drift test failed (exit 1); reverted,
    and the suite passed again.
  • User-visible CLI/MCP output (if applicable): unchanged; the tests pin
    existing output.
  • Remaining proof gaps: executed on Windows x64 locally; the Linux CI lanes
    will exercise the same suites. Real-provider behavior is out of scope (no
    provider execution; portable fixtures only). HTTP transport and JSON Lines
    framing are not covered by this suite.

Applicable boundary regressions: this is a new contract suite rather than a
parser or inventory fix; the relevant row is "Public success and error
delivery" — exercised through real MCP paths and the compiled CLI, including
typed error projections and retained-fact assertions. Lifecycle regressions:
not applicable (no resource-acquisition or cleanup changes; test resources
close through existing owned fixtures).

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable. (the suite asserts the confidence/authority enums and the distinctions between complete negative findings, unsupported availability with a reason, version: null, and non-UTF-8 names kept as exact bytes with name_utf8_decodable: false)
  • Artifact identity, source provenance, and failed attempts remain preserved. (asserted: subject digests, provider identity, error codes/categories/remediation)
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. (in-result coverage.status: "unsupported" with a reason; inventory coverage.scope/status; summary source_coverage)

Validation performed

All commands executed locally on Windows x64, pinned Node 24.22.1 / npm 11.19.1:

  • node scripts/run-exact-tests.mjs tests/boundary/mcp/publicOutputContractMcp.test.ts — 7/7 passed (run twice; final run on this exact diff).
  • node scripts/run-development-tests.mjs focused tests/acceptance/publicOutputContractCli.test.ts — 3/3 passed (run twice), after the artifacts:skills build.
  • Drift negative probe — failed as designed (1 test, exit 1), then passed after reverting the probe.
  • npm run typecheck — pass.
  • npm run lint — 0 errors across 5393 files (pre-existing warning baseline unchanged); module-boundaries check pass.
  • npm run docs:check — pass (6 turbo tasks + readmeConsistency 9/9).
  • oxfmt --check on the new files

Review-fix validation (commit 3dd8721b0, pushed to this branch):

  • Vocabulary negative probe — temporarily added an undeclared confidence value to evidenceConfidenceSchema; the new pinned-vocabulary assertion failed with the exact set diff, and passed again after reverting.
  • node scripts/run-exact-tests.mjs tests/boundary/mcp/publicOutputContractMcp.test.ts — 7/7 across three consecutive runs; node scripts/run-development-tests.mjs focused tests/acceptance/publicOutputContractCli.test.ts — 3/3; npm run typecheck — pass; pre-push npm run check:fast — pass.
  • One earlier MCP run on this machine had a transient single-test failure whose details were not captured before three subsequent clean runs; Linux CI is the authoritative gate for this suite. — pass after formatting.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: not required — no provider
    execution; portable fixtures and built-in parsers only. The suites are
    platform-portable; local execution was Windows.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: tests create fixture
    files under test-owned temporary directories and clean up through the
    Vitest lifecycle; no network access and no target processes.

Author readiness

Human owner: @Tapraj-Pathak

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/7085 **Author:** [@Tapraj-Pathak](https://github.com/Tapraj-Pathak) **Created:** 10/25/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `test/public-output-contracts` --- ### 📝 Commits (2) - [`94015da`](https://github.com/morluto/rea/commit/94015da2d65ca8aa312fe7ce1f0e327c5f34c50a) test(contracts): pin public machine-readable analysis output contracts - [`3dd8721`](https://github.com/morluto/rea/commit/3dd8721b00aa92e6467dda71eb68d7af3dcaa422) test(contracts): pin Evidence vocabularies against the runtime schema ### 📊 Changes **5 files changed** (+685 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `docs/testing.md` (+38 -0) 📝 `src/domain/evidence.ts` (+9 -3) ➕ `tests/acceptance/publicOutputContractCli.test.ts` (+127 -0) ➕ `tests/boundary/mcp/publicOutputContractMcp.test.ts` (+329 -0) ➕ `tests/fixtures/publicOutputContract.ts` (+182 -0) </details> ### 📄 Description ## Summary Adds the focused contract-test suite requested in #6339: regression tests that pin the public machine-readable output contracts REA exposes to scripts and agent integrations, so an unintended change to result fields, documented enum values, or error representations fails CI instead of integrations. Fixes #6339 ## Problem and expected behavior Internal tests can pass while a public result field, documented enum, or error representation changes and breaks external consumers. The declared contracts — the canonical Evidence record, the stable error projection shared by CLI and MCP, and result-embedded coverage enums — had no dedicated regression coverage. Expected behavior: representative successful, partial, unsupported, and failed outputs from more than one analysis category are pinned through public CLI and MCP workflows, and the suite fails when one of those documented surfaces drifts. ## Change and scope - `tests/fixtures/publicOutputContract.ts` — shared assertions: the documented `confidence`, `authority`, and error-category vocabularies; canonical Evidence record keys, field types, and `analysis_profile` presence; error projections validated against both the runtime zod schema (`src/contracts/errorSchemas.ts`) and the committed generated `docs/error-contract.schema.json`. - `tests/boundary/mcp/publicOutputContractMcp.test.ts` — seven MCP tests over production servers (in-memory transport) across three analysis categories (byte entropy, native exports, artifact inspection plus its summary view): success, documented partial delivery, in-result `coverage.status: "unsupported"`, `unsupported_target`, `invalid_request`, advertised output-schema validation (ajv), and a drift test asserting the runtime error-code set equals the generated document's. - `tests/acceptance/publicOutputContractCli.test.ts` — the same contracts through the compiled CLI (`--json`, exit status 0/1). - `docs/testing.md` — new "Public output contract regressions" section naming what the suite protects and how intentional contract changes are reviewed and versioned: change the owning schema first, regenerate the generated error contract, update the contract fixtures in the same change with an explanation, and mark breaking changes with `!`/`BREAKING CHANGE:` plus a migration note. - `src/domain/evidence.ts` — extracts the inline confidence enum as `evidenceConfidenceSchema`, mirroring `evidenceAuthoritySchema`, so the contract fixture can compare both full runtime vocabularies (review fix; no behavior change). Intentionally not included: new contracts or production behavior changes; freezing internal implementation details or incidental output fields; real-provider lanes (the suites use portable fixtures and built-in parsers only). Contribution scope: test-only plus documentation within existing boundaries; no new providers or capabilities involving target execution, injection, mutation, or protection recovery — no maintainer scope decision required. ## Contract and boundary impact - Semantic owner and earliest changed stage: no ownership changes; the tests consume the existing owners' facts (`src/contracts/errorSchemas.ts`, `src/domain/evidence.ts`, `src/domain/native/nativeExports.ts`, `src/domain/artifactGraph.ts`, `src/domain/artifactInspection.ts`). No superseded paths removed. - Contribution scope fit and linked maintainer decision: not applicable — test coverage and docs within existing boundaries. - CLI and MCP/tool-catalog contract: none changed; the suite asserts the existing documented contracts, including advertised (post-SDK-conversion) output schemas resolved by operation name. - Provider, bridge, target-format, or platform compatibility: none changed; fixtures are synthetic ELF images and directories built by existing test helpers. - Evidence, artifact, provenance, or reconstruction contract: none changed; the suite asserts Evidence identity/provenance fields and coverage enums. - Target selection, authority, upstream behavior, and cleanup details: not applicable — no provider or target execution; test-owned resources close via `onTestFinished`. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. `docs/error-contract.schema.json` is committed and unchanged; the tests read it. No catalog or `skills/` outputs touched. Expected effects: the change is tests and documentation; the tests read locally generated fixture files and exercise REA commands on synthetic local fixtures, with no analyst-target interaction. - [x] Read existing files or captured data. (tests read locally generated fixture files) - [x] Documentation or usability only; no target interaction. ## Evidence and regression coverage - Tests added or updated: 7 MCP boundary tests + 3 CLI acceptance tests + 1 shared fixture module (executed; results under Validation). - Base reproduction or other evidence: not a behavior fix; the gap was the missing protection. Drift-gate negative probe (executed): temporarily added an undeclared `contract_mutation_probe` code to `analysisErrorProjectionSchema`; the drift test failed (exit 1); reverted, and the suite passed again. - User-visible CLI/MCP output (if applicable): unchanged; the tests pin existing output. - Remaining proof gaps: executed on Windows x64 locally; the Linux CI lanes will exercise the same suites. Real-provider behavior is out of scope (no provider execution; portable fixtures only). HTTP transport and JSON Lines framing are not covered by this suite. Applicable boundary regressions: this is a new contract suite rather than a parser or inventory fix; the relevant row is "Public success and error delivery" — exercised through real MCP paths and the compiled CLI, including typed error projections and retained-fact assertions. Lifecycle regressions: not applicable (no resource-acquisition or cleanup changes; test resources close through existing owned fixtures). For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. (the suite asserts the `confidence`/`authority` enums and the distinctions between complete negative findings, `unsupported` availability with a reason, `version: null`, and non-UTF-8 names kept as exact bytes with `name_utf8_decodable: false`) - [x] Artifact identity, source provenance, and failed attempts remain preserved. (asserted: subject digests, provider identity, error codes/categories/remediation) - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. (in-result `coverage.status: "unsupported"` with a reason; inventory `coverage.scope`/`status`; summary `source_coverage`) ## Validation performed All commands executed locally on Windows x64, pinned Node 24.22.1 / npm 11.19.1: - `node scripts/run-exact-tests.mjs tests/boundary/mcp/publicOutputContractMcp.test.ts` — 7/7 passed (run twice; final run on this exact diff). - `node scripts/run-development-tests.mjs focused tests/acceptance/publicOutputContractCli.test.ts` — 3/3 passed (run twice), after the `artifacts:skills` build. - Drift negative probe — failed as designed (1 test, exit 1), then passed after reverting the probe. - `npm run typecheck` — pass. - `npm run lint` — 0 errors across 5393 files (pre-existing warning baseline unchanged); module-boundaries check pass. - `npm run docs:check` — pass (6 turbo tasks + readmeConsistency 9/9). - `oxfmt --check` on the new files Review-fix validation (commit 3dd8721b0, pushed to this branch): - Vocabulary negative probe — temporarily added an undeclared confidence value to `evidenceConfidenceSchema`; the new pinned-vocabulary assertion failed with the exact set diff, and passed again after reverting. - `node scripts/run-exact-tests.mjs tests/boundary/mcp/publicOutputContractMcp.test.ts` — 7/7 across three consecutive runs; `node scripts/run-development-tests.mjs focused tests/acceptance/publicOutputContractCli.test.ts` — 3/3; `npm run typecheck` — pass; pre-push `npm run check:fast` — pass. - One earlier MCP run on this machine had a transient single-test failure whose details were not captured before three subsequent clean runs; Linux CI is the authoritative gate for this suite. — pass after formatting. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real Hopper/Ghidra, browser, or OS coverage: not required — no provider execution; portable fixtures and built-in parsers only. The suites are platform-portable; local execution was Windows. - Package or release metadata impact: none. - Security, privacy, process, or containment review: tests create fixture files under test-owned temporary directories and clean up through the Vitest lifecycle; no network access and no target processes. ## Author readiness <!-- The human owner completes these declarations before requesting review. The PR author is the owner by default. A different owner must explicitly accept ownership on this PR. AI agents must leave these declarations to a human to dictate to them or there could be catastrophic consequences for many users of REA. Draft PRs may leave them unchecked. These declarations record initial readiness when opening the PR for review; later pushes do not require a new human sign-off. Keep the description and validation evidence current as the work changes. See CONTRIBUTING.md#pull-request-ownership-and-readiness. --> Human owner: @Tapraj-Pathak - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#7630
No description provided.