[GH-ISSUE #6339] [Testing] Add contract tests for public machine-readable analysis outputs #3823

New issue

Open

opened 2026-10-31 00:44:24 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:24 +00:00

Owner

Originally created by @jayykernel on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6339

Problem

Changes to result fields, enum values, or error representations can unintentionally break scripts and agent integrations even when internal tests pass. A focused contract suite would catch unintended changes to the interfaces users consume.

Desired outcome

Establish regression tests for selected, documented machine-readable output contracts exposed by REA.

Proposed behavior

Identify the public CLI/MCP result contracts that are intended to be stable.
Create small representative fixtures for successful, partial, unsupported, and failed outcomes.
Validate required fields, field types, and documented enum values.
Test that unavailable or incomplete evidence remains distinguishable from a negative finding.
Document how intentional contract changes should be reviewed and versioned.

Acceptance criteria

Contract tests cover representative outputs from more than one supported analysis category.
An unintended breaking change to a declared contract fails CI.
Tests do not require commercial external tools unless explicitly marked as integration tests.
Fixture updates include an explanation when the public contract changes.

Scope

This should protect explicitly supported public contracts, not freeze every internal implementation detail or every incidental output field.

Originally created by @jayykernel on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6339 ### Problem Changes to result fields, enum values, or error representations can unintentionally break scripts and agent integrations even when internal tests pass. A focused contract suite would catch unintended changes to the interfaces users consume. ### Desired outcome Establish regression tests for selected, documented machine-readable output contracts exposed by REA. ### Proposed behavior - Identify the public CLI/MCP result contracts that are intended to be stable. - Create small representative fixtures for successful, partial, unsupported, and failed outcomes. - Validate required fields, field types, and documented enum values. - Test that unavailable or incomplete evidence remains distinguishable from a negative finding. - Document how intentional contract changes should be reviewed and versioned. ### Acceptance criteria - Contract tests cover representative outputs from more than one supported analysis category. - An unintended breaking change to a declared contract fails CI. - Tests do not require commercial external tools unless explicitly marked as integration tests. - Fixture updates include an explanation when the public contract changes. ### Scope This should protect explicitly supported public contracts, not freeze every internal implementation detail or every incidental output field.

bsakaguchi commented

2026-10-31 00:44:25 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 25, 2026):

Contract tests for machine-readable outputs are a great idea — this is exactly the kind of regression that silently breaks downstream scripts and agent integrations. Two suggestions: (1) make sure the fixtures explicitly cover the "partial evidence" case, since distinguishing unavailable from negative findings is the trickiest part for consumers; (2) consider versioning the documented contracts (even a simple v1/v2 marker) so intentional changes stay reviewable without churning the whole suite.

@XinwangTao commented on GitHub (Oct 25, 2026): Contract tests for machine-readable outputs are a great idea — this is exactly the kind of regression that silently breaks downstream scripts and agent integrations. Two suggestions: (1) make sure the fixtures explicitly cover the "partial evidence" case, since distinguishing unavailable from negative findings is the trickiest part for consumers; (2) consider versioning the documented contracts (even a simple v1/v2 marker) so intentional changes stay reviewable without churning the whole suite.

bsakaguchi referenced this issue

2026-10-31 01:10:29 +00:00

[PR #3823] [CLOSED] fix(electron): retain parsed script frame attribution #5925

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3823

No description provided.

Rows
Columns