morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3938] [Bug] Advertised screenshot comparison example fails PNG checksum validation #1546

New issue

Closed

opened 2026-10-16 21:06:59 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:06:59 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3938

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

The production CLI returned exit status 1 with this observed JSON response:

{"error":"Browser observation failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"compare_web_screenshots","issues":[{"path":[],"reason":"invalid_format","message":"Invalid PNG chunk checksum"}]}}

The production stdio MCP call returned isError: true and the same typed error inside its text content. The example read from the running server matched the artifact above.

Independent byte inspection found 70 decoded bytes and the advertised SHA-256. The IDAT chunk stores CRC 3f35af59, but CRC-32 of its type and data is 3dc848ef. Both other chunk checksums match, and the IDAT data inflates successfully. A control changed only that CRC and recomputed the enclosing artifact digest. Both production entry points then returned an identical comparison of one pixel with zero changed pixels. The control's digest was cdb30873bdf16770bfea1fe86e44db7476e504c2dca1542b0660b20f47f523a7.

Reproduction

From the baseline checkout with its built runtime available, assign the artifact copied from the live comparison tool's advertised example:

artifact='{"sha256":"153cf6c9a526a63053a37b10234c2fd85df38887c2dc0a800d90abfa6631d01c","bytes":70,"media_type":"image/png","data_base64":"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PzWvWQAAAABJRU5ErkJggg=="}'

Submit that artifact on both sides of the supported CLI comparison command:

HOME="$(mktemp -d)" node scripts/rea.mjs compare-web-screenshots "$artifact" "$artifact" --json

Observe exit status 1 and the checksum error below. The same artifact was also tested through JSON file arguments with the same result.
For the independently executed MCP path, start node scripts/rea.mjs mcp with an isolated home directory and the default schema profile, read tools/list, and locate compare_web_screenshots. Submit its first inputSchema.examples object unchanged as the arguments of tools/call for that tool. The advertised object has identical before and after artifacts above and channel_threshold: 0.

Expected behavior

An advertised example should satisfy the documented promise that every advertised example is a valid canonical input and should demonstrate the tool's supported PNG comparison behavior. Comparing the example's identical one-pixel screenshots should return successful Evidence with status: identical, compared_pixels: 1, and changed_pixels: 0. A deliberately malformed PNG should still fail validation.

Impact and scope

Callers following the default full catalog's published example receive a request failure instead of comparison Evidence. The failure affects that concrete example; the tested valid control succeeds through the same CLI and MCP entry points. No data loss or general comparison outage was observed.

Workaround: A control with the same compressed pixel data, a valid IDAT CRC, and a matching artifact digest succeeds through both production adapters without changing REA or weakening validation.

REA version

rea 6.5.0 at source commit 4692bb3330; current-main source checked at 22ec8bdb2f

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Metadata-consistent 70-byte, one-pixel PNG artifact from the default full MCP catalog; no browser required.

Interface

MCP

Earliest failure stage

Request parsing or validation

Relevant output or evidence

src/contracts/browserToolContracts.ts:281-309 defines the advertised example; src/browser/PngVisualDiff.ts:126-129 rejects its IDAT CRC. docs/mcp-contracts.md:662-678 requires valid canonical advertised examples; docs/browser-observation.md:35-38 documents local PNG comparison. CLI and production stdio MCP both rejected the exact catalog input; changing only the bad CRC and enclosing digest succeeded through both.

Typed outcome or recovery expected

The unchanged advertised example should return comparison Evidence with status identical, one compared pixel and zero changed pixels. Malformed PNG CRCs must remain rejected.

Analysis or suspected cause

Source-derived: the artifact schema validates base64, digest and size before comparison parses PNG chunks. The example contains an invalid IDAT CRC; the PNG validator correctly rejects it. The compact MCP schema profile omits examples and does not publish this artifact.

Additional context

Scope is the default full catalog example, rather than a general screenshot-comparison failure.

Freshness check: the failing source is unchanged on current upstream main at 22ec8bdb2f. The only change among the cited files is unrelated CLI output-flag normalization. Runtime observations above are from the stated baseline; current-main persistence is source-derived.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3938 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Observed behavior The production CLI returned exit status 1 with this observed JSON response: ```json {"error":"Browser observation failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"compare_web_screenshots","issues":[{"path":[],"reason":"invalid_format","message":"Invalid PNG chunk checksum"}]}} ``` The production stdio MCP call returned `isError: true` and the same typed error inside its text content. The example read from the running server matched the artifact above. Independent byte inspection found 70 decoded bytes and the advertised SHA-256. The IDAT chunk stores CRC `3f35af59`, but CRC-32 of its type and data is `3dc848ef`. Both other chunk checksums match, and the IDAT data inflates successfully. A control changed only that CRC and recomputed the enclosing artifact digest. Both production entry points then returned an identical comparison of one pixel with zero changed pixels. The control's digest was `cdb30873bdf16770bfea1fe86e44db7476e504c2dca1542b0660b20f47f523a7`. ### Reproduction 1. From the baseline checkout with its built runtime available, assign the artifact copied from the live comparison tool's advertised example: ```sh artifact='{"sha256":"153cf6c9a526a63053a37b10234c2fd85df38887c2dc0a800d90abfa6631d01c","bytes":70,"media_type":"image/png","data_base64":"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PzWvWQAAAABJRU5ErkJggg=="}' ``` 2. Submit that artifact on both sides of the supported CLI comparison command: ```sh HOME="$(mktemp -d)" node scripts/rea.mjs compare-web-screenshots "$artifact" "$artifact" --json ``` 3. Observe exit status 1 and the checksum error below. The same artifact was also tested through JSON file arguments with the same result. 4. For the independently executed MCP path, start `node scripts/rea.mjs mcp` with an isolated home directory and the default schema profile, read `tools/list`, and locate `compare_web_screenshots`. Submit its first `inputSchema.examples` object unchanged as the `arguments` of `tools/call` for that tool. The advertised object has identical `before` and `after` artifacts above and `channel_threshold: 0`. ### Expected behavior An advertised example should satisfy the documented promise that every advertised example is a valid canonical input and should demonstrate the tool's supported PNG comparison behavior. Comparing the example's identical one-pixel screenshots should return successful Evidence with `status: identical`, `compared_pixels: 1`, and `changed_pixels: 0`. A deliberately malformed PNG should still fail validation. ### Impact and scope Callers following the default full catalog's published example receive a request failure instead of comparison Evidence. The failure affects that concrete example; the tested valid control succeeds through the same CLI and MCP entry points. No data loss or general comparison outage was observed. Workaround: A control with the same compressed pixel data, a valid IDAT CRC, and a matching artifact digest succeeds through both production adapters without changing REA or weakening validation. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860; current-main source checked at 22ec8bdb2f7d4626587f23b1eed4a207df600386 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Metadata-consistent 70-byte, one-pixel PNG artifact from the default full MCP catalog; no browser required. ### Interface MCP ### Earliest failure stage Request parsing or validation ### Relevant output or evidence `src/contracts/browserToolContracts.ts:281-309` defines the advertised example; `src/browser/PngVisualDiff.ts:126-129` rejects its IDAT CRC. `docs/mcp-contracts.md:662-678` requires valid canonical advertised examples; `docs/browser-observation.md:35-38` documents local PNG comparison. CLI and production stdio MCP both rejected the exact catalog input; changing only the bad CRC and enclosing digest succeeded through both. ### Typed outcome or recovery expected The unchanged advertised example should return comparison Evidence with status `identical`, one compared pixel and zero changed pixels. Malformed PNG CRCs must remain rejected. ### Analysis or suspected cause Source-derived: the artifact schema validates base64, digest and size before comparison parses PNG chunks. The example contains an invalid IDAT CRC; the PNG validator correctly rejects it. The compact MCP schema profile omits examples and does not publish this artifact. ### Additional context Scope is the default full catalog example, rather than a general screenshot-comparison failure. Freshness check: the failing source is unchanged on current upstream main at 22ec8bdb2f7d4626587f23b1eed4a207df600386. The only change among the cited files is unrelated CLI output-flag normalization. Runtime observations above are from the stated baseline; current-main persistence is source-derived.

bsakaguchi commented

2026-10-16 21:07:00 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Confirmed: the advertised example artifact ships an invalid IDAT CRC, so the very first thing a user does with compare_web_screenshots fails checksum validation. The mismatch is trivial to prove (CRC-32 of type+data differs from the stored CRC), so the artifact should be regenerated with correct checksums — and it would be worth adding a CI-style check that re-validates any committed example artifact against the same PNG validator the CLI uses.

@XinwangTao commented on GitHub (Oct 15, 2026): Confirmed: the advertised example artifact ships an invalid IDAT CRC, so the very first thing a user does with compare_web_screenshots fails checksum validation. The mismatch is trivial to prove (CRC-32 of type+data differs from the stored CRC), so the artifact should be regenerated with correct checksums — and it would be worth adding a CI-style check that re-validates any committed example artifact against the same PNG validator the CLI uses.

bsakaguchi referenced this issue

2026-10-17 21:12:18 +00:00

[PR #1546] [MERGED] test(clients): verify native OpenCode integration #2962

bsakaguchi closed this issue

2026-10-31 00:31:22 +00:00

bsakaguchi commented

2026-10-31 00:31:23 +00:00

Author

Owner

@morluto commented on GitHub (Oct 23, 2026):

Fixed in 1275817c7. The named advertised screenshot example now uses a checksum-valid PNG and its matching digest. The actual contract example is parsed and passed to production PNG comparison; it returns one identical pixel. Verified by tests/boundary/browser/browserToolContractExamples.test.ts.

@morluto commented on GitHub (Oct 23, 2026): Fixed in [1275817c7](https://github.com/morluto/rea/commit/1275817c75cf8d76109fcf0621e1a9beaeb1d11c). The named advertised screenshot example now uses a checksum-valid PNG and its matching digest. The actual contract example is parsed and passed to production PNG comparison; it returns one identical pixel. Verified by `tests/boundary/browser/browserToolContractExamples.test.ts`.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1546

No description provided.

Rows
Columns