[PR #2988] feat(browser): add screenshot bounds and region comparisons #5542

New issue

Open

opened 2026-10-31 01:07:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:07:26 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2988
Author: @kissshhot
Created: 10/13/2026
Status: 🔄 Open

Base: main ← Head: codex/screenshot-region-comparisons

📝 Commits (5)

e2a39c4 feat(browser): add screenshot bounds and region comparisons
22f99ca fix(browser): validate spatial consistency and scope verifier CI
3be9c41 fix(browser): bound fixture text rendering across platforms
fa14454 fix(browser): enforce screenshot verifier cleanup and CI ownership
b220107 fix(ci): route all screenshot comparison owners to browser checks

📊 Changes

19 files changed (+1574 additions, -37 deletions)

View changed files

📝 .github/workflows/real-browser.yml (+3 -0)
📝 docs/browser-observation.md (+53 -0)
📝 package.json (+1 -0)
📝 scripts/ci/scopes.mjs (+8 -0)
➕ scripts/fixtures/visual-coding-page.html (+87 -0)
➕ scripts/lib/browser-screenshot-verifier-cleanup.d.mts (+8 -0)
➕ scripts/lib/browser-screenshot-verifier-cleanup.mjs (+49 -0)
➕ scripts/verify-screenshot-regions.mjs (+236 -0)
📝 src/application/BrowserEvidence.ts (+1 -0)
📝 src/browser/PngVisualDiff.ts (+16 -34)
➕ src/browser/ScreenshotRegionDiff.test.ts (+364 -0)
➕ src/browser/ScreenshotRegionDiff.ts (+186 -0)
📝 src/cli/browserAdvancedCommands.ts (+10 -0)
📝 src/contracts/browserToolContracts.ts (+2 -1)
➕ src/domain/screenshotRegion.ts (+91 -0)
📝 src/domain/webScreenshot.ts (+21 -0)
📝 tests/boundary/filesystem/ciChangeScope.test.ts (+53 -2)
➕ tests/boundary/mcp/screenshotSpatial.test.ts (+244 -0)
➕ tests/process-global/screenshotRegionVerifierCleanup.test.ts (+141 -0)

📄 Description

Add changed_bounds and optional regions to the existing CLI/MCP screenshot comparison. Callers can locate visual differences and verify a repaired component while retaining whole-image metrics. The implementation shares decoded PNGs, keeps the algorithm and tests separate, and preserves region validation, resource limits and Evidence provenance.

Evaluation

Configured model: DeepSeek V4 Flash.

We authored four HTML/CSS/JavaScript templates—dashboard, storefront, settings and article—and created three CSS defect variants per template. Each task gives the agent a runnable faulty draft and target desktop, mobile and open-dialog screenshots. The agent edits source, recaptures screenshots and verifies existing interactions; the reference implementation and injected edits are hidden.

Example: storefront-02. The draft changes the primary button from blue to brown, hides a badge and reduces the card gap from 18 px to 8 px. The agent must restore the target appearance across all three states while preserving the membership dialog, keyboard controls and form submission.

The complete planned matrix ran: 12 cases × 2 versions × 3 repeats = 72/72 runs, with matched model, prompt and budgets. All runs are included.

Visual metric	Upstream	This change
Mean error removed	75.8%	87.1%
Mean residual changed-pixel ratio	1.549%	0.739%

Mean error removed increased 11.4 percentage points; mean residual error fell 52.3%. Error removed is computed per run from the three-state mean pixel error relative to the initial draft, then averaged across runs. These post-hoc results show lower visual error without improved full-task success or token efficiency. Region-level metrics support targeted follow-up inspection.

All 72 runs preserved the 15 checked interactions. Independent NumPy calculations matched 376 whole-image comparisons and 8 requested region metrics; target hashes and recorded usage were verified. Evaluation infrastructure and artifacts remain outside the patch.

Validation

179 focused tests passed on the updated PR branch (macOS): pixel metrics, region/whole-image consistency, CLI/MCP parity, Evidence retention, CI scope and cleanup failure paths.
15 real Chromium cases passed on macOS across desktop/mobile and device scale 2, with verified empty process lineage. Injected browser-close failure and live-descendant controls both exit nonzero without a completed report. The close-failure control exited successfully before the fix.
Earlier Linux validation passed 258 focused tests and 15 real Chromium cases; an injected change outside the heading failed the unchanged bounds assertion. The cleanup revision awaits Linux CI.
Typecheck, formatting, Knip, metadata, architecture/module boundaries and test discovery passed.
After rebasing, check and docs:check hit existing upstream failures: a function-length lint error in ManagedCilDecoding.test.ts and Turkish README link drift. The affected files are unchanged by this PR.

AI-assisted implementation and verification with Codex.

Author readiness

Human owner: @kissshhot

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2988 **Author:** [@kissshhot](https://github.com/kissshhot) **Created:** 10/13/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `codex/screenshot-region-comparisons` --- ### 📝 Commits (5) - [`e2a39c4`](https://github.com/morluto/rea/commit/e2a39c47973571427c0a5441f4bb40b7561e1273) feat(browser): add screenshot bounds and region comparisons - [`22f99ca`](https://github.com/morluto/rea/commit/22f99ca13467fc67aad0b2bdc975b6db48e30537) fix(browser): validate spatial consistency and scope verifier CI - [`3be9c41`](https://github.com/morluto/rea/commit/3be9c4173c7e010c792c29e237991301ad7fdf1a) fix(browser): bound fixture text rendering across platforms - [`fa14454`](https://github.com/morluto/rea/commit/fa144543816f3e74e535dc2169a097641b5d8653) fix(browser): enforce screenshot verifier cleanup and CI ownership - [`b220107`](https://github.com/morluto/rea/commit/b22010750f2952cfce47cf13183f2a1afa0ef544) fix(ci): route all screenshot comparison owners to browser checks ### 📊 Changes **19 files changed** (+1574 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/real-browser.yml` (+3 -0) 📝 `docs/browser-observation.md` (+53 -0) 📝 `package.json` (+1 -0) 📝 `scripts/ci/scopes.mjs` (+8 -0) ➕ `scripts/fixtures/visual-coding-page.html` (+87 -0) ➕ `scripts/lib/browser-screenshot-verifier-cleanup.d.mts` (+8 -0) ➕ `scripts/lib/browser-screenshot-verifier-cleanup.mjs` (+49 -0) ➕ `scripts/verify-screenshot-regions.mjs` (+236 -0) 📝 `src/application/BrowserEvidence.ts` (+1 -0) 📝 `src/browser/PngVisualDiff.ts` (+16 -34) ➕ `src/browser/ScreenshotRegionDiff.test.ts` (+364 -0) ➕ `src/browser/ScreenshotRegionDiff.ts` (+186 -0) 📝 `src/cli/browserAdvancedCommands.ts` (+10 -0) 📝 `src/contracts/browserToolContracts.ts` (+2 -1) ➕ `src/domain/screenshotRegion.ts` (+91 -0) 📝 `src/domain/webScreenshot.ts` (+21 -0) 📝 `tests/boundary/filesystem/ciChangeScope.test.ts` (+53 -2) ➕ `tests/boundary/mcp/screenshotSpatial.test.ts` (+244 -0) ➕ `tests/process-global/screenshotRegionVerifierCleanup.test.ts` (+141 -0) </details> ### 📄 Description Add `changed_bounds` and optional `regions` to the existing CLI/MCP screenshot comparison. Callers can locate visual differences and verify a repaired component while retaining whole-image metrics. The implementation shares decoded PNGs, keeps the algorithm and tests separate, and preserves region validation, resource limits and Evidence provenance. ## Evaluation Configured model: **DeepSeek V4 Flash**. We authored four HTML/CSS/JavaScript templates—dashboard, storefront, settings and article—and created three CSS defect variants per template. Each task gives the agent a runnable faulty draft and target desktop, mobile and open-dialog screenshots. The agent edits source, recaptures screenshots and verifies existing interactions; the reference implementation and injected edits are hidden. **Example: storefront-02.** The draft changes the primary button from blue to brown, hides a badge and reduces the card gap from 18 px to 8 px. The agent must restore the target appearance across all three states while preserving the membership dialog, keyboard controls and form submission. The complete planned matrix ran: **12 cases × 2 versions × 3 repeats = 72/72 runs**, with matched model, prompt and budgets. All runs are included. | Visual metric | Upstream | This change | | --- | ---: | ---: | | Mean error removed | 75.8% | **87.1%** | | Mean residual changed-pixel ratio | 1.549% | **0.739%** | Mean error removed increased **11.4 percentage points**; mean residual error fell **52.3%**. Error removed is computed per run from the three-state mean pixel error relative to the initial draft, then averaged across runs. These post-hoc results show lower visual error without improved full-task success or token efficiency. Region-level metrics support targeted follow-up inspection. All 72 runs preserved the 15 checked interactions. Independent NumPy calculations matched **376 whole-image comparisons and 8 requested region metrics**; target hashes and recorded usage were verified. Evaluation infrastructure and artifacts remain outside the patch. ## Validation - 179 focused tests passed on the updated PR branch (macOS): pixel metrics, region/whole-image consistency, CLI/MCP parity, Evidence retention, CI scope and cleanup failure paths. - 15 real Chromium cases passed on macOS across desktop/mobile and device scale 2, with verified empty process lineage. Injected browser-close failure and live-descendant controls both exit nonzero without a completed report. The close-failure control exited successfully before the fix. - Earlier Linux validation passed 258 focused tests and 15 real Chromium cases; an injected change outside the heading failed the unchanged bounds assertion. The cleanup revision awaits Linux CI. - Typecheck, formatting, Knip, metadata, architecture/module boundaries and test discovery passed. - After rebasing, `check` and `docs:check` hit existing upstream failures: a function-length lint error in `ManagedCilDecoding.test.ts` and Turkish README link drift. The affected files are unchanged by this PR. AI-assisted implementation and verification with Codex. ## Author readiness Human owner: @kissshhot - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:07:26 +00:00

bsakaguchi referenced this issue

2026-11-01 21:44:13 +00:00

[PR #5501] [MERGED] fix(native): admit universal Mach-O files without a host slice #6942

bsakaguchi referenced this issue

2026-11-01 21:44:42 +00:00

[PR #5650] [MERGED] fix(apple): read LC_LAZY_LOAD_DYLIB_INFO dependency payloads #7013