morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #1891] [MERGED] feat(mcp): derive an API surface view from retained web network captures #4883

New issue

Closed

opened 2026-10-31 00:56:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:56:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1891
Author: @mvanhorn
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/12/2026
Merged by: @morluto

Base: main ← Head: feat/web-capture-api-surface-view

📝 Commits (10+)

be38b6f feat(mcp): derive an API surface view from retained web network captures
7ef0e8d fix(mcp): fail closed on sensitive parents and budget api-surface path expansion
51b71b2 fix(mcp): keep recorded api-surface paths and budget only templated segments
0c7d602 fix(mcp): read api-surface paths only from canonical URLs and charge new values once
0ee5971 fix(mcp): reject authority-less api-surface URLs and key groups by rule positions
b5c3b9a fix(mcp): label api-surface confidence from all groups and cap transient path memory
e848938 fix(mcp): budget api-surface dedup sets and narrow detail outputs
3e0a725 fix(mcp): bound api-surface path text and record limitations
ef7b437 fix(mcp): keep api-surface refusals and path splits inside the reply budget
d926092 fix(mcp): budget api-surface method text before group keys

📊 Changes

34 files changed (+2533 additions, -189 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/evidence-workflows.md (+6 -0)
📝 docs/mcp-contracts.md (+7 -0)
📝 docs/web-network-captures.md (+73 -0)
📝 scripts/verify-web-network-captures.mjs (+24 -0)
📝 src/application/analysisView/AnalysisViewService.test.ts (+147 -0)
📝 src/application/analysisView/AnalysisViewService.ts (+23 -2)
📝 src/browser/history/CaptureRedaction.test.ts (+46 -0)
📝 src/browser/history/CaptureRedaction.ts (+5 -4)
📝 src/browser/history/HistoricalCaptureDecoder.ts (+5 -8)
➕ src/contracts/analysisDetail.test.ts (+61 -0)
📝 src/contracts/analysisDetail.ts (+12 -5)
➕ src/contracts/analysisViewApiSurfaceExample.ts (+73 -0)
📝 src/contracts/analysisViewToolContracts.ts (+6 -1)
📝 src/contracts/javascript/electronToolContracts.ts (+2 -0)
📝 src/contracts/native/binaryDiagnosticsToolContracts.ts (+5 -1)
📝 src/contracts/toolInputReferences.test.ts (+28 -0)
➕ src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts (+173 -0)
➕ src/domain/analysisView/WebNetworkApiSurfaceAccumulator.ts (+467 -0)
📝 src/domain/analysisView/analysisView.ts (+71 -2)
📝 src/domain/analysisView/binaryLayoutView.ts (+1 -1)

...and 14 more files

📄 Description

Summary

Agents can now turn a retained HAR capture into its API surface: inspect_analysis_view with view: {"kind": "api-surface"} groups recorded requests by method, origin and path template, and each group cites the parent record ordinals it came from. It builds on the retained-view workflow from #1180 and adds no new tool.

No issue is linked. This is a new view kind on an existing tool, proposed here with the implementation.

Demo

Full-quality MP4

The sample is a generated 22-request HAR for a made-up shop.example.test, not third-party traffic. Every number in the video is copied from the CLI output on this branch.

Problem and expected behavior

inspect_web_network_capture returns one record per request in producer order. On a single-page app that means many near-identical records (GET /v1/users/1042, /v1/users/2210, ...), and an agent has to read all of them to answer "which endpoints does this app call, with what statuses". Nothing in REA groups recorded requests today.

With this change, inspect_analysis_view reads a retained (or inline) inspect_web_network_capture HAR Evidence and returns groups (method, origin, path_template, templated_segments, record_ordinals, statuses, content_types and coarse flags such as single-sample, mixed-statuses and mixed-content-types), a records summary of examined, total and excluded counts that stays separate from group paging coverage, and excluded_records with a reason for each record whose URL is redacted (null) or unparseable, or whose request or method is missing.

Change and scope

Path segments are templated only when they match one of four exact formats: numeric (ASCII digits), uuid (RFC 9562 text form), hex (32+ hex digits) and iso-date (a valid calendar YYYY-MM-DD). Placeholders are named after the rule ({numeric}), never after a neighbouring segment, and there is no fuzzy or cross-sample matching. So /api/v1 vs /api/v2, PascalCase routes like /api/CreateDocument vs /api/ListDocuments, the invalid date 2026-13-45 and short ids like abc123 stay literal. I kept the rule set this narrow on purpose: anything looser starts inventing structure the capture doesn't show.

A format match is still a pattern match, so each templated segment carries basis: "inferred" with every observed literal, and the view Evidence uses confidence: "inferred" when any segment on the page was templated. Pages with only literal paths stay derived, and the existing view kinds are unchanged. No header, body, query or fragment values are copied into groups, and query strings are not part of the group key.

The api-surface result variant has its own artifact schema with a nullable digest, because a capture writes artifact.sha256: null when the digest matched a sensitive_values declaration. The parent-digest integrity check runs only when both digests exist, so the shared result schema used by the other views is not relaxed.

The view reads the recorded path of canonical scheme://authority/path URLs before WHATWG dot-segment removal, so /a/%2e%2e/admin and /admin stay apart; other spellings (https:host/x, https:///user@host/x) are excluded as url-unparseable rather than guessed. Parents inspected with sensitive_values are refused with a typed unsupported error, because the view writes templates and limitation text it cannot check against declarations the parent does not retain. Each path is bounded before splitting, and only newly retained templated segments and distinct observed values count against the capture's 96 MiB reply budget.

Out of scope: native mitmproxy parents (typed unsupported error), body or schema inference (#882), handler identity or causality claims (#774), and any fetching or replay.

Contract and boundary impact

Semantic owner and earliest changed stage: domain src/domain/analysisView/webNetworkApiSurfaceView.ts, dispatched from projectAnalysisView; AnalysisViewService picks the Evidence confidence.
CLI and MCP/tool-catalog contract: inspect_analysis_view / inspect-analysis-view accept view.kind: "api-surface" with optional offset/limit; parent_operation adds inspect_web_network_capture. Contract example in src/contracts/analysisViewApiSurfaceExample.ts comes from a fixture run.
Provider, bridge, target-format, or platform compatibility: none. HAR parents only.
Evidence, artifact, provenance, or reconstruction contract: derived view Evidence links to the parent capture; confidence is inferred when templating fired; record ordinals, exclusions and parent limitations are preserved.
Process execution, authorization, cleanup, or containment impact: none beyond recording the derived Evidence record.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: authored skill text in .agents/skills/reverse-engineer-anything/references/evidence-workflows.md; no version bump needed after rebasing on main (already 39), and metadata:check passes. Build outputs not committed.

Evidence and regression coverage

Tests added or updated: webNetworkApiSurfaceView.test.ts (sensitive parents refused; HAR status: 0 and empty MIME treated as unknown; per-path and retained-value budgets; slashless and authority-less URLs excluded; encoded dot segments kept; literal {numeric} paths kept apart from templated ones; each rule fires and nothing else does; v1/v2, Report2/Report3 and PascalCase pairs stay separate; deterministic result for the same parent; group paging exhausts; ordinal-subset parent; null/unparseable URLs excluded; mitmproxy parent unsupported); AnalysisViewService.test.ts (inferred vs derived, existing views unchanged, null-digest parent); MCP and CLI boundary tests; scripts/verify-web-network-captures.mjs gains an api-surface step.
Base reproduction or other evidence: source-derived. On bcd350c the view request union has no api-surface kind and parent_operation has no inspect_web_network_capture, so a capture parent cannot be projected.
User-visible CLI/MCP output: the 22-request sample HAR gives 12 groups with records: {examined: 22, total: 22, excluded: 0} and confidence: "inferred"; capture output 41,717 bytes, view output 6,673 bytes. Example group: GET /v1/users/{numeric}, record_ordinals: [1, 2, 3, 4, 5], statuses: [200, 404], flags: ["mixed-statuses"], observed values 1042, 2210, 3377, 4120, 9999.
Remaining proof gaps: verify:web:network-captures needs Linux and mitmdump 12.2.3, so the new step did not run on this macOS host. The api-surface path is covered by the MCP and CLI boundary tests.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

All on macOS with Node 26 and the pinned npm 11.16.0.

npm run test:focused -- tests/boundary/cli/analysisViewCli.test.ts tests/boundary/mcp/analysisViewMcp.test.ts \
  src/domain/analysisView/webNetworkApiSurfaceView.test.ts src/application/analysisView/AnalysisViewService.test.ts \
  src/domain/analysisView/analysisView.test.ts          # pass: 5 files, 60 tests
npm run check:fast                                      # pass
npm run check                                           # pass
npm run docs:check                                      # pass
npm run verify:test-discovery                           # pass
rea inspect-web-network-capture shop.har har --json     # sample HAR, then:
rea inspect-analysis-view view.json --json              # 12 groups, output as above

npm run check:pr had six failures on the first run. Five (process and Hopper timing) passed on rerun. The remaining one, machoSliceReader.test.ts ("checks real /bin/ls slice ranges against its FAT table"), fails the same way on bcd350c without this change on this host.

Compatibility, safety, and release

Breaking changes or migration steps: none. Additive view kind.
Real Hopper/Ghidra, browser, or OS coverage: macOS only; Linux mitmdump lane not run.
Package or release metadata impact: skill reference text only.
Security, privacy, process, or containment review: the view reads caller-supplied or same-session Evidence only, copies no header, body or query values, and makes no network requests.

Review checklist

This PR addresses a concrete problem or an agreed enhancement.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1891 **Author:** [@mvanhorn](https://github.com/mvanhorn) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/12/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `feat/web-capture-api-surface-view` --- ### 📝 Commits (10+) - [`be38b6f`](https://github.com/morluto/rea/commit/be38b6f7c2815062a5b8f90d3406c749f0626595) feat(mcp): derive an API surface view from retained web network captures - [`7ef0e8d`](https://github.com/morluto/rea/commit/7ef0e8dd33b648c483b6f338126064917bcffcd7) fix(mcp): fail closed on sensitive parents and budget api-surface path expansion - [`51b71b2`](https://github.com/morluto/rea/commit/51b71b2d3a92c20976ce1d8d1dc49773dab6aacf) fix(mcp): keep recorded api-surface paths and budget only templated segments - [`0c7d602`](https://github.com/morluto/rea/commit/0c7d6027678b1e28a0ef707d53489bd4efc90465) fix(mcp): read api-surface paths only from canonical URLs and charge new values once - [`0ee5971`](https://github.com/morluto/rea/commit/0ee597123c5991559e1ec50a2d4fe0b8524e970e) fix(mcp): reject authority-less api-surface URLs and key groups by rule positions - [`b5c3b9a`](https://github.com/morluto/rea/commit/b5c3b9a966dd9d8efa8ec2774f99796f80551a63) fix(mcp): label api-surface confidence from all groups and cap transient path memory - [`e848938`](https://github.com/morluto/rea/commit/e848938506719b28bfa36b914d0697c5cefa24ee) fix(mcp): budget api-surface dedup sets and narrow detail outputs - [`3e0a725`](https://github.com/morluto/rea/commit/3e0a725f157e9ab400455dc7efc27a49e8610fbe) fix(mcp): bound api-surface path text and record limitations - [`ef7b437`](https://github.com/morluto/rea/commit/ef7b437e9d90030da19e7ef8bf5853426ef6a877) fix(mcp): keep api-surface refusals and path splits inside the reply budget - [`d926092`](https://github.com/morluto/rea/commit/d92609296959867d0ef717ffde1beed82d4775f9) fix(mcp): budget api-surface method text before group keys ### 📊 Changes **34 files changed** (+2533 additions, -189 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/evidence-workflows.md` (+6 -0) 📝 `docs/mcp-contracts.md` (+7 -0) 📝 `docs/web-network-captures.md` (+73 -0) 📝 `scripts/verify-web-network-captures.mjs` (+24 -0) 📝 `src/application/analysisView/AnalysisViewService.test.ts` (+147 -0) 📝 `src/application/analysisView/AnalysisViewService.ts` (+23 -2) 📝 `src/browser/history/CaptureRedaction.test.ts` (+46 -0) 📝 `src/browser/history/CaptureRedaction.ts` (+5 -4) 📝 `src/browser/history/HistoricalCaptureDecoder.ts` (+5 -8) ➕ `src/contracts/analysisDetail.test.ts` (+61 -0) 📝 `src/contracts/analysisDetail.ts` (+12 -5) ➕ `src/contracts/analysisViewApiSurfaceExample.ts` (+73 -0) 📝 `src/contracts/analysisViewToolContracts.ts` (+6 -1) 📝 `src/contracts/javascript/electronToolContracts.ts` (+2 -0) 📝 `src/contracts/native/binaryDiagnosticsToolContracts.ts` (+5 -1) 📝 `src/contracts/toolInputReferences.test.ts` (+28 -0) ➕ `src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts` (+173 -0) ➕ `src/domain/analysisView/WebNetworkApiSurfaceAccumulator.ts` (+467 -0) 📝 `src/domain/analysisView/analysisView.ts` (+71 -2) 📝 `src/domain/analysisView/binaryLayoutView.ts` (+1 -1) _...and 14 more files_ </details> ### 📄 Description ## Summary Agents can now turn a retained HAR capture into its API surface: `inspect_analysis_view` with `view: {"kind": "api-surface"}` groups recorded requests by method, origin and path template, and each group cites the parent record ordinals it came from. It builds on the retained-view workflow from #1180 and adds no new tool. No issue is linked. This is a new view kind on an existing tool, proposed here with the implementation. ## Demo ![API surface view demo](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/2026-10-11-013-feat-rea-web-capture-api-surface-view/preview.gif) [Full-quality MP4](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/2026-10-11-013-feat-rea-web-capture-api-surface-view/out.mp4) The sample is a generated 22-request HAR for a made-up `shop.example.test`, not third-party traffic. Every number in the video is copied from the CLI output on this branch. ## Problem and expected behavior `inspect_web_network_capture` returns one record per request in producer order. On a single-page app that means many near-identical records (`GET /v1/users/1042`, `/v1/users/2210`, ...), and an agent has to read all of them to answer "which endpoints does this app call, with what statuses". Nothing in REA groups recorded requests today. With this change, `inspect_analysis_view` reads a retained (or inline) `inspect_web_network_capture` HAR Evidence and returns `groups` (method, origin, `path_template`, `templated_segments`, `record_ordinals`, `statuses`, `content_types` and coarse `flags` such as `single-sample`, `mixed-statuses` and `mixed-content-types`), a `records` summary of examined, total and excluded counts that stays separate from group paging `coverage`, and `excluded_records` with a reason for each record whose URL is redacted (`null`) or unparseable, or whose request or method is missing. ## Change and scope Path segments are templated only when they match one of four exact formats: `numeric` (ASCII digits), `uuid` (RFC 9562 text form), `hex` (32+ hex digits) and `iso-date` (a valid calendar `YYYY-MM-DD`). Placeholders are named after the rule (`{numeric}`), never after a neighbouring segment, and there is no fuzzy or cross-sample matching. So `/api/v1` vs `/api/v2`, PascalCase routes like `/api/CreateDocument` vs `/api/ListDocuments`, the invalid date `2026-13-45` and short ids like `abc123` stay literal. I kept the rule set this narrow on purpose: anything looser starts inventing structure the capture doesn't show. A format match is still a pattern match, so each templated segment carries `basis: "inferred"` with every observed literal, and the view Evidence uses `confidence: "inferred"` when any segment on the page was templated. Pages with only literal paths stay `derived`, and the existing view kinds are unchanged. No header, body, query or fragment values are copied into groups, and query strings are not part of the group key. The `api-surface` result variant has its own artifact schema with a nullable digest, because a capture writes `artifact.sha256: null` when the digest matched a `sensitive_values` declaration. The parent-digest integrity check runs only when both digests exist, so the shared result schema used by the other views is not relaxed. The view reads the recorded path of canonical `scheme://authority/path` URLs before WHATWG dot-segment removal, so `/a/%2e%2e/admin` and `/admin` stay apart; other spellings (`https:host/x`, `https:///user@host/x`) are excluded as `url-unparseable` rather than guessed. Parents inspected with `sensitive_values` are refused with a typed unsupported error, because the view writes templates and limitation text it cannot check against declarations the parent does not retain. Each path is bounded before splitting, and only newly retained templated segments and distinct observed values count against the capture's 96 MiB reply budget. Out of scope: native mitmproxy parents (typed unsupported error), body or schema inference (#882), handler identity or causality claims (#774), and any fetching or replay. ## Contract and boundary impact - Semantic owner and earliest changed stage: domain `src/domain/analysisView/webNetworkApiSurfaceView.ts`, dispatched from `projectAnalysisView`; `AnalysisViewService` picks the Evidence confidence. - CLI and MCP/tool-catalog contract: `inspect_analysis_view` / `inspect-analysis-view` accept `view.kind: "api-surface"` with optional `offset`/`limit`; `parent_operation` adds `inspect_web_network_capture`. Contract example in `src/contracts/analysisViewApiSurfaceExample.ts` comes from a fixture run. - Provider, bridge, target-format, or platform compatibility: none. HAR parents only. - Evidence, artifact, provenance, or reconstruction contract: derived view Evidence links to the parent capture; `confidence` is `inferred` when templating fired; record ordinals, exclusions and parent limitations are preserved. - Process execution, authorization, cleanup, or containment impact: none beyond recording the derived Evidence record. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: authored skill text in `.agents/skills/reverse-engineer-anything/references/evidence-workflows.md`; no version bump needed after rebasing on main (already 39), and `metadata:check` passes. Build outputs not committed. ## Evidence and regression coverage - Tests added or updated: `webNetworkApiSurfaceView.test.ts` (sensitive parents refused; HAR `status: 0` and empty MIME treated as unknown; per-path and retained-value budgets; slashless and authority-less URLs excluded; encoded dot segments kept; literal `{numeric}` paths kept apart from templated ones; each rule fires and nothing else does; v1/v2, Report2/Report3 and PascalCase pairs stay separate; deterministic result for the same parent; group paging exhausts; ordinal-subset parent; null/unparseable URLs excluded; mitmproxy parent unsupported); `AnalysisViewService.test.ts` (`inferred` vs `derived`, existing views unchanged, null-digest parent); MCP and CLI boundary tests; `scripts/verify-web-network-captures.mjs` gains an api-surface step. - Base reproduction or other evidence: source-derived. On `bcd350c` the view request union has no `api-surface` kind and `parent_operation` has no `inspect_web_network_capture`, so a capture parent cannot be projected. - User-visible CLI/MCP output: the 22-request sample HAR gives 12 groups with `records: {examined: 22, total: 22, excluded: 0}` and `confidence: "inferred"`; capture output 41,717 bytes, view output 6,673 bytes. Example group: `GET /v1/users/{numeric}`, `record_ordinals: [1, 2, 3, 4, 5]`, `statuses: [200, 404]`, `flags: ["mixed-statuses"]`, observed values `1042, 2210, 3377, 4120, 9999`. - Remaining proof gaps: `verify:web:network-captures` needs Linux and mitmdump 12.2.3, so the new step did not run on this macOS host. The api-surface path is covered by the MCP and CLI boundary tests. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed All on macOS with Node 26 and the pinned npm 11.16.0. ``` npm run test:focused -- tests/boundary/cli/analysisViewCli.test.ts tests/boundary/mcp/analysisViewMcp.test.ts \ src/domain/analysisView/webNetworkApiSurfaceView.test.ts src/application/analysisView/AnalysisViewService.test.ts \ src/domain/analysisView/analysisView.test.ts # pass: 5 files, 60 tests npm run check:fast # pass npm run check # pass npm run docs:check # pass npm run verify:test-discovery # pass rea inspect-web-network-capture shop.har har --json # sample HAR, then: rea inspect-analysis-view view.json --json # 12 groups, output as above ``` `npm run check:pr` had six failures on the first run. Five (process and Hopper timing) passed on rerun. The remaining one, `machoSliceReader.test.ts` ("checks real /bin/ls slice ranges against its FAT table"), fails the same way on `bcd350c` without this change on this host. ## Compatibility, safety, and release - Breaking changes or migration steps: none. Additive view kind. - Real Hopper/Ghidra, browser, or OS coverage: macOS only; Linux mitmdump lane not run. - Package or release metadata impact: skill reference text only. - Security, privacy, process, or containment review: the view reads caller-supplied or same-session Evidence only, copies no header, body or query values, and makes no network requests. ## Review checklist - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:56:23 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4883

No description provided.

Rows
Columns