morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3622] [CLOSED] feat(analysis-view): report value-free request header facets in API surfaces #5858

New issue

Closed

opened 2026-10-31 01:09:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:09:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3622
Author: @Y0UW1SH
Created: 10/14/2026
Status: ❌ Closed

Base: main ← Head: feat/3592-api-surface-header-facets

📝 Commits (9)

600fdb2 feat(analysis-view): report value-free request header facets in API surfaces
8a62a70 test(mcp): inherit the suite timeout for the target-free session workflow
bacaee4 chore(analysis-view): sync header facets with main
d973785 test(verify): retain host tools for Ghidra prerequisite rejection
7e36ba2 chore(ci): sync main and format plist regression
45b8c3b test(plist): align archive regression with canonical NaN labels
e7a3952 chore(ci): integrate upstream kernel descriptor cleanup fix
df5b396 fix(clients): avoid quadratic trailing newline scans in TOML
b87da3d chore: reconcile upstream fixes and restore PR mergeability

📊 Changes

19 files changed (+1879 additions, -36 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/evidence-workflows.md (+5 -0)
📝 docs/mcp-contracts.md (+2 -2)
📝 docs/web-network-captures.md (+57 -1)
📝 scripts/ci/scopes.mjs (+6 -0)
📝 scripts/verify-web-network-captures.mjs (+42 -0)
📝 src/application/ClientConfigurationDocument.capacity.test.ts (+37 -0)
📝 src/application/analysisView/AnalysisViewService.test.ts (+67 -0)
📝 src/application/analysisView/AnalysisViewService.ts (+5 -6)
📝 src/contracts/analysisViewToolContracts.ts (+1 -1)
📝 src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts (+228 -12)
📝 src/domain/analysisView/WebNetworkApiSurfaceAccumulator.ts (+94 -5)
📝 src/domain/analysisView/webNetworkApiSurfaceView.test.ts (+428 -0)
📝 src/domain/analysisView/webNetworkApiSurfaceView.ts (+32 -4)
➕ src/domain/analysisView/webNetworkRequestHeaderFacets.ts (+524 -0)
➕ src/domain/asciiLowercase.ts (+3 -0)
📝 src/domain/javascript/htmlArtifactReferences.ts (+1 -3)
📝 tests/boundary/cli/analysisViewCli.test.ts (+148 -1)
📝 tests/boundary/mcp/analysisViewMcp.test.ts (+138 -1)
📝 tests/fixtures/historicalHar.ts (+61 -0)

📄 Description

Summary

This PR adds an opt-in facet to the api-surface view over retained HAR Evidence. For each endpoint group it reports the recorded request header names, cookie names and set-cookie names, with value-free signals: how many records carry a header, how many distinct values it had, their length range and character alphabet, and fixed-rule inferences labelled inferred. An agent can see which endpoints carry per-request signature, nonce or timestamp headers and trace the producing code, without header or cookie values entering its context.

This is Part A of #3592 only. Part B (keeping the auth scheme token from redacted Authorization values) is deferred pending the maintainer decision on #3592; CaptureRedaction and the HAR producer are unchanged, so authorization is reported only as present and redacted.

It follows the merged opt-in facet include_query_parameter_names (#2982, #2913) and builds on #1891. Open #3425 changes the same view and accumulator; see "Overlap with #3425" below. Related: #2914, #2884, #2915.

Refs #3592

Problem and expected behavior

Analyst question: "Which endpoints in this capture are authenticated or signed, under which header and cookie names? Which headers change on every request, like a signature, nonce or timestamp, so I know what code to look for?"

Before this change (source-derived): api-surface copies no headers ("Query strings, fragments, credentials, headers and bodies are not copied into groups", docs/web-network-captures.md). The workaround is paging capture views with detail: "complete" and comparing header values by eye, which copies values into context.

After this change, view: {kind: "api-surface", request_header_facets: {}} (or {scope: "all"}) returns per-group header and cookie name facets. No value, prefix, substring or digest is emitted. Omitting the option leaves the output byte-identical.

Change and scope

Request: one optional object, request_header_facets?: {scope?: "script-settable" | "all"}. Its presence selects the facets and scope defaults to script-settable, so the advertised JSON Schema expresses every accepted request structurally (no refinement the schema cannot carry). The echoed view records the effective scope. Omitting it leaves groups unchanged.
script-settable (default) counts but does not list WHATWG Fetch forbidden request-header names (21 names plus Proxy-/Sec- prefixes) and HTTP/2 pseudo-headers (:-prefixed); excluded_names gives the distinct count. The value-dependent method-override rule is not applied. all lists every name.
Each group gains request_header_facets:
- request_headers[] sorted by code point: name (ASCII-lowercased), present_records, redacted_records, distinct_values (null when any present value was redacted or not text), value_length {min,max} (UTF-16, null when none retained), alphabets (first matching rule per value: decimal, hex, base64url, base64, other), variation (unknown, single-sample, constant, per-record-unique, varying) and inferences[]. Repeated field lines in one record form one ordered per-record value.
- cookie_names[] from request.cookies[].name and set_cookie_names[] from response.cookies[].name, each {name, record_ordinals} with exact spelling, listed in both scopes.
- unavailable_header_name_ordinals, unavailable_cookie_name_ordinals, unavailable_set_cookie_name_ordinals: records whose names are unknown, including an empty cookies array next to a recorded cookie / set-cookie header.
Redaction is read from the producer's redactions sidecar (/request/headers/N/value), not inferred from names.
inferences (fixed vocabulary, basis: "inferred", never derived from names), both requiring at least 3 present records: per-request-token-candidate (all distinct, length ≥16, no other, not all decimal) and epoch-timestamp-candidate (all decimal, one length of 10 or 13, more than one distinct value).
A top-level header_inference_groups, present only with the selection, counts groups with any header inference across the full grouping, like templated_groups. Confidence is inferred on every page when templated_groups > 0 || header_inference_groups > 0; the application reads these domain counts instead of walking page groups.
Values are hashed in memory (SHA-256 over length-prefixed UTF-16 code units), one header name at a time so one hash state is live. Borrowed values are fed to the hash in fixed 1,024-code-unit slices, so the temporary encoding is bounded instead of growing with a value's length (a 16 MiB value previously produced a full-length temporary buffer). Per-record scratch (name copies, borrowed value lists, maps, one hash state and one slice) is reserved from line counts before reading; new rows, digests, excluded names, cookie names and ordinals are admitted against the existing 96 MiB projection budget before a group changes. Digests are never serialized. The sensitive_values_unretained refusal is unchanged.
Files: new src/domain/analysisView/webNetworkRequestHeaderFacets.ts (scope rule, alphabets, per-record reading, per-group collector, output schema); new src/domain/asciiLowercase.ts, shared with src/domain/javascript/htmlArtifactReferences.ts (which drops its private copy); WebNetworkApiSurfaceAccumulator.ts (options-object constructor, scratch/admission, forView, header_inference_groups); webNetworkApiSurfaceView.ts (request fields, group and count fields, limitation); AnalysisViewService.ts (forView, confidence); tool description in analysisViewToolContracts.ts; docs/web-network-captures.md, docs/mcp-contracts.md, the skill's evidence-workflows.md.

Differences from the pre-implementation draft: one optional request_header_facets: {scope?} object instead of a bare enum; added redacted_records, header_inference_groups and the three unavailable_*_ordinals lists so unknown stays distinct from absent; the epoch rule also needs 3 records; inferences is always present; no second advertised contract example (the #2982 precedent added none, and examples are forwarded in every input schema); Part B not implemented.

Overlap with #3425. Both PRs add a WebNetworkApiSurfaceAccumulator.forView(view) with the same signature, which conflicts in that hunk and in the constructor (resolution: keep both selections). To compose with #3425 and the sibling endpoint-comparison PR (#3591 work, which adds an owner key), the constructor now takes one options object, {maximumBytes?, includeQueryParameterNames?, requestHeaderScope?}, exposed as readonly properties; whichever lands second adds one key instead of reordering positional parameters. The AnalysisViewService and view-default forView call sites are identical to #3425's. The other hunks (schema fields, group schema, group state, admission, page projection, docs, skill text, tests appended at file ends) were placed to merge without conflict.

CI follow-up (separate commit 8a62a7065). tests/boundary/mcp/sessionMcp.test.ts > "target-free MCP workflow" kept a test-specific 10 s limit. On the macOS runner it took 11.6 s and failed the Real Apple artifact lane in the CI re-run of 600fdb2b0; that lane is not touched by this change. The test now inherits the shared suite deadline (30 s, 60 s under coverage), with assertions and cleanup unchanged. The other three 10 s limits in that file took 0.05 to 2 s on the same runner and are left as they are.

Non-goals: no values, prefixes or hashes emitted; no secrecy or role guesses from names; no live CDP headers; no SAZ or mitmproxy until #2884/#2915; no replay, signature verification or key recovery; no claim that a name absent from the HAR was absent on the wire.

Follow the contribution scope.
Unresolved scope, authority, side effects, or required verification block
integration. Link the required maintainer scope decision for new providers or
capabilities involving target execution, injection, mutation, or protection
recovery; ordinary fixes within existing boundaries may state "not applicable".
Draft PRs may leave the scope decision pending and the readiness and merge checklists
unchecked. Maintainers can help resolve open questions.

Contract and boundary impact

Semantic owner and earliest changed stage: the domain API-surface projection (WebNetworkApiSurfaceAccumulator plus the new header-facet module). No producer change.
Contribution scope fit and linked maintainer decision: read-only selected-view extension within an existing capability, like #2913/#2982; a new-provider or target-execution decision is not applicable. Part B's decision on #3592 is pending and not needed for this diff.
CLI and MCP/tool-catalog contract: one optional view object, one optional group field, one optional top-level count, one tool-description sentence. No defaults change. Advertised additions measured as serialized JSON Schema: 626 bytes of input property and 2.9 KB of output facet schema; the description grows by 81 characters. Measured in cl100k_base tokens: +141 for the inspect_analysis_view name/description/input-schema projection (6,806 → 6,947) and +141 for the complete catalog (195,649 → 195,790, +0.07%); +995 with output schemas. See Completed-task cost.
Provider, bridge, target-format, or platform compatibility: HAR api-surface only; offline and provider-neutral.
Evidence, artifact, provenance, or reconstruction contract: counts are derived, inferences labelled, and every page is inferred when any group in the full grouping carries an inference; parent ID and record ordinals are cited; original spellings stay in the parent.
Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable; retained Evidence only, the source is never reopened.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: generated catalogs change through the source contract (not committed); authored skill reference updated; no dependencies.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code. (Not applicable: the facet only reads HAR Evidence that REA already retained. It runs no code against any target.)
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections. (Not applicable: it counts recorded header and cookie names. Nothing protected is recovered, and values that HAR producers redact stay redacted.)
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added:
- src/domain/analysisView/webNetworkApiSurfaceView.test.ts: default output over header-rich records pinned to the SHA-256 of the serialized projection computed by unmodified upstream/main fffb07419 code (re-baselined because that main includes null relationship_coverage in Evidence IDs); full facets for both scopes (redacted authorization/cookie, duplicate and case-variant accept, cookie names with an empty cookies array next to a Cookie header, set-cookie names); a known-value no-leak check; inference thresholds (2 samples for either inference, 15 characters, all-decimal, constant, mixed 10/13 widths, padded base64, other, a credential-like name with no special handling) and header_inference_groups presence; unreadable header/cookie names stay unknown; sensitive-parent refusal with facets selected.
- src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts: per-record scratch refused before reading with the group unchanged; each new digest admitted before a row grows (page unchanged at refusal); repeated names, values and cookie names charged once over 1,000 records; 4 Mi-code-unit values (an 8 MiB whole-value encoding) are admitted under a 256 KiB budget, per-record scratch does not grow with value length, no hash update receives more than 1,024 code units (this assertion fails with whole-value hashing), and values differing only in the last slice count as distinct.
- src/application/analysisView/AnalysisViewService.test.ts: an inference in one group makes every page inferred; selected facets without inferences stay derived with header_inference_groups: 0; the default scope is echoed; an unknown scope is an input error at view.request_header_facets.scope.
- tests/boundary/cli/analysisViewCli.test.ts: real HAR decoder and capture service in process, then the compiled CLI for omitted, script-settable and all; asserts no known value in stdout and derived vs inferred.
- scripts/verify-web-network-captures.mjs (real mitmproxy lane): for each producer HAR without sensitive declarations, it also projects api-surface with request_header_facets: {}. It asserts that every group carries the default scope, that record and group counts match the plain projection, and that no recorded request header or cookie value absent from the plain projection appears in the facet output. scripts/ci/scopes.mjs adds the accumulator, view, facet module, data admission and asciiLowercase to the web_network lane, so CI runs that lane for this PR. The sibling endpoint-comparison branch for #3591 adds three of the same lines; whichever lands second keeps the union.
- tests/boundary/mcp/analysisViewMcp.test.ts: decoded parent through MCP, inline vs retained parity, advertised input/output schema validation (combined with include_query_parameter_names), no-leak check, the advertised schema accepting {} and rejecting true, an unknown scope, an extra key and the former scope-only shape, the server applying the default scope, and the server's invalid_request for an unknown scope.
User-visible CLI output, produced by the compiled rea inspect-analysis-view --json on Evidence decoded from the test HAR (/v1/orders group, abridged to the facet):

{"confidence":"inferred", ... "view":{"kind":"api-surface","offset":0,"limit":50,"request_header_facets":{"scope":"script-settable"}},
 "templated_groups":0,"header_inference_groups":1, ...
 "path_template":"/v1/orders","record_ordinals":[0,1,2],
 "request_header_facets":{"scope":"script-settable","excluded_names":3,"request_headers":[
  {"name":"accept","present_records":3,"redacted_records":0,"distinct_values":1,"value_length":{"min":35,"max":35},"alphabets":["other"],"variation":"constant","inferences":[]},
  {"name":"authorization","present_records":3,"redacted_records":3,"distinct_values":null,"value_length":null,"alphabets":[],"variation":"unknown","inferences":[]},
  {"name":"x-signature","present_records":3,"redacted_records":0,"distinct_values":3,"value_length":{"min":64,"max":64},"alphabets":["hex"],"variation":"per-record-unique","inferences":[{"kind":"per-request-token-candidate","basis":"inferred"}]},
  {"name":"x-timestamp","present_records":3,"redacted_records":0,"distinct_values":3,"value_length":{"min":13,"max":13},"alphabets":["decimal"],"variation":"per-record-unique","inferences":[{"kind":"epoch-timestamp-candidate","basis":"inferred"}]}],
  "unavailable_header_name_ordinals":[],"cookie_names":[{"name":"session","record_ordinals":[0,1]}],"unavailable_cookie_name_ordinals":[2],
  "set_cookie_names":[{"name":"csrf","record_ordinals":[0]}],"unavailable_set_cookie_name_ordinals":[]}}

For this 4-record, 2-group fixture the CLI JSON grew from 3,589 bytes (omitted, derived) to 5,832 bytes. Bytes do not establish token cost.

Remaining proof gaps: no real signed-API capture was tested (the cost measurement used a logged-out GitHub session); inference thresholds are untuned; the budget tests establish admission order, not a heap ceiling; the mitmproxy lane runs in CI, not locally.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable. Counts are derived; inferences[] carry basis: "inferred" and, through header_inference_groups, make the view inferred.
Artifact identity, source provenance, and failed attempts remain preserved. Parent ID and record ordinals are cited; budget refusals leave groups unchanged.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. distinct_values: null, variation: "unknown" and the unavailable_*_ordinals lists.

Completed-task cost

Requested in review, per AGENTS.md "Performance". Question: "Which request headers and cookies does each endpoint group in this recording use?"

Recording. A real HAR recorded with Playwright 1.58.2 driving headless Microsoft Edge 155. The profile was fresh and logged out, across six public GitHub pages of this repository: repository, issues, #3622 conversation and files, commits, and Actions.

1,730 entries, recorded with response bodies omitted (content: "omit"). This favours the existing workflow, whose complete record pages would otherwise carry bodies.
The first 1,000 entries, in chronological order, are used: 2,738,891 bytes, SHA-256 7191c83689e913669fd7b301684211d612c6856ef89f9051ea225654abecb1ff. Unmodified main refuses the full recording; see Capacity below.
The HAR is not published, because it holds site-set cookie names and request identifiers.

Preparation (shared, not counted).

Each build's own decodeHarCapture and WebNetworkCaptureService decode the HAR in process. The compiled decoder subprocess needs the Windows native addon, which this host lacks.
The result is admitted to a fresh stdio MCP server (node scripts/rea.mjs mcp) with import_evidence_bundle.
main and the branch decode identical parent Evidence (ev_120136b7…, 4,650,158 serialized bytes).

Workflows. All three run over MCP against the retained parent:

Existing: page api-surface (default limit 50) for groups and record ordinals. Then page capture with detail: "complete" (limit 64) for every record's headers.
Facets: page api-surface with request_header_facets: {}.
Plain: page api-surface without the option, on the branch and on main. This is what callers who don't opt in pay.

The answers agree. All 411 groups were compared:

Every facet header name occurs in that group's complete records.
Listed names plus excluded_names equal the distinct names recorded, with 0 mismatches.
Plain output is byte-identical between main and the branch.

Medians of 5 rounds. Each round starts a fresh server per workflow and runs the task 3 times, and the workflow order rotates each round.

Measurement	Existing (pages + complete records)	Facets	Plain, branch	Plain, `main`
Calls per task	25	9	9	9
Structured reply bytes	4,819,006	560,006	131,304	131,304
Request + reply tokens	1,302,526	143,824	36,819	36,819
+ selected tool schema (`inspect_analysis_view`)	1,309,473	150,771	43,766	43,625
+ complete tool catalog	1,498,316	339,614	232,609	232,468
First task in a fresh server, wall / CPU	5,986 / 6,906 ms	2,462 / 3,391 ms	1,909 / 2,438 ms	2,003 / 3,203 ms
Warm task, wall / CPU	5,701 / 6,156 ms	2,447 / 3,242 ms	1,757 / 2,133 ms	1,892 / 2,195 ms
Peak process-family working set, first task	518 MiB	461 MiB	447 MiB	447 MiB

Tokens.
- Counted with cl100k_base (tiktoken 0.13.0) over serialized values: requests (tool name plus arguments) and structured replies, each counted once.
- Catalog rows project name, description and inputSchema. Including output schemas adds 995 tokens on the branch (807,491 → 808,486).
- These are not hosted-model billing. Reasoning, final answers and agent success rates are unmeasured.
- The existing workflow's 1.3 M tokens exceed common context windows, so a real agent would need extra summarizing or sampling turns, which are not counted.
Time and memory.
- Wall time is observed by the client per task and includes MCP transport and client-side JSON parsing.
- CPU is the server family's kernel plus user time per task.
- Peak is the sum of lifetime peak working sets for the server family (node.exe and its conhost.exe), read through Win32_Process. It includes the ~410 MiB present after preparation, which is the server plus the retained parent.
- "First task" is JIT-cold, not OS-cold: builds and files were warm.
- Host: a shared Windows 11 x64 desktop, Node 24.21.0 (pinned 24.18.0 was not available), with no CPU or memory isolation.
Cost of selecting facets. Compared with plain, for this recording: about 4× the reply tokens (+106,942), about +0.6 s wall, about +1 s CPU and +14 MiB peak.
Cost for callers who don't select facets. Only the schema growth: +141 tokens for inspect_analysis_view (6,806 → 6,947) and for the complete catalog (195,649 → 195,790, +0.07%). Plain output is byte-identical, and wall and CPU differences are within run-to-run noise.

Capacity. Tested on chronological prefixes of this recording:

Plain api-surface on unmodified main admits up to 1,050 entries. It refuses 1,100 and above with "API-surface retained state and temporary work exceed the projection representation budget", including the full 1,730-entry recording (7.7 MB parent). That refusal predates this PR.
With facets, the largest admitted prefix tested is 1,000 entries (1,050 is refused). Facets therefore lower capacity by less than 5% on this recording.

Reproduction. The harness is a Node MCP client, a PowerShell Win32_Process sampler and a tiktoken counter. It is Windows-specific and is not included in this PR. It can be added under scripts/verify/ if maintainers want it in-tree.

Validation performed

Latest integration and hardening, 2026-10-18 UTC

Current head: b87da3d0e4db719bcff6625629ff6c4a3ed70207. All 39 current checks pass, including Portable tests and CI required. GitHub reports mergeable/clean. The large-TOML acceptance cases pass in 4,739 ms (Codex) and 4,804 ms (Grok Build) under the unchanged deadline; all seven descriptor-cleanup cases and all four Int64 TOML acceptance cases pass. Greptile reviewed this head with no new blocking finding. Maintainer approval remains pending.

Merged upstream 847472acf5cf4a28379e828dcd0fe55ac99d7c07. Upstream #4930 adopted the earlier TOML linear-scan and archive/verifier corrections, causing three conflicts. The resolution keeps those upstream implementations and #4937's signed 64-bit TOML support. All 19 remaining PR-diff files are unchanged by this merge, including the header facets and LF/CRLF capacity regressions.

Pinned Node 24.18.0 / npm 11.16.0: 221 tests passed across 11 files covering facets, accumulator, AnalysisViewService, HTML references, MCP/schema validity and configuration source/filesystem behavior. The added compiled-CLI facet case passed separately (222 total); its 13 sibling cases were not selected.
npm run check and git diff --check passed.
Independent probes passed 900 generated-record checks, 17 ordered-value/Unicode cases and 10 budget rejection/recovery cases. Six combined TOML checks covered exact Int64 values with 140,000 interior blank lines in LF/CRLF Codex/Grok documents, repeat registration/removal and out-of-range rejection.
No new actionable header-facet defect was found. The full suite and real-provider lanes were not rerun locally; current-head CI passed the source/built-dependent test lanes and selected real-provider/platform lanes, including real web-network captures. The existing host process-ownership limitation below remains disclosed. No acceptance deadline was increased, and the earlier completed-task benchmark was not rerun.

Earlier CI follow-up, 2026-10-18 UTC

Previous head: df5b3963ffd384f1840e1a923305dc41736a7d47. All 40 checks passed, including Portable tests and CI required. The formerly failing large-TOML CLI file passes both cases: Codex 6,961 ms and Grok Build 6,934 ms, within the unchanged 30-second deadline. Its shard passes 1,635 tests (31 skipped). The descriptor-cleanup file also passes all 7 cases on this head. Greptile reviewed this exact commit without a new blocking finding. Maintainer approval remains pending.

Latest root cause and correction: run 39083611361 passed the descriptor-cleanup shard but timed out the existing Codex large-TOML acceptance test at 30 seconds. The TOML writer used an unanchored greedy newline-suffix regex that repeatedly rescanned an interior run of 140,000 blank lines. Commit df5b396 replaces both suffix trims with a backward LF-only scan. CRLF handling, foreign text, trailing spaces, configuration values and idempotence are preserved. The existing acceptance-test deadline is unchanged.
Validation on the published correction: 59 configuration tests passed: capacity/shape (13), filesystem environment (22), BOM (16), and layered environment (8). New source regressions exercise 140,000 interior blank lines with LF and CRLF through registration, repeat registration and removal. npm run check passed with pinned Node 24.18.0 / npm 11.16.0; git diff --check passed.
Measured source workflow: parsing the 140,000-blank-line document, adding REA, repeating registration and removing it took 29,013 ms first-run / 28,089 ms warm before the fix, versus 111 ms / 69 ms afterward. CPU was 29,103 / 28,172 ms before and 207 / 127 ms after. Single-process peak RSS was 108,260 / 123,532 KiB before and 119,896 / 157,252 KiB after; these are process high-water marks, not isolated per-operation allocations. No child processes ran in this measurement. Both versions preserved the same foreign configuration and repeated output. This is a producer workflow measurement, not a full CLI or agent-token benchmark; process startup, filesystem work and agent token usage were not measured.
Local CLI limitation: the unchanged large-TOML acceptance file cannot reach setup here because host ps fails with fatal library error, lookup self, producing process-ownership unavailable. Both scenarios stop at the dry-run prerequisite locally. CI owns the complete CLI verification.
Previous cleanup failure is confirmed fixed: integration e7a3952 includes upstream 2183c95 for #4926, which retains the exact kernel-limit descriptor cleanup owner and retry lease. On e7a3952, CI Test shard (3/5) passed 1,863 tests (71 skipped), including all 7 recording-descriptor cleanup tests and all 9 verifier-file-resource tests. A local independent pass had 112 tests pass, 3 permission skips and one host-limited capture case; both originally failing kernel cases pass.
Earlier CI corrections: d973785 preserves host PATH in the absent-Ghidra fixture; 7e36ba2 integrates upstream and fixes plist-test formatting; 45b8c3b aligns the archive test with upstream's documented observed "nan" representation. The corrected verifier file (9 tests) and archive file (21 tests) subsequently passed CI.
Previous-head scope: source tests, static/docs/build, real web-network captures, Windows/macOS and native package lanes passed on e7a3952; its sole test failure was the TOML timeout above. Those successes are historical evidence, not a substitute for current-head CI.

Integration refresh, 2026-10-17

Integration merge: bacaee4d433b75fa4e906dd5127baf6747123151, merging upstream/main f0b9ef34c404c1c65f8167502db972b0bd708403 without rewriting the existing commits. GitHub reports the conflicts resolved.

The only conflict was the shared tool-description string in src/contracts/analysisViewToolContracts.ts. The resolution preserves both upstream's JavaScript-facts guidance and this PR's opt-in header-facet guidance. This refresh does not change facet behavior.
The earlier session-test timeout change is now already on main, so sessionMcp.test.ts is no longer part of the PR diff.
Linux validation with pinned Node 24.18.0 / npm 11.16.0 and the current locked dependencies: the focused command below produced 423 passed, 3 failed across 10 files. Every added header-facet case passed.
All three failures also reproduce on an isolated, unmodified main at f0b9ef34c: the existing compiled HAR test reports process-ownership unavailable because ps returns fatal library error, lookup self; the two existing target-free MCP lifecycle cases return no result. The baseline run of those two files gave 19 passed, 3 failed. These are disclosed baseline/local verification gaps, not a claim of a fully passing local suite.
npm run check passed: typecheck, lint, formatting, knip, metadata, module boundaries and architecture guards. npm run docs:check passed, including its 5 consistency tests. Focused preparation compiled the runtime and generated the MCP catalog successfully.
A fresh independent source review found no actionable introduced defect. Read-only probes under Node 24.18.0 passed for ordered duplicate fields, Unicode hashing across chunk boundaries, mixed retained/redacted values and scope filtering.
The previous head 8a62a7065 had 39 successful check runs, including web_network / real-web-network-captures. The refreshed head requires its own CI result; prior CI is not being presented as validation of the new merge.
Not rerun in this refresh: the full suite, the real mitmproxy producer lane locally, or the earlier Windows completed-task performance measurements.

Focused validation command

npm run test:focused -- \
  src/domain/analysisView/webNetworkApiSurfaceView.test.ts \
  src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts \
  src/application/analysisView/AnalysisViewService.test.ts \
  src/domain/javascript/htmlArtifactReferences.test.ts \
  tests/boundary/cli/analysisViewCli.test.ts \
  tests/boundary/mcp/analysisViewMcp.test.ts \
  tests/boundary/mcp/toolSchemaValidity.test.ts \
  tests/boundary/mcp/sessionMcp.test.ts \
  tests/boundary/filesystem/ciChangeScope.test.ts \
  tests/boundary/filesystem/ciRequiredLanes.test.ts

For the baseline, the CLI and session MCP files above were run against unmodified f0b9ef34c with the same pinned toolchain.

Earlier validation

Windows 11 x64, Node 24.21.0. The branch is rebased onto upstream/main a40ee741c (2026-10-15).

npm run test:local -- src/domain/analysisView/webNetworkApiSurfaceView.test.ts src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts src/application/analysisView/AnalysisViewService.test.ts src/domain/javascript/htmlArtifactReferences.test.ts: 128 passed. The first view and accumulator tests failed before the implementation. The long-value test failed with whole-value hashing restored.
npm run test:focused -- tests/boundary/mcp/analysisViewMcp.test.ts tests/boundary/cli/analysisViewCli.test.ts: 30 passed, 1 failed.
- The failure is the existing "decodes HAR credentials and projects recorded API paths through the compiled workflows". The compiled HAR decoder reports "REA Windows native controls are unavailable" because native/windows/build is built only on the Linux/MinGW lane.
- It fails the same way on clean upstream/main. The new CLI test avoids that decoder path.
npm run test:focused -- tests/boundary/filesystem/ciChangeScope.test.ts tests/boundary/filesystem/ciRequiredLanes.test.ts: 228 passed, 2 skipped.
npm run test:focused -- tests/boundary/cli/cliInputSchemaRequired.test.ts (alone): 5 passed.
npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts (alone): 14 passed.
npm run check:fast, npm run format:check, npm run knip, npm run docs:check, npm run mcp-catalog:check and npm run verify:test-discovery: all passed.
npm run test:focused -- tests/boundary/mcp/serverIdentity.test.ts: 3 failed, 8 passed.
- The failures are one registration_path_mismatch assertion and two 10 s timeouts.
- The same 3 fail on clean upstream/main a40ee741c on this host.
New verifier assertion: run locally against the recorded GitHub HAR above through the branch's MCP server, it passed. With a recorded header value injected into a facet row, it failed as intended.
Completed-task cost: the five-round MCP measurement above, plus capacity probes on main and the branch.
After 8a62a7065: npm run test:focused -- tests/boundary/mcp/sessionMcp.test.ts gave 9 passed. The workflow test took 20.6 s on this host, so it would fail under the old 10 s limit.
CI on 600fdb2b0: all 38 checks passed. They include web_network / real-web-network-captures, which now runs for this change and exercises the new facet check on real mitmproxy-produced HARs.
Not run: npm run verify:web:network-captures, because the real mitmproxy lane needs Linux and REA_MITMDUMP_COMMAND locally; it runs in CI. Also not run: the full suite, check:pr, and any Linux or macOS execution.
Review history:
- An independent high-effort code review of the first version found five items. Four were fixed:
  - Confidence comes from a full-grouping header_inference_groups count.
  - Only one hash state is live at a time, with honest scratch charges.
  - Both inferences require at least 3 records.
  - asciiLowercase is shared.
- One cosmetic refactor of RequestHeaderFacetCollector.admissionBytes was deferred.
- Greptile then flagged two more items, both fixed in this commit:
  - Header values are hashed in bounded slices, so long values stay within the budget.
  - request_header_facets: {scope?} replaces the two-field form, so the advertised schema cannot express a request the server rejects.
- Maintainer review asked for a completed-task performance comparison; it is under Completed-task cost. The same review noted the skipped web_network lane; its scope now includes the api-surface files, and its verifier checks facets.
- Rebased onto a40ee741c. The conflicts were the tool description sentence (upstream added typed module references) and test imports.

Compatibility, safety, and release

Breaking changes or migration steps: none; additive and opt-in. Default view output is unchanged (pinned digest and the published example test).
Real Hopper/Ghidra, browser, or OS coverage: no engine; Windows locally. The real mitmproxy HAR lane now runs in CI for this change, checks facets on producer HARs, and passed on 600fdb2b0; it was not run locally.
Package or release metadata impact: catalog descriptions and schemas change; no dependencies.
Security, privacy, process, or containment review: values, prefixes and digests are never emitted. Lengths, distinct counts and alphabets are still metadata about values, which is why the facet is opt-in. Inferences use value shape only, never names. Nothing is replayed, verified or recovered. The WHATWG forbidden-name list is public reference data.

Author readiness

Human owner: @Y0UW1SH

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3622 **Author:** [@Y0UW1SH](https://github.com/Y0UW1SH) **Created:** 10/14/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/3592-api-surface-header-facets` --- ### 📝 Commits (9) - [`600fdb2`](https://github.com/morluto/rea/commit/600fdb2b090e1524a81bb549f921a25f5f3010a0) feat(analysis-view): report value-free request header facets in API surfaces - [`8a62a70`](https://github.com/morluto/rea/commit/8a62a70652bfd9a2f6a7c619aaba2e6d947b8896) test(mcp): inherit the suite timeout for the target-free session workflow - [`bacaee4`](https://github.com/morluto/rea/commit/bacaee4d433b75fa4e906dd5127baf6747123151) chore(analysis-view): sync header facets with main - [`d973785`](https://github.com/morluto/rea/commit/d97378526d3a19b00e0ba413b7fa262f4f66a16e) test(verify): retain host tools for Ghidra prerequisite rejection - [`7e36ba2`](https://github.com/morluto/rea/commit/7e36ba27301d8a1e688256d019306ed0406586a9) chore(ci): sync main and format plist regression - [`45b8c3b`](https://github.com/morluto/rea/commit/45b8c3bf21bad2b736ed8ec5986cc5cc33be9d74) test(plist): align archive regression with canonical NaN labels - [`e7a3952`](https://github.com/morluto/rea/commit/e7a3952d660ea1c0b4a12784fcacbbc95973592d) chore(ci): integrate upstream kernel descriptor cleanup fix - [`df5b396`](https://github.com/morluto/rea/commit/df5b3963ffd384f1840e1a923305dc41736a7d47) fix(clients): avoid quadratic trailing newline scans in TOML - [`b87da3d`](https://github.com/morluto/rea/commit/b87da3d0e4db719bcff6625629ff6c4a3ed70207) chore: reconcile upstream fixes and restore PR mergeability ### 📊 Changes **19 files changed** (+1879 additions, -36 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/evidence-workflows.md` (+5 -0) 📝 `docs/mcp-contracts.md` (+2 -2) 📝 `docs/web-network-captures.md` (+57 -1) 📝 `scripts/ci/scopes.mjs` (+6 -0) 📝 `scripts/verify-web-network-captures.mjs` (+42 -0) 📝 `src/application/ClientConfigurationDocument.capacity.test.ts` (+37 -0) 📝 `src/application/analysisView/AnalysisViewService.test.ts` (+67 -0) 📝 `src/application/analysisView/AnalysisViewService.ts` (+5 -6) 📝 `src/contracts/analysisViewToolContracts.ts` (+1 -1) 📝 `src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts` (+228 -12) 📝 `src/domain/analysisView/WebNetworkApiSurfaceAccumulator.ts` (+94 -5) 📝 `src/domain/analysisView/webNetworkApiSurfaceView.test.ts` (+428 -0) 📝 `src/domain/analysisView/webNetworkApiSurfaceView.ts` (+32 -4) ➕ `src/domain/analysisView/webNetworkRequestHeaderFacets.ts` (+524 -0) ➕ `src/domain/asciiLowercase.ts` (+3 -0) 📝 `src/domain/javascript/htmlArtifactReferences.ts` (+1 -3) 📝 `tests/boundary/cli/analysisViewCli.test.ts` (+148 -1) 📝 `tests/boundary/mcp/analysisViewMcp.test.ts` (+138 -1) 📝 `tests/fixtures/historicalHar.ts` (+61 -0) </details> ### 📄 Description ## Summary This PR adds an opt-in facet to the `api-surface` view over retained HAR Evidence. For each endpoint group it reports the recorded request header names, cookie names and set-cookie names, with value-free signals: how many records carry a header, how many distinct values it had, their length range and character alphabet, and fixed-rule inferences labelled `inferred`. An agent can see which endpoints carry per-request signature, nonce or timestamp headers and trace the producing code, without header or cookie values entering its context. This is Part A of #3592 only. Part B (keeping the auth scheme token from redacted `Authorization` values) is **deferred pending the maintainer decision on #3592**; `CaptureRedaction` and the HAR producer are unchanged, so `authorization` is reported only as present and redacted. It follows the merged opt-in facet `include_query_parameter_names` ([#2982](https://github.com/morluto/rea/pull/2982), [#2913](https://github.com/morluto/rea/issues/2913)) and builds on [#1891](https://github.com/morluto/rea/pull/1891). Open [#3425](https://github.com/morluto/rea/pull/3425) changes the same view and accumulator; see "Overlap with #3425" below. Related: [#2914](https://github.com/morluto/rea/issues/2914), [#2884](https://github.com/morluto/rea/pull/2884), [#2915](https://github.com/morluto/rea/issues/2915). Refs #3592 ## Problem and expected behavior Analyst question: "Which endpoints in this capture are authenticated or signed, under which header and cookie names? Which headers change on every request, like a signature, nonce or timestamp, so I know what code to look for?" Before this change (source-derived): `api-surface` copies no headers ("Query strings, fragments, credentials, headers and bodies are not copied into groups", `docs/web-network-captures.md`). The workaround is paging `capture` views with `detail: "complete"` and comparing header values by eye, which copies values into context. After this change, `view: {kind: "api-surface", request_header_facets: {}}` (or `{scope: "all"}`) returns per-group header and cookie name facets. No value, prefix, substring or digest is emitted. Omitting the option leaves the output byte-identical. ## Change and scope - Request: one optional object, `request_header_facets?: {scope?: "script-settable" | "all"}`. Its presence selects the facets and `scope` defaults to `script-settable`, so the advertised JSON Schema expresses every accepted request structurally (no refinement the schema cannot carry). The echoed `view` records the effective scope. Omitting it leaves groups unchanged. - `script-settable` (default) counts but does not list WHATWG Fetch forbidden request-header names (21 names plus `Proxy-`/`Sec-` prefixes) and HTTP/2 pseudo-headers (`:`-prefixed); `excluded_names` gives the distinct count. The value-dependent method-override rule is not applied. `all` lists every name. - Each group gains `request_header_facets`: - `request_headers[]` sorted by code point: `name` (ASCII-lowercased), `present_records`, `redacted_records`, `distinct_values` (`null` when any present value was redacted or not text), `value_length {min,max}` (UTF-16, `null` when none retained), `alphabets` (first matching rule per value: `decimal`, `hex`, `base64url`, `base64`, `other`), `variation` (`unknown`, `single-sample`, `constant`, `per-record-unique`, `varying`) and `inferences[]`. Repeated field lines in one record form one ordered per-record value. - `cookie_names[]` from `request.cookies[].name` and `set_cookie_names[]` from `response.cookies[].name`, each `{name, record_ordinals}` with exact spelling, listed in both scopes. - `unavailable_header_name_ordinals`, `unavailable_cookie_name_ordinals`, `unavailable_set_cookie_name_ordinals`: records whose names are unknown, including an empty `cookies` array next to a recorded `cookie` / `set-cookie` header. - Redaction is read from the producer's `redactions` sidecar (`/request/headers/N/value`), not inferred from names. - `inferences` (fixed vocabulary, `basis: "inferred"`, never derived from names), both requiring at least 3 present records: `per-request-token-candidate` (all distinct, length ≥16, no `other`, not all `decimal`) and `epoch-timestamp-candidate` (all `decimal`, one length of 10 or 13, more than one distinct value). - A top-level `header_inference_groups`, present only with the selection, counts groups with any header inference across the full grouping, like `templated_groups`. Confidence is `inferred` on every page when `templated_groups > 0 || header_inference_groups > 0`; the application reads these domain counts instead of walking page groups. - Values are hashed in memory (SHA-256 over length-prefixed UTF-16 code units), one header name at a time so one hash state is live. Borrowed values are fed to the hash in fixed 1,024-code-unit slices, so the temporary encoding is bounded instead of growing with a value's length (a 16 MiB value previously produced a full-length temporary buffer). Per-record scratch (name copies, borrowed value lists, maps, one hash state and one slice) is reserved from line counts before reading; new rows, digests, excluded names, cookie names and ordinals are admitted against the existing 96 MiB projection budget before a group changes. Digests are never serialized. The `sensitive_values_unretained` refusal is unchanged. - Files: new `src/domain/analysisView/webNetworkRequestHeaderFacets.ts` (scope rule, alphabets, per-record reading, per-group collector, output schema); new `src/domain/asciiLowercase.ts`, shared with `src/domain/javascript/htmlArtifactReferences.ts` (which drops its private copy); `WebNetworkApiSurfaceAccumulator.ts` (options-object constructor, scratch/admission, `forView`, `header_inference_groups`); `webNetworkApiSurfaceView.ts` (request fields, group and count fields, limitation); `AnalysisViewService.ts` (`forView`, confidence); tool description in `analysisViewToolContracts.ts`; `docs/web-network-captures.md`, `docs/mcp-contracts.md`, the skill's `evidence-workflows.md`. Differences from the pre-implementation draft: one optional `request_header_facets: {scope?}` object instead of a bare enum; added `redacted_records`, `header_inference_groups` and the three `unavailable_*_ordinals` lists so unknown stays distinct from absent; the epoch rule also needs 3 records; `inferences` is always present; no second advertised contract example (the #2982 precedent added none, and examples are forwarded in every input schema); Part B not implemented. **Overlap with #3425.** Both PRs add a `WebNetworkApiSurfaceAccumulator.forView(view)` with the same signature, which conflicts in that hunk and in the constructor (resolution: keep both selections). To compose with #3425 and the sibling endpoint-comparison PR (#3591 work, which adds an `owner` key), the constructor now takes one options object, `{maximumBytes?, includeQueryParameterNames?, requestHeaderScope?}`, exposed as readonly properties; whichever lands second adds one key instead of reordering positional parameters. The `AnalysisViewService` and view-default `forView` call sites are identical to #3425's. The other hunks (schema fields, group schema, group state, admission, page projection, docs, skill text, tests appended at file ends) were placed to merge without conflict. **CI follow-up (separate commit `8a62a7065`).** `tests/boundary/mcp/sessionMcp.test.ts` > "target-free MCP workflow" kept a test-specific 10 s limit. On the macOS runner it took 11.6 s and failed the Real Apple artifact lane in the CI re-run of `600fdb2b0`; that lane is not touched by this change. The test now inherits the shared suite deadline (30 s, 60 s under coverage), with assertions and cleanup unchanged. The other three 10 s limits in that file took 0.05 to 2 s on the same runner and are left as they are. **Non-goals:** no values, prefixes or hashes emitted; no secrecy or role guesses from names; no live CDP headers; no SAZ or mitmproxy until #2884/#2915; no replay, signature verification or key recovery; no claim that a name absent from the HAR was absent on the wire. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". Draft PRs may leave the scope decision pending and the readiness and merge checklists unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact - Semantic owner and earliest changed stage: the domain API-surface projection (`WebNetworkApiSurfaceAccumulator` plus the new header-facet module). No producer change. - Contribution scope fit and linked maintainer decision: read-only selected-view extension within an existing capability, like #2913/#2982; a new-provider or target-execution decision is not applicable. Part B's decision on #3592 is pending and not needed for this diff. - CLI and MCP/tool-catalog contract: one optional view object, one optional group field, one optional top-level count, one tool-description sentence. No defaults change. Advertised additions measured as serialized JSON Schema: 626 bytes of input property and 2.9 KB of output facet schema; the description grows by 81 characters. Measured in `cl100k_base` tokens: +141 for the `inspect_analysis_view` name/description/input-schema projection (6,806 → 6,947) and +141 for the complete catalog (195,649 → 195,790, +0.07%); +995 with output schemas. See Completed-task cost. - Provider, bridge, target-format, or platform compatibility: HAR `api-surface` only; offline and provider-neutral. - Evidence, artifact, provenance, or reconstruction contract: counts are derived, inferences labelled, and every page is `inferred` when any group in the full grouping carries an inference; parent ID and record ordinals are cited; original spellings stay in the parent. - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable; retained Evidence only, the source is never reopened. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: generated catalogs change through the source contract (not committed); authored skill reference updated; no dependencies. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. (Not applicable: the facet only reads HAR Evidence that REA already retained. It runs no code against any target.) - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. (Not applicable: it counts recorded header and cookie names. Nothing protected is recovered, and values that HAR producers redact stay redacted.) - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added: - `src/domain/analysisView/webNetworkApiSurfaceView.test.ts`: default output over header-rich records pinned to the SHA-256 of the serialized projection computed by unmodified `upstream/main` `fffb07419` code (re-baselined because that main includes null `relationship_coverage` in Evidence IDs); full facets for both scopes (redacted `authorization`/`cookie`, duplicate and case-variant `accept`, cookie names with an empty `cookies` array next to a `Cookie` header, set-cookie names); a known-value no-leak check; inference thresholds (2 samples for either inference, 15 characters, all-decimal, constant, mixed 10/13 widths, padded base64, `other`, a credential-like name with no special handling) and `header_inference_groups` presence; unreadable header/cookie names stay unknown; sensitive-parent refusal with facets selected. - `src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts`: per-record scratch refused before reading with the group unchanged; each new digest admitted before a row grows (page unchanged at refusal); repeated names, values and cookie names charged once over 1,000 records; 4 Mi-code-unit values (an 8 MiB whole-value encoding) are admitted under a 256 KiB budget, per-record scratch does not grow with value length, no hash update receives more than 1,024 code units (this assertion fails with whole-value hashing), and values differing only in the last slice count as distinct. - `src/application/analysisView/AnalysisViewService.test.ts`: an inference in one group makes every page `inferred`; selected facets without inferences stay `derived` with `header_inference_groups: 0`; the default scope is echoed; an unknown scope is an input error at `view.request_header_facets.scope`. - `tests/boundary/cli/analysisViewCli.test.ts`: real HAR decoder and capture service in process, then the compiled CLI for omitted, `script-settable` and `all`; asserts no known value in stdout and `derived` vs `inferred`. - `scripts/verify-web-network-captures.mjs` (real mitmproxy lane): for each producer HAR without sensitive declarations, it also projects `api-surface` with `request_header_facets: {}`. It asserts that every group carries the default scope, that record and group counts match the plain projection, and that no recorded request header or cookie value absent from the plain projection appears in the facet output. `scripts/ci/scopes.mjs` adds the accumulator, view, facet module, data admission and `asciiLowercase` to the `web_network` lane, so CI runs that lane for this PR. The sibling endpoint-comparison branch for #3591 adds three of the same lines; whichever lands second keeps the union. - `tests/boundary/mcp/analysisViewMcp.test.ts`: decoded parent through MCP, inline vs retained parity, advertised input/output schema validation (combined with `include_query_parameter_names`), no-leak check, the advertised schema accepting `{}` and rejecting `true`, an unknown scope, an extra key and the former scope-only shape, the server applying the default scope, and the server's `invalid_request` for an unknown scope. - User-visible CLI output, produced by the compiled `rea inspect-analysis-view --json` on Evidence decoded from the test HAR (`/v1/orders` group, abridged to the facet): ```json {"confidence":"inferred", ... "view":{"kind":"api-surface","offset":0,"limit":50,"request_header_facets":{"scope":"script-settable"}}, "templated_groups":0,"header_inference_groups":1, ... "path_template":"/v1/orders","record_ordinals":[0,1,2], "request_header_facets":{"scope":"script-settable","excluded_names":3,"request_headers":[ {"name":"accept","present_records":3,"redacted_records":0,"distinct_values":1,"value_length":{"min":35,"max":35},"alphabets":["other"],"variation":"constant","inferences":[]}, {"name":"authorization","present_records":3,"redacted_records":3,"distinct_values":null,"value_length":null,"alphabets":[],"variation":"unknown","inferences":[]}, {"name":"x-signature","present_records":3,"redacted_records":0,"distinct_values":3,"value_length":{"min":64,"max":64},"alphabets":["hex"],"variation":"per-record-unique","inferences":[{"kind":"per-request-token-candidate","basis":"inferred"}]}, {"name":"x-timestamp","present_records":3,"redacted_records":0,"distinct_values":3,"value_length":{"min":13,"max":13},"alphabets":["decimal"],"variation":"per-record-unique","inferences":[{"kind":"epoch-timestamp-candidate","basis":"inferred"}]}], "unavailable_header_name_ordinals":[],"cookie_names":[{"name":"session","record_ordinals":[0,1]}],"unavailable_cookie_name_ordinals":[2], "set_cookie_names":[{"name":"csrf","record_ordinals":[0]}],"unavailable_set_cookie_name_ordinals":[]}} ``` For this 4-record, 2-group fixture the CLI JSON grew from 3,589 bytes (omitted, `derived`) to 5,832 bytes. Bytes do not establish token cost. - Remaining proof gaps: no real signed-API capture was tested (the cost measurement used a logged-out GitHub session); inference thresholds are untuned; the budget tests establish admission order, not a heap ceiling; the mitmproxy lane runs in CI, not locally. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. Counts are derived; `inferences[]` carry `basis: "inferred"` and, through `header_inference_groups`, make the view inferred. - [x] Artifact identity, source provenance, and failed attempts remain preserved. Parent ID and record ordinals are cited; budget refusals leave groups unchanged. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. `distinct_values: null`, `variation: "unknown"` and the `unavailable_*_ordinals` lists. ## Completed-task cost Requested in review, per AGENTS.md "Performance". Question: "Which request headers and cookies does each endpoint group in this recording use?" **Recording.** A real HAR recorded with Playwright 1.58.2 driving headless Microsoft Edge 155. The profile was fresh and logged out, across six public GitHub pages of this repository: repository, issues, #3622 conversation and files, commits, and Actions. - 1,730 entries, recorded with response bodies omitted (`content: "omit"`). This favours the existing workflow, whose complete record pages would otherwise carry bodies. - The first 1,000 entries, in chronological order, are used: 2,738,891 bytes, SHA-256 `7191c83689e913669fd7b301684211d612c6856ef89f9051ea225654abecb1ff`. Unmodified `main` refuses the full recording; see Capacity below. - The HAR is not published, because it holds site-set cookie names and request identifiers. **Preparation (shared, not counted).** - Each build's own `decodeHarCapture` and `WebNetworkCaptureService` decode the HAR in process. The compiled decoder subprocess needs the Windows native addon, which this host lacks. - The result is admitted to a fresh stdio MCP server (`node scripts/rea.mjs mcp`) with `import_evidence_bundle`. - `main` and the branch decode identical parent Evidence (`ev_120136b7…`, 4,650,158 serialized bytes). **Workflows.** All three run over MCP against the retained parent: - **Existing:** page `api-surface` (default limit 50) for groups and record ordinals. Then page `capture` with `detail: "complete"` (limit 64) for every record's headers. - **Facets:** page `api-surface` with `request_header_facets: {}`. - **Plain:** page `api-surface` without the option, on the branch and on `main`. This is what callers who don't opt in pay. **The answers agree.** All 411 groups were compared: - Every facet header name occurs in that group's complete records. - Listed names plus `excluded_names` equal the distinct names recorded, with 0 mismatches. - Plain output is byte-identical between `main` and the branch. Medians of 5 rounds. Each round starts a fresh server per workflow and runs the task 3 times, and the workflow order rotates each round. | Measurement | Existing (pages + complete records) | Facets | Plain, branch | Plain, `main` | | ------------------------------------------------ | ----------------------------------: | --------------: | --------------: | ----------------: | | Calls per task | 25 | 9 | 9 | 9 | | Structured reply bytes | 4,819,006 | 560,006 | 131,304 | 131,304 | | Request + reply tokens | 1,302,526 | 143,824 | 36,819 | 36,819 | | + selected tool schema (`inspect_analysis_view`) | 1,309,473 | 150,771 | 43,766 | 43,625 | | + complete tool catalog | 1,498,316 | 339,614 | 232,609 | 232,468 | | First task in a fresh server, wall / CPU | 5,986 / 6,906 ms | 2,462 / 3,391 ms | 1,909 / 2,438 ms | 2,003 / 3,203 ms | | Warm task, wall / CPU | 5,701 / 6,156 ms | 2,447 / 3,242 ms | 1,757 / 2,133 ms | 1,892 / 2,195 ms | | Peak process-family working set, first task | 518 MiB | 461 MiB | 447 MiB | 447 MiB | - **Tokens.** - Counted with `cl100k_base` (tiktoken 0.13.0) over serialized values: requests (tool name plus arguments) and structured replies, each counted once. - Catalog rows project name, description and `inputSchema`. Including output schemas adds 995 tokens on the branch (807,491 → 808,486). - These are not hosted-model billing. Reasoning, final answers and agent success rates are unmeasured. - The existing workflow's 1.3 M tokens exceed common context windows, so a real agent would need extra summarizing or sampling turns, which are not counted. - **Time and memory.** - Wall time is observed by the client per task and includes MCP transport and client-side JSON parsing. - CPU is the server family's kernel plus user time per task. - Peak is the sum of lifetime peak working sets for the server family (`node.exe` and its `conhost.exe`), read through `Win32_Process`. It includes the ~410 MiB present after preparation, which is the server plus the retained parent. - "First task" is JIT-cold, not OS-cold: builds and files were warm. - Host: a shared Windows 11 x64 desktop, Node 24.21.0 (pinned 24.18.0 was not available), with no CPU or memory isolation. - **Cost of selecting facets.** Compared with plain, for this recording: about 4× the reply tokens (+106,942), about +0.6 s wall, about +1 s CPU and +14 MiB peak. - **Cost for callers who don't select facets.** Only the schema growth: +141 tokens for `inspect_analysis_view` (6,806 → 6,947) and for the complete catalog (195,649 → 195,790, +0.07%). Plain output is byte-identical, and wall and CPU differences are within run-to-run noise. **Capacity.** Tested on chronological prefixes of this recording: - Plain `api-surface` on unmodified `main` admits up to 1,050 entries. It refuses 1,100 and above with "API-surface retained state and temporary work exceed the projection representation budget", including the full 1,730-entry recording (7.7 MB parent). That refusal predates this PR. - With facets, the largest admitted prefix tested is 1,000 entries (1,050 is refused). Facets therefore lower capacity by less than 5% on this recording. **Reproduction.** The harness is a Node MCP client, a PowerShell `Win32_Process` sampler and a tiktoken counter. It is Windows-specific and is not included in this PR. It can be added under `scripts/verify/` if maintainers want it in-tree. ## Validation performed ### Latest integration and hardening, 2026-10-18 UTC Current head: `b87da3d0e4db719bcff6625629ff6c4a3ed70207`. **[All 39 current checks pass](https://github.com/morluto/rea/actions/runs/39090519423), including Portable tests and CI required.** GitHub reports mergeable/clean. The large-TOML acceptance cases pass in 4,739 ms (Codex) and 4,804 ms (Grok Build) under the unchanged deadline; all seven descriptor-cleanup cases and all four Int64 TOML acceptance cases pass. Greptile reviewed this head with no new blocking finding. Maintainer approval remains pending. Merged upstream `847472acf5cf4a28379e828dcd0fe55ac99d7c07`. Upstream #4930 adopted the earlier TOML linear-scan and archive/verifier corrections, causing three conflicts. The resolution keeps those upstream implementations and #4937's signed 64-bit TOML support. All 19 remaining PR-diff files are unchanged by this merge, including the header facets and LF/CRLF capacity regressions. - Pinned Node 24.18.0 / npm 11.16.0: **221 tests passed across 11 files** covering facets, accumulator, AnalysisViewService, HTML references, MCP/schema validity and configuration source/filesystem behavior. The added compiled-CLI facet case passed separately (**222 total**); its 13 sibling cases were not selected. - `npm run check` and `git diff --check` passed. - Independent probes passed 900 generated-record checks, 17 ordered-value/Unicode cases and 10 budget rejection/recovery cases. Six combined TOML checks covered exact Int64 values with 140,000 interior blank lines in LF/CRLF Codex/Grok documents, repeat registration/removal and out-of-range rejection. - No new actionable header-facet defect was found. The full suite and real-provider lanes were not rerun locally; current-head CI passed the source/built-dependent test lanes and selected real-provider/platform lanes, including real web-network captures. The existing host process-ownership limitation below remains disclosed. No acceptance deadline was increased, and the earlier completed-task benchmark was not rerun. ### Earlier CI follow-up, 2026-10-18 UTC Previous head: `df5b3963ffd384f1840e1a923305dc41736a7d47`. **[All 40 checks passed](https://github.com/morluto/rea/actions/runs/39084704062), including Portable tests and CI required.** The formerly failing large-TOML CLI file passes both cases: Codex 6,961 ms and Grok Build 6,934 ms, within the unchanged 30-second deadline. Its shard passes 1,635 tests (31 skipped). The descriptor-cleanup file also passes all 7 cases on this head. Greptile reviewed this exact commit without a new blocking finding. Maintainer approval remains pending. - **Latest root cause and correction:** [run 39083611361](https://github.com/morluto/rea/actions/runs/39083611361/job/117545671933) passed the descriptor-cleanup shard but timed out the existing Codex large-TOML acceptance test at 30 seconds. The TOML writer used an unanchored greedy newline-suffix regex that repeatedly rescanned an interior run of 140,000 blank lines. Commit `df5b396` replaces both suffix trims with a backward LF-only scan. CRLF handling, foreign text, trailing spaces, configuration values and idempotence are preserved. The existing acceptance-test deadline is unchanged. - **Validation on the published correction:** 59 configuration tests passed: capacity/shape (13), filesystem environment (22), BOM (16), and layered environment (8). New source regressions exercise 140,000 interior blank lines with LF and CRLF through registration, repeat registration and removal. `npm run check` passed with pinned Node 24.18.0 / npm 11.16.0; `git diff --check` passed. - **Measured source workflow:** parsing the 140,000-blank-line document, adding REA, repeating registration and removing it took 29,013 ms first-run / 28,089 ms warm before the fix, versus 111 ms / 69 ms afterward. CPU was 29,103 / 28,172 ms before and 207 / 127 ms after. Single-process peak RSS was 108,260 / 123,532 KiB before and 119,896 / 157,252 KiB after; these are process high-water marks, not isolated per-operation allocations. No child processes ran in this measurement. Both versions preserved the same foreign configuration and repeated output. This is a producer workflow measurement, not a full CLI or agent-token benchmark; process startup, filesystem work and agent token usage were not measured. - **Local CLI limitation:** the unchanged large-TOML acceptance file cannot reach setup here because host `ps` fails with `fatal library error, lookup self`, producing `process-ownership` unavailable. Both scenarios stop at the dry-run prerequisite locally. CI owns the complete CLI verification. - **Previous cleanup failure is confirmed fixed:** integration `e7a3952` includes upstream `2183c95` for #4926, which retains the exact kernel-limit descriptor cleanup owner and retry lease. On `e7a3952`, CI Test shard (3/5) passed 1,863 tests (71 skipped), including all 7 recording-descriptor cleanup tests and all 9 verifier-file-resource tests. A local independent pass had 112 tests pass, 3 permission skips and one host-limited capture case; both originally failing kernel cases pass. - **Earlier CI corrections:** `d973785` preserves host PATH in the absent-Ghidra fixture; `7e36ba2` integrates upstream and fixes plist-test formatting; `45b8c3b` aligns the archive test with upstream's documented observed `"nan"` representation. The corrected verifier file (9 tests) and archive file (21 tests) subsequently passed CI. - **Previous-head scope:** source tests, static/docs/build, real web-network captures, Windows/macOS and native package lanes passed on `e7a3952`; its sole test failure was the TOML timeout above. Those successes are historical evidence, not a substitute for current-head CI. ### Integration refresh, 2026-10-17 Integration merge: `bacaee4d433b75fa4e906dd5127baf6747123151`, merging `upstream/main` `f0b9ef34c404c1c65f8167502db972b0bd708403` without rewriting the existing commits. GitHub reports the conflicts resolved. - The only conflict was the shared tool-description string in `src/contracts/analysisViewToolContracts.ts`. The resolution preserves both upstream's JavaScript-facts guidance and this PR's opt-in header-facet guidance. This refresh does not change facet behavior. - The earlier session-test timeout change is now already on `main`, so `sessionMcp.test.ts` is no longer part of the PR diff. - Linux validation with pinned **Node 24.18.0 / npm 11.16.0** and the current locked dependencies: the focused command below produced **423 passed, 3 failed** across 10 files. Every added header-facet case passed. - All three failures also reproduce on an isolated, unmodified `main` at `f0b9ef34c`: the existing compiled HAR test reports `process-ownership` unavailable because `ps` returns `fatal library error, lookup self`; the two existing target-free MCP lifecycle cases return no `result`. The baseline run of those two files gave **19 passed, 3 failed**. These are disclosed baseline/local verification gaps, not a claim of a fully passing local suite. - `npm run check` passed: typecheck, lint, formatting, knip, metadata, module boundaries and architecture guards. `npm run docs:check` passed, including its 5 consistency tests. Focused preparation compiled the runtime and generated the MCP catalog successfully. - A fresh independent source review found no actionable introduced defect. Read-only probes under Node 24.18.0 passed for ordered duplicate fields, Unicode hashing across chunk boundaries, mixed retained/redacted values and scope filtering. - The previous head `8a62a7065` had **39 successful check runs**, including `web_network / real-web-network-captures`. The refreshed head requires its own CI result; prior CI is not being presented as validation of the new merge. - Not rerun in this refresh: the full suite, the real mitmproxy producer lane locally, or the earlier Windows completed-task performance measurements. <details> <summary>Focused validation command</summary> ```bash npm run test:focused -- \ src/domain/analysisView/webNetworkApiSurfaceView.test.ts \ src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts \ src/application/analysisView/AnalysisViewService.test.ts \ src/domain/javascript/htmlArtifactReferences.test.ts \ tests/boundary/cli/analysisViewCli.test.ts \ tests/boundary/mcp/analysisViewMcp.test.ts \ tests/boundary/mcp/toolSchemaValidity.test.ts \ tests/boundary/mcp/sessionMcp.test.ts \ tests/boundary/filesystem/ciChangeScope.test.ts \ tests/boundary/filesystem/ciRequiredLanes.test.ts ``` For the baseline, the CLI and session MCP files above were run against unmodified `f0b9ef34c` with the same pinned toolchain. </details> ### Earlier validation Windows 11 x64, Node 24.21.0. The branch is rebased onto `upstream/main` `a40ee741c` (2026-10-15). - `npm run test:local -- src/domain/analysisView/webNetworkApiSurfaceView.test.ts src/domain/analysisView/WebNetworkApiSurfaceAccumulator.test.ts src/application/analysisView/AnalysisViewService.test.ts src/domain/javascript/htmlArtifactReferences.test.ts`: 128 passed. The first view and accumulator tests failed before the implementation. The long-value test failed with whole-value hashing restored. - `npm run test:focused -- tests/boundary/mcp/analysisViewMcp.test.ts tests/boundary/cli/analysisViewCli.test.ts`: 30 passed, 1 failed. - The failure is the existing "decodes HAR credentials and projects recorded API paths through the compiled workflows". The compiled HAR decoder reports "REA Windows native controls are unavailable" because `native/windows/build` is built only on the Linux/MinGW lane. - It fails the same way on clean `upstream/main`. The new CLI test avoids that decoder path. - `npm run test:focused -- tests/boundary/filesystem/ciChangeScope.test.ts tests/boundary/filesystem/ciRequiredLanes.test.ts`: 228 passed, 2 skipped. - `npm run test:focused -- tests/boundary/cli/cliInputSchemaRequired.test.ts` (alone): 5 passed. - `npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts` (alone): 14 passed. - `npm run check:fast`, `npm run format:check`, `npm run knip`, `npm run docs:check`, `npm run mcp-catalog:check` and `npm run verify:test-discovery`: all passed. - `npm run test:focused -- tests/boundary/mcp/serverIdentity.test.ts`: 3 failed, 8 passed. - The failures are one `registration_path_mismatch` assertion and two 10 s timeouts. - The same 3 fail on clean `upstream/main` `a40ee741c` on this host. - New verifier assertion: run locally against the recorded GitHub HAR above through the branch's MCP server, it passed. With a recorded header value injected into a facet row, it failed as intended. - Completed-task cost: the five-round MCP measurement above, plus capacity probes on `main` and the branch. - After `8a62a7065`: `npm run test:focused -- tests/boundary/mcp/sessionMcp.test.ts` gave 9 passed. The workflow test took 20.6 s on this host, so it would fail under the old 10 s limit. - CI on `600fdb2b0`: all 38 checks passed. They include `web_network / real-web-network-captures`, which now runs for this change and exercises the new facet check on real mitmproxy-produced HARs. - Not run: `npm run verify:web:network-captures`, because the real mitmproxy lane needs Linux and `REA_MITMDUMP_COMMAND` locally; it runs in CI. Also not run: the full suite, `check:pr`, and any Linux or macOS execution. - **Review history:** - An independent high-effort code review of the first version found five items. Four were fixed: - Confidence comes from a full-grouping `header_inference_groups` count. - Only one hash state is live at a time, with honest scratch charges. - Both inferences require at least 3 records. - `asciiLowercase` is shared. - One cosmetic refactor of `RequestHeaderFacetCollector.admissionBytes` was deferred. - Greptile then flagged two more items, both fixed in this commit: - Header values are hashed in bounded slices, so long values stay within the budget. - `request_header_facets: {scope?}` replaces the two-field form, so the advertised schema cannot express a request the server rejects. - Maintainer review asked for a completed-task performance comparison; it is under Completed-task cost. The same review noted the skipped `web_network` lane; its scope now includes the api-surface files, and its verifier checks facets. - Rebased onto `a40ee741c`. The conflicts were the tool description sentence (upstream added typed module references) and test imports. ## Compatibility, safety, and release - Breaking changes or migration steps: none; additive and opt-in. Default view output is unchanged (pinned digest and the published example test). - Real Hopper/Ghidra, browser, or OS coverage: no engine; Windows locally. The real mitmproxy HAR lane now runs in CI for this change, checks facets on producer HARs, and passed on `600fdb2b0`; it was not run locally. - Package or release metadata impact: catalog descriptions and schemas change; no dependencies. - Security, privacy, process, or containment review: values, prefixes and digests are never emitted. Lengths, distinct counts and alphabets are still metadata about values, which is why the facet is opt-in. Inferences use value shape only, never names. Nothing is replayed, verified or recovered. The WHATWG forbidden-name list is public reference data. ## Author readiness  Human owner: @Y0UW1SH - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:09:58 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:45:18 +00:00

[PR #5858] [MERGED] fix(signals): report unknown decoder help IDs as invalid input #7089

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5858

No description provided.

Rows
Columns