[GH-ISSUE #2671] fix(capture): native capture loses ordinary URL text and original bytes #972

New issue

Closed

opened 2026-10-15 18:30:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:30:00 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2671

Summary

Inspecting a native mitmproxy capture can replace a recorded Location value with a different destination and discard the original bytes from returned Evidence. An HTTPS URL containing a backslash followed by ordinary path text with an at-sign triggers the failure even when no userinfo is present. The same projection also handles Referer, Origin and native request.path fields.

Steps to reproduce

Use either source revision below. Run this command from the checkout root with Python. It executes the unchanged native adapter's record projection on one owned header fixture, without importing mitmproxy or starting a proxy:

python3 - <<'PY'
import ast, base64, hashlib, json, math, re
from pathlib import Path

source = ast.parse(Path("bridge/mitmproxy/capture.py").read_text())
names = {"CaptureFailure", "pointer", "project_record"}
nodes = [node for node in source.body if
    isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in names
    or isinstance(node, ast.Assign) and any(
        isinstance(target, ast.Name) and target.id == "CREDENTIALS"
        for target in node.targets)]
exec(compile(ast.Module(body=nodes, type_ignores=[]), "capture-projection", "exec"))
value = b"https://example.test\\part@tail"
result = project_record({"response": {"headers": [[b"Location", value]]}}, [])
print(json.dumps(result, indent=2))
PY

The one-header fixture above was executed. A separate source test also passed the resulting record through the shared capture service with an injected port. Full native CLI/MCP reproduction was not run. That step requires Linux, caller-supplied mitmdump 12.2.3 selected through REA_MITMDUMP_COMMAND, and a native tnetstring capture carrying these same header bytes. The offline adapter accepts dictionary states and ordered byte headers without requiring this value to be a canonical URL.

Expected behavior

The reported Location value remains exactly https://example.test\part@tail, with one backslash in the actual bytes. Its original byte sidecar remains retained with content, size and digest. The historical-capture contract preserves ordinary path/query text and removes only applicable userinfo spans. Here the shared recorded-URL parser reports no userinfo.

Actual behavior

The executed native projection returned https://tail. The byte sidecar for that header value had the following observed output:

{
  "pointer": "/response/headers/0/1",
  "representation": "producer-bytes",
  "state": "redacted",
  "content_base64": null,
  "bytes": null,
  "sha256": null
}

The projected record passes the record schema, and the shared capture service retains the changed Location text in normalized_result. With a forward slash instead of the backslash, the same projection preserves both text and bytes. The shared HAR text projector also preserves the original backslash spelling. These observations establish the projection and shared-consumer failure; they do not establish a completed native mitmdump or installed-package run.

Evidence

Expected source: AGENTS.md, Product and Authority, requires preserving caller-selected URLs and local evidence; docs/web-network-captures.md:200-207 promises unchanged noncredential path/query text; src/domain/recordedTransportUrl.ts:37-60 treats a backslash as the HTTPS authority terminator.
Failure source: Executed unchanged project_record in bridge/mitmproxy/capture.py:44-146 on the inline one-header fixture. Its URL projection at lines 61-74 returned the changed text, and its byte projection at lines 92-108 omitted the original bytes. An executed schema/shared-service check showed the wrong value survives src/application/WebNetworkCaptureService.ts:70-103 and Evidence construction.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: incomplete
Missing fact: The native public operation's reply for these bytes on Linux with the configured pinned mitmdump executable; the projection and injected-port service were executed independently instead.

These observations apply to inspect_web_network_capture and inspect-web-network-capture with format: "mitmproxy" and no explicit sensitive-value exclusions. Applicability to the full native operation is source-derived; its Linux/mitmdump runtime reply remains the missing verification stated above.

Runtime or environment

Source revision: a2d8b94dd5. Source checks used Node 24.18.0 / npm 11.16.0 and Python 3.14.7. The real native adapter requires Linux and the pinned mitmdump profile. This report does not assert that a published package was tested. Current source recheck at 3ae826ceba passed three focused projection, forward-slash control and schema/shared-service checks with the same toolchain; native CLI/MCP and published-package execution remain untested.

Restoration check

The inline native header fixture must return the exact backslash-containing value and retain its original-byte sidecar. The existing forward-slash control must continue retaining text and bytes. Canonical userinfo removal must retain its existing behavior so preservation of ordinary path text does not turn into indiscriminate retention of every URL. The shared service must return the restored native record unchanged in Evidence.

Additional context

The historical import capability is described in the original feature report. This failure concerns preservation of a supported native field after decoding. Exporting to HAR might change the producer's original spelling, so that route has not been established as a faithful workaround. Related: #2600 and #2601 concern separate SAZ input-classification and XML-validation failures in the same inspection workflow.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A bounded recorded URL spelling produces a different reported destination and omits its original-byte sidecar, which can mislead offline capture analysis. The original capture is not modified, and affected-user frequency is unknown.
Workaround status: unverified
Workaround basis: The shared HAR projector preserves this spelling, but a producer export that keeps the original value was not tested. Reading the original native artifact avoids relying on the incorrect projection, but a complete alternative inspection workflow was not verified.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2671 ## Summary Inspecting a native mitmproxy capture can replace a recorded Location value with a different destination and discard the original bytes from returned Evidence. An HTTPS URL containing a backslash followed by ordinary path text with an at-sign triggers the failure even when no userinfo is present. The same projection also handles Referer, Origin and native request.path fields. ## Steps to reproduce Use either source revision below. Run this command from the checkout root with Python. It executes the unchanged native adapter's record projection on one owned header fixture, without importing mitmproxy or starting a proxy: ```sh python3 - <<'PY' import ast, base64, hashlib, json, math, re from pathlib import Path source = ast.parse(Path("bridge/mitmproxy/capture.py").read_text()) names = {"CaptureFailure", "pointer", "project_record"} nodes = [node for node in source.body if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in names or isinstance(node, ast.Assign) and any( isinstance(target, ast.Name) and target.id == "CREDENTIALS" for target in node.targets)] exec(compile(ast.Module(body=nodes, type_ignores=[]), "capture-projection", "exec")) value = b"https://example.test\\part@tail" result = project_record({"response": {"headers": [[b"Location", value]]}}, []) print(json.dumps(result, indent=2)) PY ``` The one-header fixture above was executed. A separate source test also passed the resulting record through the shared capture service with an injected port. Full native CLI/MCP reproduction was not run. That step requires Linux, caller-supplied mitmdump 12.2.3 selected through REA_MITMDUMP_COMMAND, and a native tnetstring capture carrying these same header bytes. The offline adapter accepts dictionary states and ordered byte headers without requiring this value to be a canonical URL. ## Expected behavior The reported Location value remains exactly `https://example.test\part@tail`, with one backslash in the actual bytes. Its original byte sidecar remains retained with content, size and digest. The historical-capture contract preserves ordinary path/query text and removes only applicable userinfo spans. Here the shared recorded-URL parser reports no userinfo. ## Actual behavior The executed native projection returned `https://tail`. The byte sidecar for that header value had the following observed output: ```json { "pointer": "/response/headers/0/1", "representation": "producer-bytes", "state": "redacted", "content_base64": null, "bytes": null, "sha256": null } ``` The projected record passes the record schema, and the shared capture service retains the changed Location text in normalized_result. With a forward slash instead of the backslash, the same projection preserves both text and bytes. The shared HAR text projector also preserves the original backslash spelling. These observations establish the projection and shared-consumer failure; they do not establish a completed native mitmdump or installed-package run. ## Evidence - Expected source: `AGENTS.md`, Product and Authority, requires preserving caller-selected URLs and local evidence; `docs/web-network-captures.md:200-207` promises unchanged noncredential path/query text; `src/domain/recordedTransportUrl.ts:37-60` treats a backslash as the HTTPS authority terminator. - Failure source: Executed unchanged `project_record` in `bridge/mitmproxy/capture.py:44-146` on the inline one-header fixture. Its URL projection at lines 61-74 returned the changed text, and its byte projection at lines 92-108 omitted the original bytes. An executed schema/shared-service check showed the wrong value survives `src/application/WebNetworkCaptureService.ts:70-103` and Evidence construction. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: incomplete - Missing fact: The native public operation's reply for these bytes on Linux with the configured pinned mitmdump executable; the projection and injected-port service were executed independently instead. These observations apply to `inspect_web_network_capture` and `inspect-web-network-capture` with `format: "mitmproxy"` and no explicit sensitive-value exclusions. Applicability to the full native operation is source-derived; its Linux/mitmdump runtime reply remains the missing verification stated above. ## Runtime or environment Source revision: a2d8b94dd50dab8d966239dd63ab84718aa8b4b4. Source checks used Node 24.18.0 / npm 11.16.0 and Python 3.14.7. The real native adapter requires Linux and the pinned mitmdump profile. This report does not assert that a published package was tested. Current source recheck at 3ae826ceba6f25b08bce50bcb51001209bc16b57 passed three focused projection, forward-slash control and schema/shared-service checks with the same toolchain; native CLI/MCP and published-package execution remain untested. ## Restoration check The inline native header fixture must return the exact backslash-containing value and retain its original-byte sidecar. The existing forward-slash control must continue retaining text and bytes. Canonical userinfo removal must retain its existing behavior so preservation of ordinary path text does not turn into indiscriminate retention of every URL. The shared service must return the restored native record unchanged in Evidence. ## Additional context The historical import capability is described in [the original feature report](https://github.com/morluto/rea/issues/882). This failure concerns preservation of a supported native field after decoding. Exporting to HAR might change the producer's original spelling, so that route has not been established as a faithful workaround. Related: #2600 and #2601 concern separate SAZ input-classification and XML-validation failures in the same inspection workflow. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A bounded recorded URL spelling produces a different reported destination and omits its original-byte sidecar, which can mislead offline capture analysis. The original capture is not modified, and affected-user frequency is unknown. - Workaround status: unverified - Workaround basis: The shared HAR projector preserves this spelling, but a producer export that keeps the original value was not tested. Reading the original native artifact avoids relying on the incorrect projection, but a complete alternative inspection workflow was not verified.

bsakaguchi

2026-10-15 18:30:00 +00:00