[GH-ISSUE #6775] [Bug] Pcap summaries silently replace malformed UTF-8 and report complete Evidence #4130

New issue

Closed

opened 2026-10-31 00:47:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6775

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public inspect-pcap-capture CLI returned successful Evidence containing http_request_uris: ["http://example.test/�"] and coverage: "complete" when a disposable simulated TShark emitted byte ff inside the URI string. That byte is invalid UTF-8. No encoding limitation or original output bytes were retained. A control producer emitting an actual UTF-8 replacement character produced the same URI and coverage, so the result cannot distinguish the two inputs.

This is an executed production CLI/command-boundary reproduction with simulated Wireshark tools. Real Wireshark dissection was not executed, and this does not establish how often a real Wireshark installation produces malformed output.

Reproduction

From the repository root with the compiled runtime present, these self-contained fixtures and CLI invocation exercise the same producer output used in the observed run. Everything stays in a disposable /tmp directory; no network or hardware access occurs.

mkdir -p /tmp/rea-pcap-utf8/home /tmp/rea-pcap-utf8/config /tmp/rea-pcap-utf8/cache /tmp/rea-pcap-utf8/tmp
python - <<'PY'
from pathlib import Path
import struct
p=Path('/tmp/rea-pcap-utf8')
p.joinpath('capture.pcap').write_bytes(struct.pack('<IHHIIII',0xa1b2c3d4,2,4,0,0,65535,1)+struct.pack('<IIII',0,0,0,0))
p.joinpath('tshark').write_bytes(b'''#!/bin/sh
if [ "$1" = "--version" ]; then printf 'TShark (Wireshark) 4.6.6-stub\\n'; exit 0; fi
printf '[{"_source":{"layers":{"frame.number":["1"],"frame.protocols":["eth:ip:tcp:http"],"http.request.full_uri":["http://example.test/'
printf '\\377'
printf '"]}}}]'
''')
p.joinpath('capinfos').write_text('''#!/bin/sh
if [ "$1" = "--version" ]; then printf 'Capinfos (Wireshark) 4.6.6-stub\\n'; exit 0; fi
printf 'File type: pcap\\nFile encapsulation: Ethernet\\nNumber of packets: 1\\n'
''')
for name in ['tshark','capinfos']:p.joinpath(name).chmod(0o755)
PY
HOME=/tmp/rea-pcap-utf8/home XDG_CONFIG_HOME=/tmp/rea-pcap-utf8/config XDG_CACHE_HOME=/tmp/rea-pcap-utf8/cache TMPDIR=/tmp/rea-pcap-utf8/tmp REA_TSHARK_COMMAND=/tmp/rea-pcap-utf8/tshark REA_CAPINFOS_COMMAND=/tmp/rea-pcap-utf8/capinfos node scripts/rea.mjs inspect-pcap-capture /tmp/rea-pcap-utf8/capture.pcap --json

The capture is a 40-byte pcap header plus one zero-length packet. The simulated tools deliberately return the selected-field records above rather than dissecting that packet. Both tools pass REA's version and executable identity checks and run through the actual owned-process and Linux resource-limiter boundary.

To construct the control used in the investigation, replace printf '\377' in the generated tshark script with printf '\357\277\275', save it as tshark-literal, make it executable, and repeat the CLI invocation with REA_TSHARK_COMMAND pointing to tshark-literal. One malformed-output run and one literal-character control were executed; neither used a mocked process launcher.

Expected behavior

Reject malformed producer encoding as unreadable output, retaining an explicitly marked diagnostic rather than reporting the replacement-decoded URI as a literal observed value. Preserve a valid literal U+FFFD character as legitimate text.

The pcap guide describes literal field values as producer observations and URI retention without rewriting other bytes. It also states that malformed output does not return partial success.

Impact and scope

Observed once for malformed stdout at the public CLI boundary. An invalid byte becomes an invented replacement character in successful historical-reference Evidence with complete coverage and no encoding warning. This loses the distinction between malformed producer output and a valid literal replacement character. Real Wireshark frequency, MCP execution, and other pcap streams were not tested. The source-derived affected boundary is shared by capinfos and TShark. No security impact is claimed.

REA version

6.7.0; source revision 98ca6c82705bf8c44f814620636852f938d827ca.

Runtime and platform

Node.js v26.12.1; Linux x64. Disposable /bin/sh Wireshark simulators with 4.6.6-stub banners and the real Linux prlimit; no installed Wireshark was used.

Target and artifact context

Self-created inert 40-byte pcap with one zero-length packet; one simulated HTTP URI field. No user configuration, live capture, external service, or hardware.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

A strict UTF-8 decode of the actual simulated TShark stdout failed; the captured byte stream contained exactly one ff byte. The CLI's projected result and control were:

producer_byte_ff_count=1
producer_utf8=invalid
result {"uris":["http://example.test/�"],"coverage":"complete","raw_result":null,"limitations":[]}
control {"uris":["http://example.test/�"],"coverage":"complete","raw_result":null,"limitations":[]}

Here limitations is the subset of returned limitations mentioning UTF, replacement, or encoding; that subset was empty. The full CLI output contained only the normal capture and resource limitations, and stderr was empty.

Source-derived cause: the process snapshot decoder correctly records invalidUtf8: true, but runPcapCommand returns only the replacement-decoded stdout/stderr strings. The capture path then parses that repaired string as JSON. This describes the observed adapter failure; it does not claim real Wireshark emitted those bytes.

Typed outcome or recovery expected

An encoding-aware unreadable_output refusal with the affected stream identified. A literal U+FFFD control should remain successful.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The actual production CLI silently changes malformed output into complete successful Evidence, losing a distinction relevant to literal field provenance. Real-tool incidence remains unmeasured.
Workaround status: unverified
Workaround basis: Independently validating the selected producer's byte output could detect this condition, but an end-user recovery workflow was not exercised.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6775 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The public `inspect-pcap-capture` CLI returned successful Evidence containing `http_request_uris: ["http://example.test/�"]` and `coverage: "complete"` when a disposable simulated TShark emitted byte `ff` inside the URI string. That byte is invalid UTF-8. No encoding limitation or original output bytes were retained. A control producer emitting an actual UTF-8 replacement character produced the same URI and coverage, so the result cannot distinguish the two inputs. This is an executed production CLI/command-boundary reproduction with simulated Wireshark tools. Real Wireshark dissection was not executed, and this does not establish how often a real Wireshark installation produces malformed output. ## Reproduction From the repository root with the compiled runtime present, these self-contained fixtures and CLI invocation exercise the same producer output used in the observed run. Everything stays in a disposable `/tmp` directory; no network or hardware access occurs. ```sh mkdir -p /tmp/rea-pcap-utf8/home /tmp/rea-pcap-utf8/config /tmp/rea-pcap-utf8/cache /tmp/rea-pcap-utf8/tmp python - <<'PY' from pathlib import Path import struct p=Path('/tmp/rea-pcap-utf8') p.joinpath('capture.pcap').write_bytes(struct.pack('<IHHIIII',0xa1b2c3d4,2,4,0,0,65535,1)+struct.pack('<IIII',0,0,0,0)) p.joinpath('tshark').write_bytes(b'''#!/bin/sh if [ "$1" = "--version" ]; then printf 'TShark (Wireshark) 4.6.6-stub\\n'; exit 0; fi printf '[{"_source":{"layers":{"frame.number":["1"],"frame.protocols":["eth:ip:tcp:http"],"http.request.full_uri":["http://example.test/' printf '\\377' printf '"]}}}]' ''') p.joinpath('capinfos').write_text('''#!/bin/sh if [ "$1" = "--version" ]; then printf 'Capinfos (Wireshark) 4.6.6-stub\\n'; exit 0; fi printf 'File type: pcap\\nFile encapsulation: Ethernet\\nNumber of packets: 1\\n' ''') for name in ['tshark','capinfos']:p.joinpath(name).chmod(0o755) PY HOME=/tmp/rea-pcap-utf8/home XDG_CONFIG_HOME=/tmp/rea-pcap-utf8/config XDG_CACHE_HOME=/tmp/rea-pcap-utf8/cache TMPDIR=/tmp/rea-pcap-utf8/tmp REA_TSHARK_COMMAND=/tmp/rea-pcap-utf8/tshark REA_CAPINFOS_COMMAND=/tmp/rea-pcap-utf8/capinfos node scripts/rea.mjs inspect-pcap-capture /tmp/rea-pcap-utf8/capture.pcap --json ``` The capture is a 40-byte pcap header plus one zero-length packet. The simulated tools deliberately return the selected-field records above rather than dissecting that packet. Both tools pass REA's version and executable identity checks and run through the actual owned-process and Linux resource-limiter boundary. To construct the control used in the investigation, replace `printf '\377'` in the generated `tshark` script with `printf '\357\277\275'`, save it as `tshark-literal`, make it executable, and repeat the CLI invocation with `REA_TSHARK_COMMAND` pointing to `tshark-literal`. One malformed-output run and one literal-character control were executed; neither used a mocked process launcher. ## Expected behavior Reject malformed producer encoding as unreadable output, retaining an explicitly marked diagnostic rather than reporting the replacement-decoded URI as a literal observed value. Preserve a valid literal U+FFFD character as legitimate text. The [pcap guide](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/docs/pcap-analysis.md#L70-L96) describes literal field values as producer observations and URI retention without rewriting other bytes. It also states that [malformed output does not return partial success](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/docs/pcap-analysis.md#L114-L117). ## Impact and scope Observed once for malformed stdout at the public CLI boundary. An invalid byte becomes an invented replacement character in successful historical-reference Evidence with complete coverage and no encoding warning. This loses the distinction between malformed producer output and a valid literal replacement character. Real Wireshark frequency, MCP execution, and other pcap streams were not tested. The source-derived affected boundary is shared by capinfos and TShark. No security impact is claimed. ## REA version 6.7.0; source revision `98ca6c82705bf8c44f814620636852f938d827ca`. ## Runtime and platform Node.js v26.12.1; Linux x64. Disposable `/bin/sh` Wireshark simulators with 4.6.6-stub banners and the real Linux `prlimit`; no installed Wireshark was used. ## Target and artifact context Self-created inert 40-byte pcap with one zero-length packet; one simulated HTTP URI field. No user configuration, live capture, external service, or hardware. ## Interface CLI ## Earliest failure stage Transport or result projection ## Relevant output or evidence A strict UTF-8 decode of the actual simulated TShark stdout failed; the captured byte stream contained exactly one `ff` byte. The CLI's projected result and control were: ```text producer_byte_ff_count=1 producer_utf8=invalid result {"uris":["http://example.test/�"],"coverage":"complete","raw_result":null,"limitations":[]} control {"uris":["http://example.test/�"],"coverage":"complete","raw_result":null,"limitations":[]} ``` Here `limitations` is the subset of returned limitations mentioning UTF, replacement, or encoding; that subset was empty. The full CLI output contained only the normal capture and resource limitations, and stderr was empty. Source-derived cause: the [process snapshot decoder](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/process/ProviderProcess.ts#L875-L890) correctly records `invalidUtf8: true`, but [runPcapCommand](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/pcap/PcapCommand.ts#L139-L142) returns only the replacement-decoded stdout/stderr strings. The [capture path](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/pcap/PcapProvider.ts#L946-L950) then parses that repaired string as JSON. This describes the observed adapter failure; it does not claim real Wireshark emitted those bytes. ## Typed outcome or recovery expected An encoding-aware `unreadable_output` refusal with the affected stream identified. A literal U+FFFD control should remain successful. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The actual production CLI silently changes malformed output into complete successful Evidence, losing a distinction relevant to literal field provenance. Real-tool incidence remains unmeasured. - Workaround status: unverified - Workaround basis: Independently validating the selected producer's byte output could detect this condition, but an end-user recovery workflow was not exercised.

bsakaguchi closed this issue

2026-10-31 00:47:02 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:39 +00:00

[PR #4130] [CLOSED] fix(batch): report a provider's null pseudocode as unavailable, not unreadable #6075