morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3594] [Feature] Decrypt TLS in recorded packet captures with a caller-supplied key log #1334

New issue

Open

opened 2026-10-16 21:04:23 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:04:23 +00:00

Owner

Originally created by @Y0UW1SH on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3594

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

Analyst question: "My Electron app talked to its backend over HTTPS while I recorded a pcapng, and I launched it with --ssl-key-log-file. Which requests, paths and response codes are in that recording?"

Source-derived (commit 1579b68):

inspect_packet_capture runs TShark on a private read-only copy with an empty WIRESHARK_CONFIG_DIR. Its argv has no -o preference and the tool has no key-log input (TsharkCaptureProvider.ts, docs). TLS application data therefore stays encrypted, and http.* / http2.* / http3.* fields are expected to be empty for HTTPS (not run).
inspect_pcap_capture reports TLS SNI; its guide says recorded fields "do not establish decrypted payloads" (docs). The qlog decoder opens no key log.
Seeing HTTPS content today needs a MITM recording (HAR or mitmproxy), which changes the client's trust store and network path.

Use cases

Chromium/Electron (--ssl-key-log-file), Firefox and curl (SSLKEYLOGFILE) or Node (--tls-keylog) clients that the analyst runs themselves while recording with tcpdump or Wireshark.
Recordings where inserting a proxy would change the network path being studied.

Desired outcome and success criteria

Given path plus a new tls_key_log_path, the same selected fields (e.g. http2.headers.path) return decrypted values with original frame numbers, and the result describes the key log without its contents:

"tls_key_log": { "path": "/captures/electron.keys", "sha256": "<64 hex>", "bytes": 2310,
  "lines": { "total": 15, "comment_or_blank": 1, "unrecognized": 0 },
  "labels": { "CLIENT_TRAFFIC_SECRET_0": 3, "SERVER_TRAFFIC_SECRET_0": 3 },
  "client_randoms": { "in_key_log": 3, "client_hellos_in_read_scope": 4, "matched": 3, "client_hellos_without_keys": 1 },
  "secret_values": "not-returned" },
"redactions": [{ "frame": "41", "field": "http2.headers.authorization", "reason": "explicit-sensitive-value" }]

Without tls_key_log_path, argv and output are unchanged.
With a key log, known plaintext appears for sessions whose client random is logged; sessions without keys are counted, and a match is not reported as proof of decryption.
No key-log secret value appears in CLI output, MCP responses, Evidence or diagnostics; malformed lines are counted, never echoed.

Expected interaction with targets

Read existing files or captured data, Launch or attach to a process, Not sure yet

Target and scope details (optional)

"Launch" is the existing owned TShark child, not the target. Non-goals: no live capture (REA sets no SSLKEYLOGFILE on any process), no MITM, trust-store change or replay, no arbitrary -o preferences, no HAR or api-surface conversion, no change to inspect_pcap_capture. Linux only, as today; Windows admission is proposed separately in #3115.

"Not sure yet": Wireshark can also use secrets embedded in pcapng Decryption Secrets Blocks. Whether the current tool already decrypts such files without reporting it is unverified.

Scope question for maintainers: is a caller-supplied TLS key log (secret material written by the caller's own client) an acceptable inspect_packet_capture input? If so, should known credential fields in decrypted output (e.g. http.authorization, http.cookie, http2.headers.authorization) be excluded by default even when explicitly selected, as the HAR lane redacts transport credentials?

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Proposed, not run: in the real TShark lane (scripts/verify-packet-capture.mjs), generate TLS 1.2 and 1.3 sessions between Node tls endpoints with an ephemeral self-signed certificate, write the key log from Node's keylog event, and wrap the bytes in synthetic frames as the existing generated PCAPs do. Check: no HTTP fields without the key log; known plaintext with it; counts with one extra key line and one session missing keys; CLI/MCP parity; no secret hex value in any output. Add a pcapng with an injected secrets block (editcap --inject-secrets) to settle the open question above.

Alternatives considered

MITM recording (HAR or mitmproxy): changes the client's trust store and network path.
Running Wireshark or TShark by hand with -o tls.keylog_file:…: outside REA Evidence, with no secret exclusion or provenance.
inspect_pcap_capture and qlog inspection: SNI and QUIC events only, no decrypted HTTP.

Possible direction

A proposal, not a requirement:

Optional absolute tls_key_log_path and sensitive_values inputs on inspect_packet_capture.
Copy the key log read-only into the existing private workspace, pass -o tls.keylog_file:<copy>, and recheck its digest after the run, as is done for the capture copy.
A key-log parser (proposed src/domain/captures/tlsKeyLog.ts) that counts NSS labels and client randoms and never echoes tokens; a second bounded pass on ClientHello tls.handshake.random for match counts.
Apply an internal secret-exclusion set plus caller sensitive_values to fields, stderr and errors, listing removals in redactions.
Rebase on #3403 / #3407 if TShark selection gets one owner first.

References

https://github.com/morluto/rea/pull/2318 (inspect_packet_capture; replaced https://github.com/morluto/rea/pull/2130), https://github.com/morluto/rea/pull/2340, https://github.com/morluto/rea/pull/1997, https://github.com/morluto/rea/pull/2277
https://github.com/morluto/rea/issues/3115, https://github.com/morluto/rea/issues/3403, https://github.com/morluto/rea/pull/3407
SSLKEYLOGFILE format: https://datatracker.ietf.org/doc/draft-ietf-tls-keylogfile/; Wireshark TLS decryption: https://wiki.wireshark.org/TLS; TShark manual: https://www.wireshark.org/docs/man-pages/tshark.html

Originally created by @Y0UW1SH on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3594 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation Analyst question: "My Electron app talked to its backend over HTTPS while I recorded a pcapng, and I launched it with `--ssl-key-log-file`. Which requests, paths and response codes are in that recording?" Source-derived (commit 1579b68): - `inspect_packet_capture` runs TShark on a private read-only copy with an empty `WIRESHARK_CONFIG_DIR`. Its argv has no `-o` preference and the tool has no key-log input ([`TsharkCaptureProvider.ts`](https://github.com/morluto/rea/blob/1579b68/src/captures/TsharkCaptureProvider.ts#L308), [docs](https://github.com/morluto/rea/blob/1579b68/docs/packet-captures.md)). TLS application data therefore stays encrypted, and `http.*` / `http2.*` / `http3.*` fields are expected to be empty for HTTPS (not run). - `inspect_pcap_capture` reports TLS SNI; its guide says recorded fields "do not establish decrypted payloads" ([docs](https://github.com/morluto/rea/blob/1579b68/docs/pcap-analysis.md)). The qlog decoder opens no key log. - Seeing HTTPS content today needs a MITM recording (HAR or mitmproxy), which changes the client's trust store and network path. ### Use cases - Chromium/Electron (`--ssl-key-log-file`), Firefox and curl (`SSLKEYLOGFILE`) or Node (`--tls-keylog`) clients that the analyst runs themselves while recording with tcpdump or Wireshark. - Recordings where inserting a proxy would change the network path being studied. ### Desired outcome and success criteria Given `path` plus a new `tls_key_log_path`, the same selected fields (e.g. `http2.headers.path`) return decrypted values with original frame numbers, and the result describes the key log without its contents: ```json "tls_key_log": { "path": "/captures/electron.keys", "sha256": "<64 hex>", "bytes": 2310, "lines": { "total": 15, "comment_or_blank": 1, "unrecognized": 0 }, "labels": { "CLIENT_TRAFFIC_SECRET_0": 3, "SERVER_TRAFFIC_SECRET_0": 3 }, "client_randoms": { "in_key_log": 3, "client_hellos_in_read_scope": 4, "matched": 3, "client_hellos_without_keys": 1 }, "secret_values": "not-returned" }, "redactions": [{ "frame": "41", "field": "http2.headers.authorization", "reason": "explicit-sensitive-value" }] ``` - Without `tls_key_log_path`, argv and output are unchanged. - With a key log, known plaintext appears for sessions whose client random is logged; sessions without keys are counted, and a match is not reported as proof of decryption. - No key-log secret value appears in CLI output, MCP responses, Evidence or diagnostics; malformed lines are counted, never echoed. ### Expected interaction with targets Read existing files or captured data, Launch or attach to a process, Not sure yet ### Target and scope details (optional) "Launch" is the existing owned TShark child, not the target. Non-goals: no live capture (REA sets no `SSLKEYLOGFILE` on any process), no MITM, trust-store change or replay, no arbitrary `-o` preferences, no HAR or `api-surface` conversion, no change to `inspect_pcap_capture`. Linux only, as today; Windows admission is proposed separately in #3115. "Not sure yet": Wireshark can also use secrets embedded in pcapng Decryption Secrets Blocks. Whether the current tool already decrypts such files without reporting it is unverified. **Scope question for maintainers:** is a caller-supplied TLS key log (secret material written by the caller's own client) an acceptable `inspect_packet_capture` input? If so, should known credential fields in decrypted output (e.g. `http.authorization`, `http.cookie`, `http2.headers.authorization`) be excluded by default even when explicitly selected, as the HAR lane redacts transport credentials? ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Proposed, not run: in the real TShark lane (`scripts/verify-packet-capture.mjs`), generate TLS 1.2 and 1.3 sessions between Node `tls` endpoints with an ephemeral self-signed certificate, write the key log from Node's `keylog` event, and wrap the bytes in synthetic frames as the existing generated PCAPs do. Check: no HTTP fields without the key log; known plaintext with it; counts with one extra key line and one session missing keys; CLI/MCP parity; no secret hex value in any output. Add a pcapng with an injected secrets block (`editcap --inject-secrets`) to settle the open question above. ### Alternatives considered - MITM recording (HAR or mitmproxy): changes the client's trust store and network path. - Running Wireshark or TShark by hand with `-o tls.keylog_file:…`: outside REA Evidence, with no secret exclusion or provenance. - `inspect_pcap_capture` and qlog inspection: SNI and QUIC events only, no decrypted HTTP. ### Possible direction A proposal, not a requirement: - Optional absolute `tls_key_log_path` and `sensitive_values` inputs on `inspect_packet_capture`. - Copy the key log read-only into the existing private workspace, pass `-o tls.keylog_file:<copy>`, and recheck its digest after the run, as is done for the capture copy. - A key-log parser (proposed `src/domain/captures/tlsKeyLog.ts`) that counts NSS labels and client randoms and never echoes tokens; a second bounded pass on ClientHello `tls.handshake.random` for match counts. - Apply an internal secret-exclusion set plus caller `sensitive_values` to fields, stderr and errors, listing removals in `redactions`. - Rebase on #3403 / #3407 if TShark selection gets one owner first. ### References - https://github.com/morluto/rea/pull/2318 (`inspect_packet_capture`; replaced https://github.com/morluto/rea/pull/2130), https://github.com/morluto/rea/pull/2340, https://github.com/morluto/rea/pull/1997, https://github.com/morluto/rea/pull/2277 - https://github.com/morluto/rea/issues/3115, https://github.com/morluto/rea/issues/3403, https://github.com/morluto/rea/pull/3407 - SSLKEYLOGFILE format: https://datatracker.ietf.org/doc/draft-ietf-tls-keylogfile/; Wireshark TLS decryption: https://wiki.wireshark.org/TLS; TShark manual: https://www.wireshark.org/docs/man-pages/tshark.html

bsakaguchi commented

2026-10-16 21:04:25 +00:00

Author

Owner

@Y0UW1SH commented on GitHub (Oct 15, 2026):

Scope decision request for the implementation in #3643

#3643 implements this proposal. Review there found no blocking code issue, but integration is on hold until a maintainer records a scope decision here that covers its final behavior (per CONTRIBUTING's fail-closed rule). #3643 is back in draft until then.

The PR takes the conservative option at each point below. A reply such as "accepted as implemented in #3643", or which points to change or drop, would be enough to proceed.

Secret input. Accept a caller-supplied TLS key log (tls_key_log_path) as an optional inspect_packet_capture input? REA copies it into the run's private workspace, passes only that copy to TShark, and deletes the copy right after the run. Output reports counts only (lines, labels, client randoms), never secret values or line text.
TShark output text. For decrypting runs, withhold all TShark stdout and stderr text, keeping only byte counts? The alternative is to return a bounded subset, such as filter-syntax errors.
Credential fields. Refuse display filters that name known credential fields, and remove those fields from output even when the caller selects them, listing each removal under redactions?
Known gap. Credentials can still appear in generic containers: http.request.line, http2.header.value, HTTP/3 header values, NTLMSSP and Kerberos subfields, and raw payload bytes. #3643 does not remove these and documents that it doesn't. Which do you prefer?
- (a) Accept this as a documented limitation.
- (b) Widen the exclusion in #3643.
- (c) Refuse those fields whenever a key log is used.

Not asked here: whether plaintext captures should get the same credential exclusion. That affects the tool without a key log, so it would be a separate issue.

@Y0UW1SH commented on GitHub (Oct 15, 2026): **Scope decision request for the implementation in #3643** #3643 implements this proposal. Review there found no blocking code issue, but integration is on hold until a maintainer records a scope decision here that covers its final behavior (per CONTRIBUTING's fail-closed rule). #3643 is back in draft until then. The PR takes the conservative option at each point below. A reply such as "accepted as implemented in #3643", or which points to change or drop, would be enough to proceed. 1. **Secret input.** Accept a caller-supplied TLS key log (`tls_key_log_path`) as an optional `inspect_packet_capture` input? REA copies it into the run's private workspace, passes only that copy to TShark, and deletes the copy right after the run. Output reports counts only (lines, labels, client randoms), never secret values or line text. 2. **TShark output text.** For decrypting runs, withhold all TShark stdout and stderr text, keeping only byte counts? The alternative is to return a bounded subset, such as filter-syntax errors. 3. **Credential fields.** Refuse display filters that name known credential fields, and remove those fields from output even when the caller selects them, listing each removal under `redactions`? 4. **Known gap.** Credentials can still appear in generic containers: `http.request.line`, `http2.header.value`, HTTP/3 header values, NTLMSSP and Kerberos subfields, and raw payload bytes. #3643 does not remove these and documents that it doesn't. Which do you prefer? - (a) Accept this as a documented limitation. - (b) Widen the exclusion in #3643. - (c) Refuse those fields whenever a key log is used. Not asked here: whether plaintext captures should get the same credential exclusion. That affects the tool without a key log, so it would be a separate issue.

bsakaguchi referenced this issue

2026-10-17 21:10:29 +00:00

[PR #1334] [MERGED] docs: celebrate 40,000 GitHub stars 🎉 #2805

bsakaguchi referenced this issue

2026-10-17 21:10:30 +00:00

[PR #1340] [MERGED] chore(release): sync 6.2.0 checkpoint into main #2808

bsakaguchi referenced this issue

2026-10-17 21:10:33 +00:00

[PR #1344] [MERGED] fix(ci): scope verification and reuse native runners #2810

bsakaguchi referenced this issue

2026-10-17 21:10:37 +00:00

[PR #1349] [MERGED] docs: celebrate 50,000 GitHub stars 🎉 #2815

bsakaguchi commented

2026-10-31 00:30:40 +00:00

Author

Owner

@Y0UW1SH commented on GitHub (Oct 17, 2026):

@morluto @baron: please make the explicit accept/reject/scope decision for #3643 required by the changes-requested review. The final proposed boundary is now narrower at 6fd11bd:

Accept only a key log written by the caller's own TLS client, passed to TShark through an owned private read-only copy with integrity checks, deletion and retained cleanup retry authority.
Return key-log identity/counts without key-log values; withhold decrypting producer diagnostics throughout returned errors. Keep completed key-log read buffers and readable file owners private when closing or retrying cleanup fails.
Remove named authorization/cookie fields and URI/authority fields containing structural URL userinfo, recording removals.
Admit only the guide's explicit 30 packet/HTTP metadata fields; refuse generic headers, decoded authentication, raw/payload and unreviewed fields, plus every non-null display filter. Require method context for HTTP request URIs and refuse uncertain flattened contexts.

Selected metadata may still contain application data or caller-embedded secrets; this does not claim arbitrary content-based DLP. Existing no-key-log/plaintext handling, pcapng embedded secrets, and MCP Evidence retention remain explicit policy/verification limits.

The latest file-close privacy fix passed 141 focused tests and static checks, with all three new regressions proven to fail on the prior code. All 43 current-commit check runs passed, including real TShark (137 tests passed, one skipped; all 31 verifier cases). Greptile added no comments; Codex completed, with only the maintainer scope thread unresolved. The branch includes upstream's WASM test-isolation repair through main bb212ff.

The human's initial readiness declaration permits maintainer review. The material-scope checkbox remains unchecked, and this work will not merge or release the capability before a maintainer decision covering the final diff. Please state whether this bounded form is accepted, rejected, or needs specific further constraints.

@Y0UW1SH commented on GitHub (Oct 17, 2026): @morluto @baron: please make the explicit accept/reject/scope decision for #3643 required by the changes-requested review. The final proposed boundary is now narrower at [`6fd11bd`](https://github.com/morluto/rea/pull/3643/commits/6fd11bdd0258b5c6e938ba863ceef6bdb3111308): 1. Accept only a key log written by the caller's own TLS client, passed to TShark through an owned private read-only copy with integrity checks, deletion and retained cleanup retry authority. 2. Return key-log identity/counts without key-log values; withhold decrypting producer diagnostics throughout returned errors. Keep completed key-log read buffers and readable file owners private when closing or retrying cleanup fails. 3. Remove named authorization/cookie fields and URI/authority fields containing structural URL userinfo, recording removals. 4. Admit only the guide's explicit 30 packet/HTTP metadata fields; refuse generic headers, decoded authentication, raw/payload and unreviewed fields, plus every non-null display filter. Require method context for HTTP request URIs and refuse uncertain flattened contexts. Selected metadata may still contain application data or caller-embedded secrets; this does not claim arbitrary content-based DLP. Existing no-key-log/plaintext handling, pcapng embedded secrets, and MCP Evidence retention remain explicit policy/verification limits. The latest file-close privacy fix passed 141 focused tests and static checks, with all three new regressions proven to fail on the prior code. All 43 current-commit check runs passed, including real TShark (137 tests passed, one skipped; all 31 verifier cases). Greptile added no comments; Codex completed, with only the maintainer scope thread unresolved. The branch includes upstream's WASM test-isolation repair through main `bb212ff`. The human's initial readiness declaration permits maintainer review. The material-scope checkbox remains unchecked, and this work will not merge or release the capability before a maintainer decision covering the final diff. Please state whether this bounded form is accepted, rejected, or needs specific further constraints.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1334

No description provided.

Rows
Columns