morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3593] [Feature] Capture value-free SSE and NDJSON stream shapes in passive page inspection #1333

New issue

Open

opened 2026-10-16 21:04:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:04:23 +00:00

Owner

Originally created by @Y0UW1SH on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3593

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Process capture or runtime observation

Current limitation

Analyst question: "What events does this chat/LLM endpoint stream, and with what payload structure, without copying the conversation into my context?"

Source-derived (commit 1579b68):

inspect_web_page reads response bodies only with include_json_body_shapes, only for application/json or +json (isJsonMediaType, responseBodyRequests), and only after the window ends. text/event-stream and NDJSON get no shape.
CDP ingestion handles request, response, loadingFinished and WebSocket events, but not Network.eventSourceMessageReceived or Network.dataReceived. WebSocket frames already have a value-free shape option.
capture_browser_scenario keeps raw response bytes (values included) for completed responses; in-flight streams get unavailable bodies (docs).

So an agent chooses between no structure and the raw conversation bytes. This is not the same as #3042 / #3044 (merged): static analysis now reports literal new EventSource(url) URLs as endpoints. That is static URL discovery; it says where a stream is requested, not what the stream carries at runtime.

Use cases

LLM chat interfaces that stream completions over SSE or NDJSON fetch responses; notification or dashboard feeds using EventSource, paired with the static call:EventSource endpoint to confirm which URL emits which event names.

Desired outcome and success criteria

With an opt-in flag (proposed include_event_stream_shapes), each admitted stream request carries names, counts and structure only:

"event_stream": { "status": "included", "format": "sse", "sources": ["completed-body"],
  "events_dispatched": 212, "trailing_partial_event": false,
  "events": [
    { "name": "message", "count": 211, "id_present": 0, "retry_present": 0,
      "data": { "json": 210, "text": 1, "json_shape": { "root_type": "object", "properties": [
        { "path": [{ "kind": "property", "name": "choices" }], "types": ["array"], "observations": 210 } ] } } },
    { "name": "done", "count": 1, "data": { "json": 0, "text": 1, "json_shape": null } } ] }

No data, id or retry value substring appears in CLI or MCP output for a fixture with known values.
A fetch() stream still open at window end stays unavailable (response-unfinished); EventSource messages and completed bodies are reported per source, never summed.
With the flag off, output is unchanged; an exhausted representation budget marks the section truncated.

Expected interaction with targets

Launch or attach to a process, Contact network services

Target and scope details (optional)

Both selected effects are existing inspect_web_page behavior (attaching to a selected CDP target through the loopback endpoint). The proposal adds no attach, no new CDP command beyond the existing Network.getResponseBody, and no requests to target servers.

Non-goals: no values, hashes or prefixes of payloads; no reconnection or replay; no Network.streamResourceContent (experimental) in v1, so open fetch() streams stay unavailable; no HAR facet (separate follow-up); no compare_web_captures support.

Scope question for maintainers: is reading text/event-stream and NDJSON response bodies under an opt-in (parsed in memory, values discarded) within the existing passive inspect_web_page boundary, or does it need its own decision? Related: recording the flag in Evidence parameters would change Evidence IDs for default calls; record it only when true?

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Proposed, not run: extend scripts/fixtures/browser-network-site.mjs (it already serves an open /stream) with a completed SSE route (names, ids, retry, comments, multi-line data, CR-only line ends), an NDJSON route and an EventSource client; assert names, counts, shapes and value absence in the real-browser lane. Unknowns: whether getResponseBody returns bodies for EventSource requests, and Chrome version variance.

Alternatives considered

include_json_body_shapes covers JSON media types only; static EventSource endpoints (#3044) give the URL only.
capture_browser_scenario content or HAR response.content.text: raw values in context; open streams unavailable.

Possible direction

A proposal, not a requirement:

A pure parser following the WHATWG "event stream interpretation" rules, plus NDJSON and RFC 7464 JSON text sequences.
Merge data shapes per event name with the existing src/domain/jsonShape.ts.
Admit event-stream and NDJSON media types in responseBodyRequests() when the flag is set, reusing the post-window read and allowed-origin filter.
Handle Network.eventSourceMessageReceived for admitted requests (retry_present: null for that source); report per-source completeness.

References

https://github.com/morluto/rea/issues/882 (closed; scope listed ongoing SSE messages), https://github.com/morluto/rea/pull/992, https://github.com/morluto/rea/issues/774
https://github.com/morluto/rea/issues/3042 and https://github.com/morluto/rea/pull/3044 (static EventSource endpoints)
WHATWG server-sent events: https://html.spec.whatwg.org/multipage/server-sent-events.html; RFC 7464: https://www.rfc-editor.org/rfc/rfc7464; CDP Network domain: https://chromedevtools.github.io/devtools-protocol/tot/Network/

Originally created by @Y0UW1SH on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3593 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Process capture or runtime observation ### Current limitation Analyst question: "What events does this chat/LLM endpoint stream, and with what payload structure, without copying the conversation into my context?" Source-derived (commit 1579b68): - `inspect_web_page` reads response bodies only with `include_json_body_shapes`, only for `application/json` or `+json` ([`isJsonMediaType`](https://github.com/morluto/rea/blob/1579b68/src/browser/CdpCaptureEventHelpers.ts#L78), [`responseBodyRequests`](https://github.com/morluto/rea/blob/1579b68/src/browser/CdpCaptureEvents.ts#L186)), and only after the window ends. `text/event-stream` and NDJSON get no shape. - CDP ingestion handles request, response, `loadingFinished` and WebSocket events, but not `Network.eventSourceMessageReceived` or `Network.dataReceived`. WebSocket frames already have a value-free shape option. - `capture_browser_scenario` keeps raw response bytes (values included) for completed responses; in-flight streams get unavailable bodies ([docs](https://github.com/morluto/rea/blob/1579b68/docs/browser-scenario-contract.md#network-evidence)). So an agent chooses between no structure and the raw conversation bytes. This is not the same as #3042 / #3044 (merged): static analysis now reports literal `new EventSource(url)` URLs as endpoints. That is static URL discovery; it says where a stream is requested, not what the stream carries at runtime. ### Use cases LLM chat interfaces that stream completions over SSE or NDJSON `fetch` responses; notification or dashboard feeds using `EventSource`, paired with the static `call:EventSource` endpoint to confirm which URL emits which event names. ### Desired outcome and success criteria With an opt-in flag (proposed `include_event_stream_shapes`), each admitted stream request carries names, counts and structure only: ```json "event_stream": { "status": "included", "format": "sse", "sources": ["completed-body"], "events_dispatched": 212, "trailing_partial_event": false, "events": [ { "name": "message", "count": 211, "id_present": 0, "retry_present": 0, "data": { "json": 210, "text": 1, "json_shape": { "root_type": "object", "properties": [ { "path": [{ "kind": "property", "name": "choices" }], "types": ["array"], "observations": 210 } ] } } }, { "name": "done", "count": 1, "data": { "json": 0, "text": 1, "json_shape": null } } ] } ``` - No `data`, `id` or `retry` value substring appears in CLI or MCP output for a fixture with known values. - A `fetch()` stream still open at window end stays `unavailable` (`response-unfinished`); EventSource messages and completed bodies are reported per source, never summed. - With the flag off, output is unchanged; an exhausted representation budget marks the section truncated. ### Expected interaction with targets Launch or attach to a process, Contact network services ### Target and scope details (optional) Both selected effects are existing `inspect_web_page` behavior (attaching to a selected CDP target through the loopback endpoint). The proposal adds no attach, no new CDP command beyond the existing `Network.getResponseBody`, and no requests to target servers. Non-goals: no values, hashes or prefixes of payloads; no reconnection or replay; no `Network.streamResourceContent` (experimental) in v1, so open `fetch()` streams stay unavailable; no HAR facet (separate follow-up); no `compare_web_captures` support. **Scope question for maintainers:** is reading `text/event-stream` and NDJSON response bodies under an opt-in (parsed in memory, values discarded) within the existing passive `inspect_web_page` boundary, or does it need its own decision? Related: recording the flag in Evidence parameters would change Evidence IDs for default calls; record it only when `true`? ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Proposed, not run: extend `scripts/fixtures/browser-network-site.mjs` (it already serves an open `/stream`) with a completed SSE route (names, ids, `retry`, comments, multi-line data, CR-only line ends), an NDJSON route and an EventSource client; assert names, counts, shapes and value absence in the real-browser lane. Unknowns: whether `getResponseBody` returns bodies for EventSource requests, and Chrome version variance. ### Alternatives considered - `include_json_body_shapes` covers JSON media types only; static EventSource endpoints (#3044) give the URL only. - `capture_browser_scenario` content or HAR `response.content.text`: raw values in context; open streams unavailable. ### Possible direction A proposal, not a requirement: - A pure parser following the WHATWG "event stream interpretation" rules, plus NDJSON and RFC 7464 JSON text sequences. - Merge `data` shapes per event name with the existing `src/domain/jsonShape.ts`. - Admit event-stream and NDJSON media types in `responseBodyRequests()` when the flag is set, reusing the post-window read and allowed-origin filter. - Handle `Network.eventSourceMessageReceived` for admitted requests (`retry_present: null` for that source); report per-source completeness. ### References - https://github.com/morluto/rea/issues/882 (closed; scope listed ongoing SSE messages), https://github.com/morluto/rea/pull/992, https://github.com/morluto/rea/issues/774 - https://github.com/morluto/rea/issues/3042 and https://github.com/morluto/rea/pull/3044 (static EventSource endpoints) - WHATWG server-sent events: https://html.spec.whatwg.org/multipage/server-sent-events.html; RFC 7464: https://www.rfc-editor.org/rfc/rfc7464; CDP Network domain: https://chromedevtools.github.io/devtools-protocol/tot/Network/

bsakaguchi commented

2026-10-16 21:04:25 +00:00

Author

Owner

@Byun-Gwan-Young commented on GitHub (Oct 15, 2026):

Hi @Y0UW1SH, thanks for the detailed proposal. Unless you're planning to
implement it yourself, I'd like to take this.

On current main (c61c108) the source pointers still hold:
responseBodyRequests() in src/browser/CdpCaptureEvents.ts only admits
JSON media types, and Network.eventSourceMessageReceived isn't handled.

Plan for a draft PR:

A pure parser following the WHATWG event-stream interpretation rules,
plus NDJSON and RFC 7464 JSON text sequences, with unit tests for
CR / LF / CRLF line ends, comments, multi-line data, retry, and a
trailing partial event.
Per-event-name data shapes merged through src/domain/jsonShape.ts;
no values, hashes or prefixes are ever emitted.
Opt-in include_event_stream_shapes on inspect_web_page. Event-stream
and NDJSON media types are admitted in responseBodyRequests() only when
it is set, and the flag is recorded in Evidence parameters only when
true, so default Evidence IDs stay unchanged.
Per-source reporting (completed body vs EventSource messages, never
summed); fetch() streams still open at window end stay unavailable.
New SSE / NDJSON / EventSource routes in
scripts/fixtures/browser-network-site.mjs, with value-absence
assertions in the real-browser lane.

@morluto, does this opt-in fit within the existing passive
inspect_web_page boundary? I'll keep the PR in draft until the scope
is confirmed.

@Byun-Gwan-Young commented on GitHub (Oct 15, 2026): Hi @Y0UW1SH, thanks for the detailed proposal. Unless you're planning to implement it yourself, I'd like to take this. On current main (c61c108) the source pointers still hold: `responseBodyRequests()` in `src/browser/CdpCaptureEvents.ts` only admits JSON media types, and `Network.eventSourceMessageReceived` isn't handled. Plan for a draft PR: 1. A pure parser following the WHATWG event-stream interpretation rules, plus NDJSON and RFC 7464 JSON text sequences, with unit tests for CR / LF / CRLF line ends, comments, multi-line data, `retry`, and a trailing partial event. 2. Per-event-name data shapes merged through `src/domain/jsonShape.ts`; no values, hashes or prefixes are ever emitted. 3. Opt-in `include_event_stream_shapes` on `inspect_web_page`. Event-stream and NDJSON media types are admitted in `responseBodyRequests()` only when it is set, and the flag is recorded in Evidence parameters only when `true`, so default Evidence IDs stay unchanged. 4. Per-source reporting (completed body vs EventSource messages, never summed); `fetch()` streams still open at window end stay `unavailable`. 5. New SSE / NDJSON / EventSource routes in `scripts/fixtures/browser-network-site.mjs`, with value-absence assertions in the real-browser lane. @morluto, does this opt-in fit within the existing passive `inspect_web_page` boundary? I'll keep the PR in draft until the scope is confirmed.

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:29 +00:00

[PR #1333] [CLOSED] fix(javascript): reject invalid package exports configurations #2806

bsakaguchi referenced this issue

2026-10-17 21:10:44 +00:00

[PR #1361] [MERGED] fix(javascript): honor module-sync package exports #2827