morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3133] test(perf): measure supported qlog summaries and retained event views #1093

New issue

Open

opened 2026-10-15 18:31:01 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:31:01 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3133

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. This is recorded qlog measurement, not a new live QUIC acquisition or cross-capture correlation feature.

Analyst question and outcome

For a trace-scoped QUIC event/timing question on supported recorded qlog JSON, do existing summary and event views reduce model input and useful-answer time?

Existing evidence and decision boundary

The earlier continuation identifies supported aioquic/qlog JSON, retained capture views, trace-scoped interval unions and endpoint-declared facts. It did not measure qlog model/task costs. Recheck current support rather than silently accepting another qlog layout or relabeling endpoint-declared loss as observed network loss.

Bounded work

Pin the actual current contract and parser support, then choose two public recordings from supported upstream layouts: a compact control and a larger trace. Record producer/version/layout, trace identity and digest; unavailable provenance remains unknown.
Fix a question about specific events, interval summaries and declared endpoint facts. Independently verify the selected raw JSON events and interval calculation. Trace-local clocks, missing events and incomplete observations must stay distinct.
Compare complete versus existing trace/event/summary selections with the same requested facts. Do not infer recovered HTTP bodies, deployed network truth or PCAP correlation from qlog metadata. Preserve event order, source positions, nullable fields, parent links and interval/coverage semantics.
Repeat two SDK pairs per recording and two actual Codex pairs for a representative equivalent workflow. Account for parsing, retained reads, catalog/discovery, extra RPCs and retries. Keep unsupported layouts as explicit unavailable outcomes.

Acceptance

Supported layout/version, trace identity, source digests and fixed questions/oracles are pinned.
Selected facts match independent raw-event/interval checks while endpoint declarations and inferred claims remain separate.
Repeated real CLI/MCP and model comparisons report equivalent completed work, token/time/resource costs and all failures.
Record whether existing controls are sufficient, whether a narrowly scoped change is justified, or whether no gain is supported.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3133 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. This is recorded qlog measurement, not a new live QUIC acquisition or cross-capture correlation feature. ## Analyst question and outcome For a trace-scoped QUIC event/timing question on supported recorded qlog JSON, do existing summary and event views reduce model input and useful-answer time? ## Existing evidence and decision boundary The earlier continuation identifies supported aioquic/qlog JSON, retained capture views, trace-scoped interval unions and endpoint-declared facts. It did not measure qlog model/task costs. Recheck current support rather than silently accepting another qlog layout or relabeling endpoint-declared loss as observed network loss. ## Bounded work 1. Pin the actual current contract and parser support, then choose two public recordings from supported upstream layouts: a compact control and a larger trace. Record producer/version/layout, trace identity and digest; unavailable provenance remains unknown. 2. Fix a question about specific events, interval summaries and declared endpoint facts. Independently verify the selected raw JSON events and interval calculation. Trace-local clocks, missing events and incomplete observations must stay distinct. 3. Compare complete versus existing trace/event/summary selections with the same requested facts. Do not infer recovered HTTP bodies, deployed network truth or PCAP correlation from qlog metadata. Preserve event order, source positions, nullable fields, parent links and interval/coverage semantics. 4. Repeat two SDK pairs per recording and two actual Codex pairs for a representative equivalent workflow. Account for parsing, retained reads, catalog/discovery, extra RPCs and retries. Keep unsupported layouts as explicit unavailable outcomes. ## Acceptance - [ ] Supported layout/version, trace identity, source digests and fixed questions/oracles are pinned. - [ ] Selected facts match independent raw-event/interval checks while endpoint declarations and inferred claims remain separate. - [ ] Repeated real CLI/MCP and model comparisons report equivalent completed work, token/time/resource costs and all failures. - [ ] Record whether existing controls are sufficient, whether a narrowly scoped change is justified, or whether no gain is supported. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi added the

labels

2026-10-15 18:31:01 +00:00

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:08:32 +00:00

[PR #1093] [MERGED] fix(javascript): preserve original BOM source coordinates #2641

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:55 +00:00

bsakaguchi commented

2026-11-03 23:47:05 +00:00

Author

Owner

@LuisTaffelli commented on GitHub (Oct 17, 2026):

Checkpoint for #3133, frozen at 9d6651e5b5296da31a88042a061bdc6f4e769f10 (Linux x86_64, Node 24.18.0/npm 11.16.0). No production/API/default/cache changes; this is not a request to close the issue.

I pinned two unchanged public recordings from https://quic.aiortc.org/logs: 29,040 bytes/132 events and 5,922,676 bytes/30,753 events. Both have JSON qlog 0.3 object events and report an aioquic server vantage point; the actual producer version remains unknown. Source URLs/digests, trace identities, fixed questions and an independent exact-integer sweep-line oracle are published in the reviewed evidence package.

Eight real SDK workflows (two complete/retained pairs per recording) passed fact, source-coordinate/time-literal, digest, parent-Evidence and SDK-converted schema checks. Both real CLI cases exactly matched SDK Evidence IDs and normalized results. A pinned public qvis draft-01 array-event control was refused as unsupported without conversion.

Complete versus summary plus four retained pages returned 37,005 versus 59,171 SDK JSON bytes for the compact recording, and 1,677,323 versus 58,695 for the larger recording. Median guarded wall time was 20.37 versus 22.17 seconds and 26.34 versus 28.25 seconds respectively. The large-recording delivery reduction is not a parser-speed or model-token result; the compact case gets worse. CPU/aggregate-cgroup peak measurements are included separately.

Four actual Codex tasks requested gpt-5.6-terra/high with fresh profiles and the same opt-in three-tool catalog. Complete/retained/retained/complete input totals were 314,543 / 174,651 / 191,073 / 129,480; uncached input 51,631 / 37,179 / 47,201 / 36,552; task wall 94.80 / 54.70 / 55.92 / 63.35 seconds. Per-response usage sums match final-turn counters. All attempts, schema errors, repeated calls and recovery costs remain counted.

All four final answers match the selected independent values, with appropriate semantic limits. However, one complete task crossed into focused views, retained tasks made extra calls, and saved initial complete-reply client text is not fully JSON-decodable, preventing a full response-envelope citation audit. SDK/CLI originals validate; malformed server output or actual submitted model context is not established. Therefore these are exploratory model observations, not two clean accepted performance pairs or a general token/speed win.

Heavy work ran serially under one CPU, 3-GiB early stop/4-GiB hard/no-swap guards. Filesystem caches were warm/unflushed; no cold cohort was measured. Payload-exit resource observations and cleanup are verified; final whole-scope counters remain unknown. One host-preflight refusal and setup failures are retained. Raw recordings, transcripts and account/configuration data stay private.

Recommendation: keep complete defaults and existing views. The useful continuation is to establish a verifiable complete-reply client receipt, then repeat the same question while retaining failures and actual model choices. No new delivery API or optimization is justified by this checkpoint.

Published text files were checked byte-for-byte against the reviewed local payload. The Gist preserves the original pre-publication review snapshot; no new experiment is claimed by this publication.

@LuisTaffelli commented on GitHub (Oct 17, 2026): Checkpoint for #3133, frozen at `9d6651e5b5296da31a88042a061bdc6f4e769f10` (Linux x86_64, Node 24.18.0/npm 11.16.0). No production/API/default/cache changes; this is not a request to close the issue. I pinned two unchanged public recordings from https://quic.aiortc.org/logs: 29,040 bytes/132 events and 5,922,676 bytes/30,753 events. Both have JSON qlog 0.3 object events and report an aioquic server vantage point; the actual producer version remains unknown. Source URLs/digests, trace identities, fixed questions and an independent exact-integer sweep-line oracle are published in the [reviewed evidence package](https://gist.github.com/LuisTaffelli/0f6638a4d7f13eee5f09b2554b789350). Eight real SDK workflows (two complete/retained pairs per recording) passed fact, source-coordinate/time-literal, digest, parent-Evidence and SDK-converted schema checks. Both real CLI cases exactly matched SDK Evidence IDs and normalized results. A pinned public qvis draft-01 array-event control was refused as unsupported without conversion. Complete versus summary plus four retained pages returned 37,005 versus 59,171 SDK JSON bytes for the compact recording, and 1,677,323 versus 58,695 for the larger recording. Median guarded wall time was 20.37 versus 22.17 seconds and 26.34 versus 28.25 seconds respectively. The large-recording delivery reduction is not a parser-speed or model-token result; the compact case gets worse. CPU/aggregate-cgroup peak measurements are included separately. Four actual Codex tasks requested `gpt-5.6-terra`/high with fresh profiles and the same opt-in three-tool catalog. Complete/retained/retained/complete input totals were 314,543 / 174,651 / 191,073 / 129,480; uncached input 51,631 / 37,179 / 47,201 / 36,552; task wall 94.80 / 54.70 / 55.92 / 63.35 seconds. Per-response usage sums match final-turn counters. All attempts, schema errors, repeated calls and recovery costs remain counted. All four final answers match the selected independent values, with appropriate semantic limits. However, one complete task crossed into focused views, retained tasks made extra calls, and saved initial complete-reply client text is not fully JSON-decodable, preventing a full response-envelope citation audit. SDK/CLI originals validate; malformed server output or actual submitted model context is not established. Therefore these are exploratory model observations, not two clean accepted performance pairs or a general token/speed win. Heavy work ran serially under one CPU, 3-GiB early stop/4-GiB hard/no-swap guards. Filesystem caches were warm/unflushed; no cold cohort was measured. Payload-exit resource observations and cleanup are verified; final whole-scope counters remain unknown. One host-preflight refusal and setup failures are retained. Raw recordings, transcripts and account/configuration data stay private. Recommendation: keep complete defaults and existing views. The useful continuation is to establish a verifiable complete-reply client receipt, then repeat the same question while retaining failures and actual model choices. No new delivery API or optimization is justified by this checkpoint. Published text files were checked byte-for-byte against the reviewed local payload. The Gist preserves the original pre-publication review snapshot; no new experiment is claimed by this publication.

bsakaguchi commented

2026-11-03 23:47:05 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Nov 3, 2026):

TL;DR: The two public recordings from the 17 Oct checkpoint are gone, so I could not rerun it. Both URLs and the logs folder on quic.aiortc.org return 404 as of 3 Nov. The qlog tests on main pass, but that is not the measurement this issue asks for.

Recordings: b171e6b78e186c78.qlog and 6e037439a5f79d47.qlog both return 404. The Internet Archive lists no snapshot for either. I used no other recording. The checkpoint's own notes say a missing source can't be swapped.
Tests: 4 qlog and capture-view test files pass at c3647ade1 (#8824), on Node 24.22.1 in a Fedora 44 VM. That is 109 passed and 1 skipped.
Main: The commit 4e716b060 (#5349) reuses checks on captures that can't change. 365fb03e7 archived experiments/analysis-efficiency/ on 27 Oct, so that PR's write-up is no longer on main.

Not run

The comparison of complete and retained views on a real recording.
The Codex pairs, or any model token or time measure.
verify:web:qlog. It needs captures I don't have.
A full Internet Archive search, since its index was offline when I tried.

Test setup

Box: Fedora 44 VM (4 vCPUs, 8 GB) on my Nobara 44 machine, Linux 7.2.11 in the VM
Node: 24.22.1
Head: c3647ade1 (#8824)

Commands

In the VM, offline, with Node 24 first on PATH:

npm ci --offline                 # exit 0
npm run test:prepare             # exit 0
npm run test:focused -- src/browser/history/QlogCapture.test.ts src/domain/analysisView/webNetworkCaptureView.test.ts src/domain/webNetworkCapture.test.ts tests/acceptance/tools/qlogCapture.test.ts
# exit 0: 4 files, 109 passed, 1 skipped

On my host:

curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/b171e6b78e186c78.qlog   # 404
curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/6e037439a5f79d47.qlog   # 404
curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/                        # 404
curl -sS 'https://archive.org/wayback/available?url=quic.aiortc.org/logs/b171e6b78e186c78.qlog' # archived_snapshots empty
curl -sS 'https://archive.org/wayback/available?url=quic.aiortc.org/logs/6e037439a5f79d47.qlog' # archived_snapshots empty

🤖 Written and tested with Claude Code (Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Nov 3, 2026): **TL;DR:** The two public recordings from the [17 Oct checkpoint](https://github.com/morluto/rea/issues/3133#issuecomment-6236555466) are gone, so I could not rerun it. Both URLs and the `logs` folder on quic.aiortc.org return 404 as of 3 Nov. The qlog tests on main pass, but that is not the measurement this issue asks for. - **Recordings:** `b171e6b78e186c78.qlog` and `6e037439a5f79d47.qlog` both return 404. The Internet Archive lists no snapshot for either. I used no other recording. The checkpoint's own notes say a missing source can't be swapped. - **Tests:** 4 qlog and capture-view test files pass at c3647ade1 (#8824), on Node 24.22.1 in a Fedora 44 VM. That is 109 passed and 1 skipped. - **Main:** The commit 4e716b060 (#5349) reuses checks on captures that can't change. 365fb03e7 archived `experiments/analysis-efficiency/` on 27 Oct, so that PR's write-up is no longer on main. ### Not run - The comparison of complete and retained views on a real recording. - The Codex pairs, or any model token or time measure. - `verify:web:qlog`. It needs captures I don't have. - A full Internet Archive search, since its index was offline when I tried. <details> <summary>Test setup</summary> - Box: Fedora 44 VM (4 vCPUs, 8 GB) on my Nobara 44 machine, Linux 7.2.11 in the VM - Node: 24.22.1 - Head: c3647ade1 (#8824) </details> <details> <summary>Commands</summary> In the VM, offline, with Node 24 first on PATH: ``` npm ci --offline # exit 0 npm run test:prepare # exit 0 npm run test:focused -- src/browser/history/QlogCapture.test.ts src/domain/analysisView/webNetworkCaptureView.test.ts src/domain/webNetworkCapture.test.ts tests/acceptance/tools/qlogCapture.test.ts # exit 0: 4 files, 109 passed, 1 skipped ``` On my host: ``` curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/b171e6b78e186c78.qlog # 404 curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/6e037439a5f79d47.qlog # 404 curl -sS -o /dev/null -w '%{http_code}\n' https://quic.aiortc.org/logs/ # 404 curl -sS 'https://archive.org/wayback/available?url=quic.aiortc.org/logs/b171e6b78e186c78.qlog' # archived_snapshots empty curl -sS 'https://archive.org/wayback/available?url=quic.aiortc.org/logs/6e037439a5f79d47.qlog' # archived_snapshots empty ``` </details> 🤖 Written and tested with Claude Code (Opus 5.5), posted with my OK 🦊

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1093

No description provided.

Rows
Columns