morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3132] test(perf): measure existing SAZ summary and retained transaction views #1092

New issue

Closed

opened 2026-10-15 18:30:59 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:30:59 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3132

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. PCAP results in #2640 are a warning against automatic summary defaults, not SAZ evidence.

Analyst question and outcome

For a fixed HTTP transaction/header/body question in a genuine Fiddler SAZ archive, do existing summary and exact retained transaction views reduce useful-answer token/time costs?

Existing evidence and decision boundary

REA already provides SAZ capture summary/pages, original archive-member coordinates and encoded source bytes. The earlier continuation identified all-record decoding before ordinal selection, but did not measure a real SAZ model cohort. Another paging API is not justified without evidence.

Bounded work

Select two digest-pinned public genuine SAZ archives with documented source provenance: a compact control and a larger archive that stays within the existing resource bounds. Pin the currently supported layout and Fiddler/parser identity where available.
Define exact requested transactions and facts before testing. Independently inspect archive members and raw HTTP material; preserve encoded bytes, body absence versus empty body, duplicate headers, record ordering and source coordinates.
Compare existing complete/summary/preview/retained selection workflows only when they answer the same question. Measure initial decoding versus subsequent retained reads, response expansion and extra RPCs. Keep current transport-credential redaction and caller evidence semantics intact; do not infer secrecy from header or path names.
Run at least two SDK pairs per archive and two actual Codex pairs on one representative question. Preserve malformed/partial outcomes. Add code only if a specific cost is reproduced and existing selectors cannot address it.

Acceptance

Corpus/layout, source bytes, exact questions and independent transaction/member checks are documented.
Equivalent current complete and selected workflows preserve original record identity, headers/body bytes, source coordinates and truthful coverage.
Repeated SDK/model observations separate decoding, retained reads, delivery and all attempts; no JSON-byte-to-token inference is made.
Publish a scoped decision and reproduction; keep no-benefit results and verify old request/Evidence behavior for any proposed change.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3132 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. PCAP results in #2640 are a warning against automatic summary defaults, not SAZ evidence. ## Analyst question and outcome For a fixed HTTP transaction/header/body question in a genuine Fiddler SAZ archive, do existing summary and exact retained transaction views reduce useful-answer token/time costs? ## Existing evidence and decision boundary REA already provides SAZ capture summary/pages, original archive-member coordinates and encoded source bytes. The earlier continuation identified all-record decoding before ordinal selection, but did not measure a real SAZ model cohort. Another paging API is not justified without evidence. ## Bounded work 1. Select two digest-pinned public genuine SAZ archives with documented source provenance: a compact control and a larger archive that stays within the existing resource bounds. Pin the currently supported layout and Fiddler/parser identity where available. 2. Define exact requested transactions and facts before testing. Independently inspect archive members and raw HTTP material; preserve encoded bytes, body absence versus empty body, duplicate headers, record ordering and source coordinates. 3. Compare existing complete/summary/preview/retained selection workflows only when they answer the same question. Measure initial decoding versus subsequent retained reads, response expansion and extra RPCs. Keep current transport-credential redaction and caller evidence semantics intact; do not infer secrecy from header or path names. 4. Run at least two SDK pairs per archive and two actual Codex pairs on one representative question. Preserve malformed/partial outcomes. Add code only if a specific cost is reproduced and existing selectors cannot address it. ## Acceptance - [x] Corpus/layout, source bytes, exact questions and independent transaction/member checks are documented. - [x] Equivalent current complete and selected workflows preserve original record identity, headers/body bytes, source coordinates and truthful coverage. - [x] Repeated SDK/model observations separate decoding, retained reads, delivery and all attempts; no JSON-byte-to-token inference is made. - [x] Publish a scoped decision and reproduction; keep no-benefit results and verify old request/Evidence behavior for any proposed change. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi

2026-10-15 18:30:59 +00:00

closed this issue
added the
performance

design

P1: high

area: domain

roadmap

enhancement
labels

bsakaguchi commented

2026-10-15 18:31:00 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

Started on fresh main ca1b2f860 after merging the sigrok checkpoint. Branch: perf/3132-saz-views-20261014; corpus checkpoint: 282cca685.

Two genuine Microsoft Xbox Live Trace Analyzer archives are pinned to upstream ad880ff00: a 13-session / 43,515-byte control and a 1,311-session / 5,223,519-byte archive (9,223,820 expanded member bytes). Downloaded Git blobs/SHA-256 and Fiddler archive comments are verified; raw archives remain private. Questions and caller order were fixed before examining transaction facts.

Five independent stored-HTTP/admission regressions and all six selected transaction/member checks pass. Baseline build and bounded dependency/typecheck/lint checks pass. SDK/model costs have not yet been measured; no performance gain or production change is claimed. Next: compare complete delivery, ordinal selection and summary/retained complete pages, then actual Codex pairs for the fixed control question.

Corpus, exact question and current verification scope.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): Started on fresh main `ca1b2f860` after merging the sigrok checkpoint. Branch: `perf/3132-saz-views-20261014`; corpus checkpoint: [282cca685](https://github.com/morluto/rea/commit/282cca6857821da967e7e469e140bda7731426f1). Two genuine Microsoft Xbox Live Trace Analyzer archives are pinned to upstream `ad880ff00`: a 13-session / 43,515-byte control and a 1,311-session / 5,223,519-byte archive (9,223,820 expanded member bytes). Downloaded Git blobs/SHA-256 and Fiddler archive comments are verified; raw archives remain private. Questions and caller order were fixed before examining transaction facts. Five independent stored-HTTP/admission regressions and all six selected transaction/member checks pass. Baseline build and bounded dependency/typecheck/lint checks pass. **SDK/model costs have not yet been measured; no performance gain or production change is claimed.** Next: compare complete delivery, ordinal selection and summary/retained complete pages, then actual Codex pairs for the fixed control question. [Corpus, exact question and current verification scope](https://github.com/morluto/rea/blob/perf/3132-saz-views-20261014/experiments/analysis-efficiency/saz-selection-workflows.md).

bsakaguchi commented

2026-10-15 18:31:00 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

SAZ measurement decision

The bounded #3132 study is complete; implementation/validation PR: https://github.com/morluto/rea/pull/3584.

Genuine immutable Microsoft Fiddler archives contain 13 and 1,311 sessions; a third genuine partial-message archive is an admission-negative probe. Source digests, archive coordinates and independent raw-HTTP/body oracles are pinned.
Corrected SDK repetitions: 12 workflows / 32 RPCs, 11 correct completed questions and one provider timeout. Successful necessary facts agree across arms. Initial collector-error attempts remain separate.
Four actual Codex attempts: three correct answers, zero prescribed single-call plans. The sole fully completed pair reduces total input 1.6% but increases elapsed time 2.0%; the failed second focus is retained without a replacement retry. Parsed reply reduction is not a model-token claim.
Decision: retain current production defaults, APIs and cache behavior. Reuse existing selectors/session-owned retained views; no stable whole-task benefit justifies a new optimization.
All 16 corrected SDK/model payload phases have verified cleanup and no observed OOM/abort under the serial one-CPU/4 GiB guard; terminal whole-scope counters and isolated decoder-stage attribution remain unknown.
Latest-main compatibility: build and relevant tests pass (15 passed, one runtime-conditional unsupported-memory case skipped); two genuine focused MCP workflows preserve the original source facts. Costs remain attributed to their frozen original runtime.

Detailed reproduction, all cost tables and limitations: https://github.com/morluto/rea/blob/perf/3132-saz-views-20261014/experiments/analysis-efficiency/saz-selection-workflows.md . Raw artifacts/results remain private; unrelated findings stay in the existing /tmp team handoff. Other platforms, Fiddler acquisition and arbitrary layouts remain unverified.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): ## SAZ measurement decision The bounded #3132 study is complete; implementation/validation PR: https://github.com/morluto/rea/pull/3584. - Genuine immutable Microsoft Fiddler archives contain 13 and 1,311 sessions; a third genuine partial-message archive is an admission-negative probe. Source digests, archive coordinates and independent raw-HTTP/body oracles are pinned. - Corrected SDK repetitions: 12 workflows / 32 RPCs, 11 correct completed questions and one provider timeout. Successful necessary facts agree across arms. Initial collector-error attempts remain separate. - Four actual Codex attempts: three correct answers, zero prescribed single-call plans. The sole fully completed pair reduces total input 1.6% but increases elapsed time 2.0%; the failed second focus is retained without a replacement retry. Parsed reply reduction is not a model-token claim. - Decision: retain current production defaults, APIs and cache behavior. Reuse existing selectors/session-owned retained views; no stable whole-task benefit justifies a new optimization. - All 16 corrected SDK/model payload phases have verified cleanup and no observed OOM/abort under the serial one-CPU/4 GiB guard; terminal whole-scope counters and isolated decoder-stage attribution remain unknown. - Latest-main compatibility: build and relevant tests pass (15 passed, one runtime-conditional unsupported-memory case skipped); two genuine focused MCP workflows preserve the original source facts. Costs remain attributed to their frozen original runtime. Detailed reproduction, all cost tables and limitations: https://github.com/morluto/rea/blob/perf/3132-saz-views-20261014/experiments/analysis-efficiency/saz-selection-workflows.md . Raw artifacts/results remain private; unrelated findings stay in the existing /tmp team handoff. Other platforms, Fiddler acquisition and arbitrary layouts remain unverified.

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:08:32 +00:00

[PR #1092] [MERGED] fix(javascript): preserve HTML carriage return source ranges #2640

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:55 +00:00