[PR #3584] [MERGED] test(perf): measure genuine SAZ selection and retained recovery #5836

Closed
opened 2026-10-31 01:09:48 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3584
Author: @N0zoM1z0
Created: 10/14/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @morluto

Base: main ← Head: perf/3132-saz-views-20261014


📝 Commits (4)

  • 2ba84d2 test(perf): pin genuine Fiddler captures and raw HTTP oracles
  • 4a91005 test(perf): measure SAZ selection and retained recovery workflows
  • a118ef2 test(perf): verify resolved SAZ Evidence selection parameters
  • 7255c01 docs(perf): retain mixed SAZ whole-task costs and failure boundaries

📊 Changes

11 files changed (+1143 additions, -0 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+6 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+1 -0)
📝 experiments/analysis-efficiency/corpus.json (+60 -0)
📝 experiments/analysis-efficiency/measure.mjs (+22 -0)
➕ experiments/analysis-efficiency/review-saz.mjs (+266 -0)
➕ experiments/analysis-efficiency/saz-entry.mjs (+326 -0)
➕ experiments/analysis-efficiency/saz-model-prompt.mjs (+30 -0)
➕ experiments/analysis-efficiency/saz-oracle.py (+121 -0)
➕ experiments/analysis-efficiency/saz-selection-workflows.md (+214 -0)
➕ experiments/analysis-efficiency/test_saz_oracle.py (+48 -0)
➕ tests/evaluation/sazWorkflowSemantics.test.ts (+49 -0)

📄 Description

Summary

Fixes #3132. Measure existing SAZ selection and retained views against genuine public captures, preserving independent byte checks and every failed attempt. The measured cohort does not establish a stable whole-task token/time improvement, so production APIs, defaults and caches remain unchanged.

Detailed corpus, reproduction, cost tables and limits: SAZ selection findings.

Problem and expected behavior

Smaller replies alone do not establish cheaper correct answers. Pin the same transaction/header/stored-body question and compare complete delivery, caller-selected ordinals and retained recovery, counting discovery, extra RPCs and failures.

Change and scope

  • Pin Microsoft's genuine 13-session and 1,311-session Fiddler captures, plus one genuine partial-message admission-negative archive. No capture binaries or raw results are committed.
  • Add an independent raw-HTTP oracle, SDK workflow, actual-Codex prompts and citation/byte grading. Validate typed Evidence with its semantic-ID owner and named input/output contracts; distinguish omitted requests from resolved Evidence parameters.
  • Recover only the exact retained reference returned by an MCP transport-budget error; preserve that failed call and all follow-ups. Preview alone cannot answer the complete-header question.
  • Retain mixed findings instead of introducing an unproven runtime optimization.

Contract and boundary impact

  • Semantic owner and earliest changed stage: experiment collection/grading only; production capture and Evidence owners unchanged.
  • Contribution scope and maintainer decision: existing passive archive reads and selectors; new target-execution/mutation capability decision not applicable.
  • CLI/MCP/catalog and provider compatibility: no contract, provider or catalog changes. Tested installed fiddler-saz stack on Linux; archive comments report Fiddler versions, without a local Fiddler acquisition claim.
  • Evidence/provenance: preserve source digests, padded IDs, archive coordinates, caller order, original header pairs, encoded binary/empty/absent bodies and semantic identity.
  • Target/cleanup: explicit pinned local paths; owned REA/Codex processes and temporary experiment profiles; no target code execution or live captured-URL fetches.
  • Generated/package/installation impact: none; no release.

Expected effects of reproducing the experiments:

  • Read existing files or captured data.
  • Launch or attach to a process. (Launch owned REA/Codex only.)
  • Inject instrumentation or run scripts/code. (Experiment/REA code only; no target instrumentation.)
  • Change process memory, registers, files, or configuration. (Private outputs and temporary profiles only.)
  • Contact network services. (Explicit public fixture downloads and requested model calls.)
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Corrected SDK cohort: 12 workflows / 32 RPCs, 11 successful questions and one provider timeout; every successful arm matches independent necessary facts. The earlier collector-error batch remains separately documented.

  • Control selected replies shrink 73.8% in parsed JSON bytes, not model tokens. Large direct selection returns 535,767 bytes versus 114,294 for successful complete-workflow recovery; retained container inventory still matters.

  • Actual Codex: 4 attempts, 3 correct answers, 0 prescribed single-call plans. The only pair with both questions completed changes total input by −1.6% and time by +2.0%. The failed second focus remains a failed task, with no replacement retry.

  • Actual selected CLI/MCP facts, parameters and semantic Evidence ID agree. A 30-second provider timeout, successful-analysis MCP receive-buffer failure and empty raw HTTP member are distinct preserved outcomes.

  • Observed, derived, and inferred claims remain distinguishable.

  • Artifact identity, source provenance, and failed attempts remain preserved.

  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • python3 experiments/analysis-efficiency/test_saz_oracle.py — 5 regressions pass; both genuine positive oracles independently verify all six selected records.
  • Focused comparison/checker tests before rebase — 12 pass; resolved-scope regression and 10 actual-Evidence grading preflight cases pass. Injected grading answers establish checker semantics, not model performance.
  • After rebase onto main 1579b6865e8e, npm run test:focused -- tests/boundary/browser/sazWebCapture.test.ts tests/evaluation/sazWorkflowSemantics.test.ts — runtime build passes; 15 tests pass, 1 intentionally skipped (unsupported decoder-memory-runtime case does not apply to Node).
  • Two actual focused MCP workflows after rebase — control and large pass their independent byte/header/member checks; source digests unchanged. These are compatibility checks, not replacement cost pairs.
  • Bounded pre-push dependency/typecheck/lint checks — pass; git diff --check passes.

Compatibility, safety, and release

  • No breaking changes, migration or release metadata changes. Existing complete defaults and session-owned views remain the baseline.
  • Linux only, two positive immutable captures, one negative partial-message source, one installed parser stack. Other platforms/acquisition/layouts are unverified.
  • Requested model is gpt-5.6-terra / high; backend identity is unknown. Owned logs show exec discovery/dispatch despite requested direct mode; direct presentation is not claimed. One tool-discovery failure remains unexplained.
  • Serial one-CPU guard, 3 GiB early stop / 4 GiB hard family memory and no swap. All 16 corrected SDK/model payload phases verify cleanup with no observed OOM/abort; SDK/model peaks 865/833 MiB. Latest-main build/test peak about 2.61 GiB. Terminal whole-scope counters and isolated decoder timings remain unknown.
  • Raw fixtures/replies/transcripts/configs stay outside the repository. Unrelated findings are in the existing /tmp team handoff.

Author readiness

Human owner: @N0zoM1z0

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output. (Production behavior is unchanged; experiment results are linked.)
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3584 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/14/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `perf/3132-saz-views-20261014` --- ### 📝 Commits (4) - [`2ba84d2`](https://github.com/morluto/rea/commit/2ba84d2f712efe17ebba9f212b2c68d209982aee) test(perf): pin genuine Fiddler captures and raw HTTP oracles - [`4a91005`](https://github.com/morluto/rea/commit/4a9100531e9fdde5a89e7ecf1580bbc610a06dac) test(perf): measure SAZ selection and retained recovery workflows - [`a118ef2`](https://github.com/morluto/rea/commit/a118ef2c794b008b9d65c17559c9441e08c831b6) test(perf): verify resolved SAZ Evidence selection parameters - [`7255c01`](https://github.com/morluto/rea/commit/7255c01c47d2a587c89960447216f7ae62aba3f3) docs(perf): retain mixed SAZ whole-task costs and failure boundaries ### 📊 Changes **11 files changed** (+1143 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+6 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+1 -0) 📝 `experiments/analysis-efficiency/corpus.json` (+60 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+22 -0) ➕ `experiments/analysis-efficiency/review-saz.mjs` (+266 -0) ➕ `experiments/analysis-efficiency/saz-entry.mjs` (+326 -0) ➕ `experiments/analysis-efficiency/saz-model-prompt.mjs` (+30 -0) ➕ `experiments/analysis-efficiency/saz-oracle.py` (+121 -0) ➕ `experiments/analysis-efficiency/saz-selection-workflows.md` (+214 -0) ➕ `experiments/analysis-efficiency/test_saz_oracle.py` (+48 -0) ➕ `tests/evaluation/sazWorkflowSemantics.test.ts` (+49 -0) </details> ### 📄 Description ## Summary Fixes #3132. Measure existing SAZ selection and retained views against genuine public captures, preserving independent byte checks and every failed attempt. The measured cohort does **not establish a stable whole-task token/time improvement**, so production APIs, defaults and caches remain unchanged. Detailed corpus, reproduction, cost tables and limits: [SAZ selection findings](https://github.com/morluto/rea/blob/perf/3132-saz-views-20261014/experiments/analysis-efficiency/saz-selection-workflows.md). ## Problem and expected behavior Smaller replies alone do not establish cheaper correct answers. Pin the same transaction/header/stored-body question and compare complete delivery, caller-selected ordinals and retained recovery, counting discovery, extra RPCs and failures. ## Change and scope - Pin Microsoft's genuine 13-session and 1,311-session Fiddler captures, plus one genuine partial-message admission-negative archive. No capture binaries or raw results are committed. - Add an independent raw-HTTP oracle, SDK workflow, actual-Codex prompts and citation/byte grading. Validate typed Evidence with its semantic-ID owner and named input/output contracts; distinguish omitted requests from resolved Evidence parameters. - Recover only the exact retained reference returned by an MCP transport-budget error; preserve that failed call and all follow-ups. Preview alone cannot answer the complete-header question. - Retain mixed findings instead of introducing an unproven runtime optimization. ## Contract and boundary impact - Semantic owner and earliest changed stage: experiment collection/grading only; production capture and Evidence owners unchanged. - Contribution scope and maintainer decision: existing passive archive reads and selectors; new target-execution/mutation capability decision not applicable. - CLI/MCP/catalog and provider compatibility: no contract, provider or catalog changes. Tested installed `fiddler-saz` stack on Linux; archive comments report Fiddler versions, without a local Fiddler acquisition claim. - Evidence/provenance: preserve source digests, padded IDs, archive coordinates, caller order, original header pairs, encoded binary/empty/absent bodies and semantic identity. - Target/cleanup: explicit pinned local paths; owned REA/Codex processes and temporary experiment profiles; no target code execution or live captured-URL fetches. - Generated/package/installation impact: none; no release. Expected effects of reproducing the experiments: - [x] Read existing files or captured data. - [x] Launch or attach to a process. (Launch owned REA/Codex only.) - [x] Inject instrumentation or run scripts/code. (Experiment/REA code only; no target instrumentation.) - [x] Change process memory, registers, files, or configuration. (Private outputs and temporary profiles only.) - [x] Contact network services. (Explicit public fixture downloads and requested model calls.) - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Corrected SDK cohort: **12 workflows / 32 RPCs, 11 successful questions and one provider timeout**; every successful arm matches independent necessary facts. The earlier collector-error batch remains separately documented. - Control selected replies shrink **73.8% in parsed JSON bytes**, not model tokens. Large direct selection returns 535,767 bytes versus 114,294 for successful complete-workflow recovery; retained container inventory still matters. - Actual Codex: **4 attempts, 3 correct answers, 0 prescribed single-call plans**. The only pair with both questions completed changes total input by **−1.6%** and time by **+2.0%**. The failed second focus remains a failed task, with no replacement retry. - Actual selected CLI/MCP facts, parameters and semantic Evidence ID agree. A 30-second provider timeout, successful-analysis MCP receive-buffer failure and empty raw HTTP member are distinct preserved outcomes. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `python3 experiments/analysis-efficiency/test_saz_oracle.py` — 5 regressions pass; both genuine positive oracles independently verify all six selected records. - Focused comparison/checker tests before rebase — 12 pass; resolved-scope regression and 10 actual-Evidence grading preflight cases pass. Injected grading answers establish checker semantics, not model performance. - After rebase onto main `1579b6865e8e`, `npm run test:focused -- tests/boundary/browser/sazWebCapture.test.ts tests/evaluation/sazWorkflowSemantics.test.ts` — runtime build passes; 15 tests pass, 1 intentionally skipped (unsupported decoder-memory-runtime case does not apply to Node). - Two actual focused MCP workflows after rebase — control and large pass their independent byte/header/member checks; source digests unchanged. These are compatibility checks, not replacement cost pairs. - Bounded pre-push dependency/typecheck/lint checks — pass; `git diff --check` passes. ## Compatibility, safety, and release - No breaking changes, migration or release metadata changes. Existing complete defaults and session-owned views remain the baseline. - Linux only, two positive immutable captures, one negative partial-message source, one installed parser stack. Other platforms/acquisition/layouts are unverified. - Requested model is `gpt-5.6-terra` / `high`; backend identity is unknown. Owned logs show exec discovery/dispatch despite requested direct mode; direct presentation is not claimed. One tool-discovery failure remains unexplained. - Serial one-CPU guard, 3 GiB early stop / 4 GiB hard family memory and no swap. All 16 corrected SDK/model payload phases verify cleanup with no observed OOM/abort; SDK/model peaks 865/833 MiB. Latest-main build/test peak about 2.61 GiB. Terminal whole-scope counters and isolated decoder timings remain unknown. - Raw fixtures/replies/transcripts/configs stay outside the repository. Unrelated findings are in the existing `/tmp` team handoff. ## Author readiness Human owner: @N0zoM1z0 - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. (Production behavior is unchanged; experiment results are linked.) - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:09:48 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5836
No description provided.