[PR #3232] [MERGED] test(perf): evaluate larger and non-HTTP packet workflows #5661

New issue

Closed

opened 2026-10-31 01:08:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:08:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3232
Author: @N0zoM1z0
Created: 10/14/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/3130-packet-fields-20261014

📝 Commits (5)

81c78a3 test(perf): measure bounded non-HTTP packet investigations
fe488cb docs(perf): record packet workflow costs and negative controls
1a25cda test(perf): enforce packet comparison and oracle semantics
c4782ec test(perf): grade cited packet observations across later reads
8358ebc test(perf): preserve authenticated facts after failed packet calls

📊 Changes

15 files changed (+1260 additions, -1 deletions)

View changed files

📝 docs/packet-captures.md (+16 -0)
📝 experiments/analysis-efficiency/README.md (+6 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+1 -0)
📝 experiments/analysis-efficiency/compare.mjs (+9 -1)
📝 experiments/analysis-efficiency/corpus.json (+52 -0)
📝 experiments/analysis-efficiency/measure.mjs (+26 -0)
➕ experiments/analysis-efficiency/packet-entry.mjs (+316 -0)
➕ experiments/analysis-efficiency/packet-field-workflows.md (+226 -0)
➕ experiments/analysis-efficiency/packet-model-prompt.mjs (+40 -0)
➕ experiments/analysis-efficiency/packet-oracle.py (+180 -0)
➕ experiments/analysis-efficiency/questions/packet-postgres.txt (+3 -0)
➕ experiments/analysis-efficiency/review-packet.mjs (+247 -0)
➕ experiments/analysis-efficiency/test_packet_oracle.py (+67 -0)
📝 tests/evaluation/analysisEfficiencyComparison.test.ts (+39 -0)
➕ tests/evaluation/packetWorkflowSemantics.test.ts (+32 -0)

📄 Description

Public packet workflows need evidence about whether field discovery and paging improve larger/non-HTTP investigations. Add pinned PostgreSQL (4.9 MB), DNS and DHCP recordings, an independent header oracle, measured SDK workflows, actual Codex trials and reproducible fact/plan grading. Document how caller-selected input limits, fields and output windows bound a concrete question.

Results:

19 real non-HTTP SDK workflows preserve the independent requested facts. Focused reply JSON decreases 42.29% / 53.71% / 12.71% for PostgreSQL / DNS / DHCP. Discovery adds engine work and increases the small DHCP reply 44.08%.
Two eligible real Codex baseline/focused pairs show total input -9.86% and +1.82%, elapsed +37.70% and -19.30%. This does not establish stable total-token or speed gains.
Six real model attempts all establish the requested facts; five follow the fixed plan. One discovery attempt omits discovery and remains a workflow failure with its actual cost retained. The earlier rejected-approval preflight is also reported separately.
Eight current-runtime HTTP-control workflows preserve useful facts and remain negative summary/page controls. Historic model results retain their original runtime pin.

Production APIs, default delivery, provider limits, Evidence identity and cleanup ownership remain unchanged. No provider cache is introduced. Raw captures, results, credentials and client configurations stay outside Git. A separate large-error-context concern is recorded privately for later repair.

Validation: 47 Python experiment regressions; initial 19 relevant MCP/packet-output/comparison tests plus 10 focused comparison/page regression tests after strengthening the experimental checks; complete typecheck with the supported single-thread option; lint, format, knip and metadata checks. Actual-transcript negative grading checks reject changed counts/order/citations and distinguish missing limitations from wrong facts. An actual 24-row SDK Evidence replay proves fact/plan separation; an independently parsed second page rejects skipped matches. The named comparator preserves delivery observations across the three new packet questions. Architecture and docs checks pass. Three actual post-rebase SDK workflows on 099b549af0ed5fed3385bf0b54cc0020b23c6717 preserve the independent PostgreSQL/DNS/DHCP facts; those integration checks are separate from the immutable measurement cohort. All heavy runs are serial under an owned one-CPU scope with 3 GiB early stop / 4 GiB hard ceiling. Payload-exit observations are verified; final whole-scope counters remain unknown.

Measured runtime: ec67aebed02dbd20fa5c7703fcd183ad0b1c343c, Linux TShark/capinfos 4.0.17, Node 24.18.0, Codex CLI 0.161.0, requested gpt-5.6-terra / high. Native Windows/macOS and larger DNS/DHCP performance remain unverified. See the curated packet checkpoint for all measurements and limitations.

Closes #3130

CI attribution: the initial CI merge included newer main 3f5b7033. An independently built pure-main checkout reproduces the same binary_session accessor rejection as the SQLite/recorded-crash lanes; that repair is outside this PR. The packet workflow also passes against that newer pure-main runtime. All four substantive review findings are addressed and their focused checks pass.

Latest-main integration verification: rebased all four experimental commits onto 29d3dcb7ea85b3be37cd81e977ae4e09cebdd2fc; head c4782eca59de8873a9978a52b870b789ad1b9108. Locked dependencies are unchanged. Build and 60 focused tests pass (2 platform skips), including the existing process CLI depth and EVM cleanup cases that failed before the new upstream fixes. Actual PostgreSQL/DNS/DHCP SDK workflows preserve input digests and match the independent raw-capture facts. Four actual-Evidence citation replays cover later/reordered/repeated captures and wrong-page-only citations; regrading all six original model records preserves their outcomes and recorded costs. These are compatibility and grading checks, separately from the published original ec67aeb measurement cohort.

Final grader-only follow-up 8358ebcc3e227e4ba7043d9a628cf3b760c46d99: individual failed or malformed calls now affect workflow correctness without discarding authenticated observations used by a recovered answer. Nine failure-injection/answer-shape replays and four actual-Evidence paging/citation replays pass, with all six original actual model outcomes/costs unchanged. Complete typecheck and lint pass; the runtime/source integration verification above remains unchanged because this follow-up changes only the experiment grader and its documentation.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3232 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/14/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/3130-packet-fields-20261014` --- ### 📝 Commits (5) - [`81c78a3`](https://github.com/morluto/rea/commit/81c78a3bf72042692b0babbecf194f21de734c17) test(perf): measure bounded non-HTTP packet investigations - [`fe488cb`](https://github.com/morluto/rea/commit/fe488cb21508bc9fab9e4f55f85aef360d573fed) docs(perf): record packet workflow costs and negative controls - [`1a25cda`](https://github.com/morluto/rea/commit/1a25cda3a4f3eef04351e1870fc73fed4d11d439) test(perf): enforce packet comparison and oracle semantics - [`c4782ec`](https://github.com/morluto/rea/commit/c4782eca59de8873a9978a52b870b789ad1b9108) test(perf): grade cited packet observations across later reads - [`8358ebc`](https://github.com/morluto/rea/commit/8358ebcc3e227e4ba7043d9a628cf3b760c46d99) test(perf): preserve authenticated facts after failed packet calls ### 📊 Changes **15 files changed** (+1260 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `docs/packet-captures.md` (+16 -0) 📝 `experiments/analysis-efficiency/README.md` (+6 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+1 -0) 📝 `experiments/analysis-efficiency/compare.mjs` (+9 -1) 📝 `experiments/analysis-efficiency/corpus.json` (+52 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+26 -0) ➕ `experiments/analysis-efficiency/packet-entry.mjs` (+316 -0) ➕ `experiments/analysis-efficiency/packet-field-workflows.md` (+226 -0) ➕ `experiments/analysis-efficiency/packet-model-prompt.mjs` (+40 -0) ➕ `experiments/analysis-efficiency/packet-oracle.py` (+180 -0) ➕ `experiments/analysis-efficiency/questions/packet-postgres.txt` (+3 -0) ➕ `experiments/analysis-efficiency/review-packet.mjs` (+247 -0) ➕ `experiments/analysis-efficiency/test_packet_oracle.py` (+67 -0) 📝 `tests/evaluation/analysisEfficiencyComparison.test.ts` (+39 -0) ➕ `tests/evaluation/packetWorkflowSemantics.test.ts` (+32 -0) </details> ### 📄 Description Public packet workflows need evidence about whether field discovery and paging improve larger/non-HTTP investigations. Add pinned PostgreSQL (4.9 MB), DNS and DHCP recordings, an independent header oracle, measured SDK workflows, actual Codex trials and reproducible fact/plan grading. Document how caller-selected input limits, fields and output windows bound a concrete question. Results: - 19 real non-HTTP SDK workflows preserve the independent requested facts. Focused reply JSON decreases 42.29% / 53.71% / 12.71% for PostgreSQL / DNS / DHCP. Discovery adds engine work and increases the small DHCP reply 44.08%. - Two eligible real Codex baseline/focused pairs show total input -9.86% and +1.82%, elapsed +37.70% and -19.30%. This does not establish stable total-token or speed gains. - Six real model attempts all establish the requested facts; five follow the fixed plan. One discovery attempt omits discovery and remains a workflow failure with its actual cost retained. The earlier rejected-approval preflight is also reported separately. - Eight current-runtime HTTP-control workflows preserve useful facts and remain negative summary/page controls. Historic model results retain their original runtime pin. Production APIs, default delivery, provider limits, Evidence identity and cleanup ownership remain unchanged. No provider cache is introduced. Raw captures, results, credentials and client configurations stay outside Git. A separate large-error-context concern is recorded privately for later repair. Validation: 47 Python experiment regressions; initial 19 relevant MCP/packet-output/comparison tests plus 10 focused comparison/page regression tests after strengthening the experimental checks; complete typecheck with the supported single-thread option; lint, format, knip and metadata checks. Actual-transcript negative grading checks reject changed counts/order/citations and distinguish missing limitations from wrong facts. An actual 24-row SDK Evidence replay proves fact/plan separation; an independently parsed second page rejects skipped matches. The named comparator preserves delivery observations across the three new packet questions. Architecture and docs checks pass. Three actual post-rebase SDK workflows on `099b549af0ed5fed3385bf0b54cc0020b23c6717` preserve the independent PostgreSQL/DNS/DHCP facts; those integration checks are separate from the immutable measurement cohort. All heavy runs are serial under an owned one-CPU scope with 3 GiB early stop / 4 GiB hard ceiling. Payload-exit observations are verified; final whole-scope counters remain unknown. Measured runtime: `ec67aebed02dbd20fa5c7703fcd183ad0b1c343c`, Linux TShark/capinfos 4.0.17, Node 24.18.0, Codex CLI 0.161.0, requested `gpt-5.6-terra` / high. Native Windows/macOS and larger DNS/DHCP performance remain unverified. See the curated packet checkpoint for all measurements and limitations. Closes #3130 CI attribution: the initial CI merge included newer main `3f5b7033`. An independently built pure-main checkout reproduces the same `binary_session` accessor rejection as the SQLite/recorded-crash lanes; that repair is outside this PR. The packet workflow also passes against that newer pure-main runtime. All four substantive review findings are addressed and their focused checks pass. Latest-main integration verification: rebased all four experimental commits onto `29d3dcb7ea85b3be37cd81e977ae4e09cebdd2fc`; head `c4782eca59de8873a9978a52b870b789ad1b9108`. Locked dependencies are unchanged. Build and 60 focused tests pass (2 platform skips), including the existing process CLI depth and EVM cleanup cases that failed before the new upstream fixes. Actual PostgreSQL/DNS/DHCP SDK workflows preserve input digests and match the independent raw-capture facts. Four actual-Evidence citation replays cover later/reordered/repeated captures and wrong-page-only citations; regrading all six original model records preserves their outcomes and recorded costs. These are compatibility and grading checks, separately from the published original `ec67aeb` measurement cohort. Final grader-only follow-up `8358ebcc3e227e4ba7043d9a628cf3b760c46d99`: individual failed or malformed calls now affect workflow correctness without discarding authenticated observations used by a recovered answer. Nine failure-injection/answer-shape replays and four actual-Evidence paging/citation replays pass, with all six original actual model outcomes/costs unchanged. Complete typecheck and lint pass; the runtime/source integration verification above remains unchanged because this follow-up changes only the experiment grader and its documentation. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:08:23 +00:00