[PR #2640] [MERGED] test(perf): preserve negative small-PCAP workflow measurements #5302

New issue

Closed

opened 2026-10-31 01:05:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:05:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2640
Author: @N0zoM1z0
Created: 10/13/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/2016-packet-real-data-20261013

📝 Commits (10+)

fb10bdc test(perf): compare complete and retained-page PCAP workflows
78b8e28 docs(perf): preserve negative small-capture token and timing results
56a90af fix(experiments): reject malformed PCAP receipt identities as failed facts
3c36892 fix(experiments): bind PCAP facts to the cited receipt chain
eac921e fix(experiments): compare stable PCAP receipt fields across repeated deliveries
2a7982c fix(experiments): verify cited PCAP metadata and exact receipt IDs
6b167f9 fix(experiments): bind PCAP prefix coverage and requested view
e5abf4a fix(experiments): reject reused PCAP page receipt IDs
d688507 fix(experiments): validate PCAP coverage and cited artifact subjects
073f6c0 fix(experiments): validate PCAP receipt ownership before observation

📊 Changes

10 files changed (+1562 additions, -20 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+4 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+1 -1)
📝 experiments/analysis-efficiency/compare.mjs (+38 -0)
📝 experiments/analysis-efficiency/corpus.json (+28 -0)
📝 experiments/analysis-efficiency/measure.mjs (+200 -0)
➕ experiments/analysis-efficiency/pcap-small-captures.md (+291 -0)
➕ experiments/analysis-efficiency/questions/pcap-http.txt (+1 -0)
📝 experiments/analysis-efficiency/review-model.py (+259 -19)
📝 experiments/analysis-efficiency/test_codex_context.py (+541 -0)
➕ tests/evaluation/analysisEfficiencyComparison.test.ts (+199 -0)

📄 Description

For two small official HTTP recordings and a fixed metadata/URI question, the existing summary/page workflow returns more than twice the JSON bytes and about 51% more total model input in both real-model pairs. Preserve the negative measurements and reproducible workflows; keep complete defaults. No production optimization is justified by this corpus. Refs #2016.

Nine experiment files preserve capture provenance, equivalent complete and summary/URI-page SDK workflows, the agent question, factual/workflow grading and curated findings. The SDK authenticates envelopes with the selected runtime's existing Evidence parser and validates coverage, target identity, requested views, producer ownership and retained-parent links. Factual grading preserves partial facts and malformed-reply diagnostics; separate workflow checks require the exact ordered calls, arguments, successful readable replies and citations. The six comparator CLI scenarios run in the existing discovered Vitest evaluation project and replace the manual suite.

Final scope: 10 files — nine experiment files and one evaluation test. No production/API/default/cache/dependency changes authored by this PR. Upstream now owns both the shared report-environment fixture and Windows inspection verification, superseding our earlier local differences. Their final files match main exactly. Original measurements and all ten final source files remain unchanged during this integration.

Validation:

Eight actual SDK workflows and four Codex CLI tasks requesting gpt-5.6-terra high; public CLI/MCP parity and independent raw-frame packet/URI/endpoint checks. All four stored model answers pass final factual and exact-workflow regrading without model reruns.
All 18 original SDK/model Evidence records independently authenticate against the pinned runtime. Actual SDK controls and consumer fault injections verify the acceptance guards. Synthetic grader fixtures establish grading semantics rather than authentication or real-provider malformed behavior.
34 context/grading regressions and six discovered comparator scenarios pass. Recorded discovery checks cover the then-configured 1,113 files exactly once; later added main tests are not included in that historical count. Real hosted Windows results are distinct from local loader/process fixtures.
Serial local work uses one CPU, 3-GiB early stop and 4-GiB hard/no-swap limits. Unavailable terminal whole-scope counters stay unknown. Strict original resource grades remain false for those unknowns. Raw captures, results, traces and configurations stay private.

Measured runtime stays 3ae826ceba6f25b08bce50bcb51001209bc16b57. Current head 36afc673f8b437521aab5df889ac12aca50d105e integrates immutable main 6a7f70fd5422489e724de54d1fd6052416b9732d, resolving the Windows verifier conflict by using its complete upstream owner. That version preserves foreign replacement identity and contents and strengthens nonempty owned-directory/teardown checks. Full bounded preflight passes in 73.953 seconds with ~2.23 GiB peak, verified payload cleanup and zero observed phase OOM/limit events; terminal counters remain unknown. No original model/SDK workflow or unchanged local full suite is rerun.

The preceding head's automatic CI 38487434090 passed source/static/build/docs, actual Windows/native/package and real application lanes, including reverse-tools coverage/mutation. One JavaScript worker cleanup case failed with an ownership-token mismatch; exact main/runtime inputs match, and its isolated original main case passes. Root cause/frequency remain unknown and the original failure is recorded in the team's /tmp handoff. One targeted retry executes only the failing shard 2/5 and fail-fast-cancelled shards 4/5 and 5/5, retaining all successful jobs. The targeted retry completes successfully: all 35 jobs and required gates pass for the preceding immutable head. It does not establish the original failure root cause or a product fix. Current-head CI passes all 35 selected jobs in run 38492231165, and normal squash merge is verified at c7105d9723. The final tree and all ten PR-owned files match the reviewed versions.

Total input worsens for this corpus; uncached input and elapsed time are mixed. No general speedup or broader-domain result is inferred. Own commits retain anonymous metadata and requested factory-droid/Kaoru co-authorship; the squash also credits the maintainer. No release.

Merged 2026-10-14 01:34:03 UTC. CI certifies executed integration 4eb2262359cb01f2fb1314db0bc18d3dc9092792 with main 41a2e84647. Actual squash parent b8513401e1 and final tree are verified; later upstream changes are not independently requalified by this PR. Anonymous primary email and all requested co-authors are recognized by GitHub. No release.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2640 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/13/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/2016-packet-real-data-20261013` --- ### 📝 Commits (10+) - [`fb10bdc`](https://github.com/morluto/rea/commit/fb10bdc546c57715b66a982b79a04b7ffa3ce4f1) test(perf): compare complete and retained-page PCAP workflows - [`78b8e28`](https://github.com/morluto/rea/commit/78b8e28b67f091bfaf8809ef3aa6498d0f6c4df4) docs(perf): preserve negative small-capture token and timing results - [`56a90af`](https://github.com/morluto/rea/commit/56a90afdded70c7b79e1fae4aa778f0153c80514) fix(experiments): reject malformed PCAP receipt identities as failed facts - [`3c36892`](https://github.com/morluto/rea/commit/3c36892ab77ac0840acb395ad70dc9fb684971e9) fix(experiments): bind PCAP facts to the cited receipt chain - [`eac921e`](https://github.com/morluto/rea/commit/eac921eb44c5974b40d25fafb2874563d885da0b) fix(experiments): compare stable PCAP receipt fields across repeated deliveries - [`2a7982c`](https://github.com/morluto/rea/commit/2a7982ce2b1fd5080f75235929ba9a84b4226608) fix(experiments): verify cited PCAP metadata and exact receipt IDs - [`6b167f9`](https://github.com/morluto/rea/commit/6b167f98123f964d9d5ebf5e0088e458bed52f61) fix(experiments): bind PCAP prefix coverage and requested view - [`e5abf4a`](https://github.com/morluto/rea/commit/e5abf4acf402d42e93accb9fbfd32fc453c44cd5) fix(experiments): reject reused PCAP page receipt IDs - [`d688507`](https://github.com/morluto/rea/commit/d6885075632ced8b3a25a418aca805ad94cd8fe2) fix(experiments): validate PCAP coverage and cited artifact subjects - [`073f6c0`](https://github.com/morluto/rea/commit/073f6c07e996aec6148a4782a56b7861cbdea1a7) fix(experiments): validate PCAP receipt ownership before observation ### 📊 Changes **10 files changed** (+1562 additions, -20 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+4 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+1 -1) 📝 `experiments/analysis-efficiency/compare.mjs` (+38 -0) 📝 `experiments/analysis-efficiency/corpus.json` (+28 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+200 -0) ➕ `experiments/analysis-efficiency/pcap-small-captures.md` (+291 -0) ➕ `experiments/analysis-efficiency/questions/pcap-http.txt` (+1 -0) 📝 `experiments/analysis-efficiency/review-model.py` (+259 -19) 📝 `experiments/analysis-efficiency/test_codex_context.py` (+541 -0) ➕ `tests/evaluation/analysisEfficiencyComparison.test.ts` (+199 -0) </details> ### 📄 Description For two small official HTTP recordings and a fixed metadata/URI question, the existing summary/page workflow returns more than twice the JSON bytes and about **51% more total model input** in both real-model pairs. Preserve the negative measurements and reproducible workflows; keep complete defaults. No production optimization is justified by this corpus. Refs #2016. Nine experiment files preserve capture provenance, equivalent complete and summary/URI-page SDK workflows, the agent question, factual/workflow grading and curated findings. The SDK authenticates envelopes with the selected runtime's existing Evidence parser and validates coverage, target identity, requested views, producer ownership and retained-parent links. Factual grading preserves partial facts and malformed-reply diagnostics; separate workflow checks require the exact ordered calls, arguments, successful readable replies and citations. The six comparator CLI scenarios run in the existing discovered Vitest evaluation project and replace the manual suite. **Final scope: 10 files — nine experiment files and one evaluation test. No production/API/default/cache/dependency changes authored by this PR.** Upstream now owns both the shared report-environment fixture and Windows inspection verification, superseding our earlier local differences. Their final files match main exactly. Original measurements and all ten final source files remain unchanged during this integration. Validation: - Eight actual SDK workflows and four Codex CLI tasks requesting gpt-5.6-terra high; public CLI/MCP parity and independent raw-frame packet/URI/endpoint checks. All four stored model answers pass final factual and exact-workflow regrading without model reruns. - All 18 original SDK/model Evidence records independently authenticate against the pinned runtime. Actual SDK controls and consumer fault injections verify the acceptance guards. Synthetic grader fixtures establish grading semantics rather than authentication or real-provider malformed behavior. - 34 context/grading regressions and six discovered comparator scenarios pass. Recorded discovery checks cover the then-configured 1,113 files exactly once; later added main tests are not included in that historical count. Real hosted Windows results are distinct from local loader/process fixtures. - Serial local work uses one CPU, 3-GiB early stop and 4-GiB hard/no-swap limits. Unavailable terminal whole-scope counters stay unknown. Strict original resource grades remain false for those unknowns. Raw captures, results, traces and configurations stay private. Measured runtime stays `3ae826ceba6f25b08bce50bcb51001209bc16b57`. Current head `36afc673f8b437521aab5df889ac12aca50d105e` integrates immutable main `6a7f70fd5422489e724de54d1fd6052416b9732d`, resolving the Windows verifier conflict by using its complete upstream owner. That version preserves foreign replacement identity and contents and strengthens nonempty owned-directory/teardown checks. Full bounded preflight passes in 73.953 seconds with ~2.23 GiB peak, verified payload cleanup and zero observed phase OOM/limit events; terminal counters remain unknown. No original model/SDK workflow or unchanged local full suite is rerun. The preceding head's automatic [CI 38487434090](https://github.com/morluto/rea/actions/runs/38487434090) passed source/static/build/docs, actual Windows/native/package and real application lanes, including reverse-tools coverage/mutation. One JavaScript worker cleanup case failed with an ownership-token mismatch; exact main/runtime inputs match, and its isolated original main case passes. Root cause/frequency remain unknown and the original failure is recorded in the team's `/tmp` handoff. One targeted retry executes only the failing shard 2/5 and fail-fast-cancelled shards 4/5 and 5/5, retaining all successful jobs. The targeted retry completes successfully: all 35 jobs and required gates pass for the preceding immutable head. It does not establish the original failure root cause or a product fix. Current-head CI passes all 35 selected jobs in run 38492231165, and normal squash merge is verified at c7105d9723c835b5e0122f5fc6cfbea86265c349. The final tree and all ten PR-owned files match the reviewed versions. Total input worsens for this corpus; uncached input and elapsed time are mixed. No general speedup or broader-domain result is inferred. Own commits retain anonymous metadata and requested factory-droid/Kaoru co-authorship; the squash also credits the maintainer. No release. Merged 2026-10-14 01:34:03 UTC. CI certifies executed integration 4eb2262359cb01f2fb1314db0bc18d3dc9092792 with main 41a2e846470bd2af40dd298d975b1a1d2fef5786. Actual squash parent b8513401e1344a7c0322b8906820db2f81b9be20 and final tree are verified; later upstream changes are not independently requalified by this PR. Anonymous primary email and all requested co-authors are recognized by GitHub. No release. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:05:34 +00:00