morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3428] [MERGED] test(perf): measure real sigrok annotation selection workflows #5739

New issue

Closed

opened 2026-10-31 01:09:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:09:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3428
Author: @N0zoM1z0
Created: 10/14/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/3131-sigrok-selection-20261014

📝 Commits (7)

d7f4e3a test(perf): pin real UART and I2C waveform oracles
9deaeff test(perf): measure authenticated signal selection workflows
250dc8e docs(perf): record mixed real sigrok task costs
86a6d3a test(perf): tighten signal corpus and interval verification
bf44273 docs(perf): verify signal workflows after upstream CI fixes
9932de3 test(perf): bound signal metadata decompression before allocation
801d679 test(perf): require UART intervals to cover sampled data bits

📊 Changes

13 files changed (+1217 additions, -7 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+6 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+1 -0)
📝 experiments/analysis-efficiency/compare.mjs (+9 -3)
📝 experiments/analysis-efficiency/corpus.json (+30 -0)
📝 experiments/analysis-efficiency/measure.mjs (+24 -0)
➕ experiments/analysis-efficiency/review-signal.mjs (+237 -0)
➕ experiments/analysis-efficiency/signal-entry.mjs (+264 -0)
➕ experiments/analysis-efficiency/signal-model-prompt.mjs (+22 -0)
➕ experiments/analysis-efficiency/signal-oracle.py (+211 -0)
➕ experiments/analysis-efficiency/signal-selection-workflows.md (+188 -0)
➕ experiments/analysis-efficiency/test_signal_oracle.py (+149 -0)
📝 tests/evaluation/analysisEfficiencyComparison.test.ts (+30 -4)
➕ tests/evaluation/signalWorkflowSemantics.test.ts (+46 -0)

📄 Description

Summary

Existing sigrok annotation selection returns substantially fewer bytes, but its complete-agent token/time benefit was unmeasured. This checkpoint adds reproducible experiments and independently verifies two real upstream hardware recordings. The observed mixed result supports keeping production APIs/defaults unchanged.

Fixes #3131.

Change and scope

Pin genuine MAX3232E UART and Wii Nunchuk I2C recordings without committing binaries. Add bounded raw-sample oracles, authenticated SDK workflows for full/focused/discovery/pages, and real-Codex prompts/grading that distinguish factual correctness from fixed-plan deviations. Failed attempts and original Evidence coverage remain visible.

Experiment/test/documentation changes only; no production cache, provider capability, target execution, API, default, package, or installation change. Maintainer scope approval for new execution capabilities is not applicable. Existing passive sigrok processes read saved recordings; acquisition is not performed.

Executed evidence

14 real SDK workflows / 22 calls preserved necessary bytes and sample intervals. Focused JSON replies shrank 74.8% UART / 63.0% I2C; these are reply bytes, not model tokens.
Eight actual Codex CLI tasks requested gpt-5.6-terra / high: 8/8 factual passes, 7/8 fixed-plan passes. One extra generic discovery call is retained and excluded from eligible fixed-plan comparisons. Eligible UART total input increased 40.7% / 9.4%; the eligible I2C pair decreased 13.4% but took 79.8% longer. No consistent complete-task token/time improvement is established.
Eight separately bounded producer runs and literal-window measurements separate producer from window cost; decoding and formatting remain combined. Generic caching is deferred.
Cohort runtime: ca7450beffff59325ce3bddc78e181693ce77177. Rebase integration is separate; original measurements were not relabeled.

Curated findings, immutable corpus, resource scope and replay instructions.

Validation performed

python3 experiments/analysis-efficiency/test_signal_oracle.py — eleven oracle admission/framing/allocation regressions passed; forged-count, ZIP64 and metadata-expansion regressions fail on their respective preceding implementations.
npm run test:local -- tests/evaluation/analysisEfficiencyComparison.test.ts tests/evaluation/signalWorkflowSemantics.test.ts — 12 tests passed.
Actual SDK Evidence replay — 24 grading checks passed (valid, recovered failures, wrong bytes, unknown citations, missing limitations); injected grading state is not additional provider/model verification. After review hardening, all 24 cases and eight original model transcripts retained their grades and original costs.
Earlier rebase integration on 3b79af28 passed 17 focused tests, including the previously failing Hopper fixture, and two actual UART/I2C MCP workflows.
Updated to main 6060de32 (PR head 9932de3f): runtime build and npm run test:focused -- tests/evaluation/analysisEfficiencyComparison.test.ts tests/evaluation/signalWorkflowSemantics.test.ts tests/boundary/mcp/signalRecordingMcp.test.ts passed all 16 tests.
npm run test:focused -- tests/boundary/filesystem/staticCheckReports.test.ts passed all eight tests, including the lint inventory/cache-input assertion previously failing upstream.
Two new real focused UART/I2C MCP integrations on that build matched independently read sample bytes and intervals; artifact digests stayed unchanged. These are compatibility checks, not additions to the original cost cohort.
Guarded pre-push dependency, single-threaded typecheck and lint checks passed on final review head 801d6797 (fresh typecheck/lint work). Verified payload-exit peak was 2.30 GiB, with no OOM/abort and cleanup verified; final whole-scope counters remain unknown.
UART range follow-up 801d6797: eleven Python oracle tests and the focused checker test pass. Ten saved real UART calls cover independently sampled data-bit centers; all eight original model transcripts and 24 injected grading cases preserve grades and original costs. Original frame/byte facts remain identical, with additional oracle coordinates only. Single-sample/shortened/shifted ranges are rejected. This replay is not an additional model cohort.

Heavy workloads ran serially with one CPU and memory/ownership guards. Verified cohort payload-exit peaks: SDK 560 MiB, model 848 MiB. Payload cleanup was verified; terminal whole-scope counters remain unknown. Coverage is Linux, two pinned recordings/configurations, the installed sigrok stack and requested model alias. Other platforms/decoders, acquisition, backend model identity and general speedups are unverified.

CI attribution

The earlier Hopper assertion failure was fixed upstream and verified after rebase. The subsequent missing lint-cache inputs for upstream publish-mcp/registry-state files reproduced on unmodified main eab789aed; upstream commit c0aa2e32b now fixes those inputs, and all eight relevant cache-report tests pass on this rebase. No separate CI repair is included in this performance PR.

The unchanged target-free Apple session MCP case passed its job-specific retry; independent macOS reproduction remains unavailable. Previous review findings are resolved. Final head 801d6797 passed 39 checks with zero failures (one superseded readiness run was cancelled), including all five test shards, Apple/Windows lanes and native package baselines. Greptile review passed 5/5 and all five review threads are resolved. Merged as ca1b2f860 on 2026-10-14; #3131 is closed. The latest UART interval review finding was reproduced, fixed and resolved.

Author readiness

Human owner: @N0zoM1z0

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

No release is part of this PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3428 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/14/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/3131-sigrok-selection-20261014` --- ### 📝 Commits (7) - [`d7f4e3a`](https://github.com/morluto/rea/commit/d7f4e3ab2e23cd7d933ded16bf97f7b308f0c5d1) test(perf): pin real UART and I2C waveform oracles - [`9deaeff`](https://github.com/morluto/rea/commit/9deaeff281120b02d0903c1eb6f4c2588edde89c) test(perf): measure authenticated signal selection workflows - [`250dc8e`](https://github.com/morluto/rea/commit/250dc8ecb03fcc680dad1028f9de2ef01f26b86b) docs(perf): record mixed real sigrok task costs - [`86a6d3a`](https://github.com/morluto/rea/commit/86a6d3acdb24cd36955275c6af4615612541ce9c) test(perf): tighten signal corpus and interval verification - [`bf44273`](https://github.com/morluto/rea/commit/bf44273bff09366dc0e0077d34528f1e12745c39) docs(perf): verify signal workflows after upstream CI fixes - [`9932de3`](https://github.com/morluto/rea/commit/9932de3f7e26b3a1414000126264994eb8d97874) test(perf): bound signal metadata decompression before allocation - [`801d679`](https://github.com/morluto/rea/commit/801d6797eabf190cdb3a1123528860899966cb44) test(perf): require UART intervals to cover sampled data bits ### 📊 Changes **13 files changed** (+1217 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+6 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+1 -0) 📝 `experiments/analysis-efficiency/compare.mjs` (+9 -3) 📝 `experiments/analysis-efficiency/corpus.json` (+30 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+24 -0) ➕ `experiments/analysis-efficiency/review-signal.mjs` (+237 -0) ➕ `experiments/analysis-efficiency/signal-entry.mjs` (+264 -0) ➕ `experiments/analysis-efficiency/signal-model-prompt.mjs` (+22 -0) ➕ `experiments/analysis-efficiency/signal-oracle.py` (+211 -0) ➕ `experiments/analysis-efficiency/signal-selection-workflows.md` (+188 -0) ➕ `experiments/analysis-efficiency/test_signal_oracle.py` (+149 -0) 📝 `tests/evaluation/analysisEfficiencyComparison.test.ts` (+30 -4) ➕ `tests/evaluation/signalWorkflowSemantics.test.ts` (+46 -0) </details> ### 📄 Description ## Summary Existing sigrok annotation selection returns substantially fewer bytes, but its complete-agent token/time benefit was unmeasured. This checkpoint adds reproducible experiments and independently verifies two real upstream hardware recordings. The observed mixed result supports keeping production APIs/defaults unchanged. Fixes #3131. ## Change and scope Pin genuine MAX3232E UART and Wii Nunchuk I2C recordings without committing binaries. Add bounded raw-sample oracles, authenticated SDK workflows for full/focused/discovery/pages, and real-Codex prompts/grading that distinguish factual correctness from fixed-plan deviations. Failed attempts and original Evidence coverage remain visible. Experiment/test/documentation changes only; no production cache, provider capability, target execution, API, default, package, or installation change. Maintainer scope approval for new execution capabilities is not applicable. Existing passive sigrok processes read saved recordings; acquisition is not performed. ## Executed evidence - 14 real SDK workflows / 22 calls preserved necessary bytes and sample intervals. Focused JSON replies shrank **74.8% UART / 63.0% I2C**; these are reply bytes, not model tokens. - Eight actual Codex CLI tasks requested `gpt-5.6-terra` / `high`: **8/8 factual passes, 7/8 fixed-plan passes**. One extra generic discovery call is retained and excluded from eligible fixed-plan comparisons. Eligible UART total input increased 40.7% / 9.4%; the eligible I2C pair decreased 13.4% but took 79.8% longer. **No consistent complete-task token/time improvement is established.** - Eight separately bounded producer runs and literal-window measurements separate producer from window cost; decoding and formatting remain combined. Generic caching is deferred. - Cohort runtime: `ca7450beffff59325ce3bddc78e181693ce77177`. Rebase integration is separate; original measurements were not relabeled. [Curated findings, immutable corpus, resource scope and replay instructions](https://github.com/morluto/rea/blob/perf/3131-sigrok-selection-20261014/experiments/analysis-efficiency/signal-selection-workflows.md). ## Validation performed - `python3 experiments/analysis-efficiency/test_signal_oracle.py` — eleven oracle admission/framing/allocation regressions passed; forged-count, ZIP64 and metadata-expansion regressions fail on their respective preceding implementations. - `npm run test:local -- tests/evaluation/analysisEfficiencyComparison.test.ts tests/evaluation/signalWorkflowSemantics.test.ts` — 12 tests passed. - Actual SDK Evidence replay — 24 grading checks passed (valid, recovered failures, wrong bytes, unknown citations, missing limitations); injected grading state is not additional provider/model verification. After review hardening, all 24 cases and eight original model transcripts retained their grades and original costs. - Earlier rebase integration on `3b79af28` passed 17 focused tests, including the previously failing Hopper fixture, and two actual UART/I2C MCP workflows. - Updated to main `6060de32` (PR head `9932de3f`): runtime build and `npm run test:focused -- tests/evaluation/analysisEfficiencyComparison.test.ts tests/evaluation/signalWorkflowSemantics.test.ts tests/boundary/mcp/signalRecordingMcp.test.ts` passed all 16 tests. - `npm run test:focused -- tests/boundary/filesystem/staticCheckReports.test.ts` passed all eight tests, including the lint inventory/cache-input assertion previously failing upstream. - Two new real focused UART/I2C MCP integrations on that build matched independently read sample bytes and intervals; artifact digests stayed unchanged. These are compatibility checks, not additions to the original cost cohort. - Guarded pre-push dependency, single-threaded typecheck and lint checks passed on final review head `801d6797` (fresh typecheck/lint work). Verified payload-exit peak was 2.30 GiB, with no OOM/abort and cleanup verified; final whole-scope counters remain unknown. - UART range follow-up `801d6797`: eleven Python oracle tests and the focused checker test pass. Ten saved real UART calls cover independently sampled data-bit centers; all eight original model transcripts and 24 injected grading cases preserve grades and original costs. Original frame/byte facts remain identical, with additional oracle coordinates only. Single-sample/shortened/shifted ranges are rejected. This replay is not an additional model cohort. Heavy workloads ran serially with one CPU and memory/ownership guards. Verified cohort payload-exit peaks: SDK 560 MiB, model 848 MiB. Payload cleanup was verified; terminal whole-scope counters remain unknown. Coverage is Linux, two pinned recordings/configurations, the installed sigrok stack and requested model alias. Other platforms/decoders, acquisition, backend model identity and general speedups are unverified. ## CI attribution The earlier Hopper assertion failure was fixed upstream and verified after rebase. The subsequent missing lint-cache inputs for upstream `publish-mcp/registry-state` files reproduced on unmodified main `eab789aed`; upstream commit `c0aa2e32b` now fixes those inputs, and all eight relevant cache-report tests pass on this rebase. No separate CI repair is included in this performance PR. The unchanged target-free Apple session MCP case passed its job-specific retry; independent macOS reproduction remains unavailable. Previous review findings are resolved. Final head `801d6797` passed 39 checks with zero failures (one superseded readiness run was cancelled), including all five test shards, Apple/Windows lanes and native package baselines. Greptile review passed 5/5 and all five review threads are resolved. Merged as `ca1b2f860` on 2026-10-14; #3131 is closed. The latest UART interval review finding was reproduced, fixed and resolved. ## Author readiness Human owner: @N0zoM1z0 - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. No release is part of this PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:09:04 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5739

No description provided.

Rows
Columns