[GH-ISSUE #3131] test(perf): measure sigrok decoder selection and repeated annotation pages #1090

New issue

Closed

opened 2026-10-15 18:30:59 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:30:59 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3131

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. This is a passive recording efficiency task, not an implementation of the separate process-cleanup defects.

Analyst question and outcome

For a named digital protocol question on a saved signal recording, do existing decoder/channel/annotation selectors and output windows reduce model input and time while preserving the same decoded facts?

Existing evidence and decision boundary

Existing recorded-signal operations already expose decoder/channel/annotation selection and offset/limit. Display windows count rendered lines, not necessarily annotations or samples, and may rerun decoding. #2016 did not measure a real sigrok task cohort; no savings or retained-decoder cache are established.

Bounded work

Use two small, digest-pinned public upstream signal recordings for supported decoder configurations. Pin sigrok/decoder versions, channels, sample/time metadata and all parameters; no device attachment or new acquisition.
Define one concrete protocol-fact question per recording. Obtain an independent raw/sample or decoder-reference oracle where feasible and distinguish decoder observations from inferred hardware behavior.
Compare existing full necessary output against explicit decoder/annotation selection and first/follow-up pages. Keep annotation-to-line/sample/time relationships, partial decoding and unavailable counts explicit; a completed display page is not a complete protocol decode.
Run repeated SDK pairs and representative actual Codex pairs only for viable equivalent workflows. Measure repeated decoding and output rendering separately. If reuse is warranted, first specify bounded retained-output ownership and cancellation/cleanup; do not add a generic cache here.

Acceptance

Public recordings, decoder/channel identities, exact questions and original byte digests are pinned.
Repeated public CLI/MCP workflows preserve necessary protocol facts and truthful sample/annotation/display coverage.
A representative repeated model cohort reports token/time/error costs and actual provider lifecycle/resource scope.
Document whether existing controls suffice, whether repeated decoding merits a separate change, or why no optimization is justified.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3131 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. This is a passive recording efficiency task, not an implementation of the separate process-cleanup defects. ## Analyst question and outcome For a named digital protocol question on a saved signal recording, do existing decoder/channel/annotation selectors and output windows reduce model input and time while preserving the same decoded facts? ## Existing evidence and decision boundary Existing recorded-signal operations already expose decoder/channel/annotation selection and offset/limit. Display windows count rendered lines, not necessarily annotations or samples, and may rerun decoding. #2016 did not measure a real sigrok task cohort; no savings or retained-decoder cache are established. ## Bounded work 1. Use two small, digest-pinned public upstream signal recordings for supported decoder configurations. Pin sigrok/decoder versions, channels, sample/time metadata and all parameters; no device attachment or new acquisition. 2. Define one concrete protocol-fact question per recording. Obtain an independent raw/sample or decoder-reference oracle where feasible and distinguish decoder observations from inferred hardware behavior. 3. Compare existing full necessary output against explicit decoder/annotation selection and first/follow-up pages. Keep annotation-to-line/sample/time relationships, partial decoding and unavailable counts explicit; a completed display page is not a complete protocol decode. 4. Run repeated SDK pairs and representative actual Codex pairs only for viable equivalent workflows. Measure repeated decoding and output rendering separately. If reuse is warranted, first specify bounded retained-output ownership and cancellation/cleanup; do not add a generic cache here. ## Acceptance - [x] Public recordings, decoder/channel identities, exact questions and original byte digests are pinned. - [x] Repeated public CLI/MCP workflows preserve necessary protocol facts and truthful sample/annotation/display coverage. - [x] A representative repeated model cohort reports token/time/error costs and actual provider lifecycle/resource scope. - [x] Document whether existing controls suffice, whether repeated decoding merits a separate change, or why no optimization is justified. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi

2026-10-15 18:30:59 +00:00

closed this issue
added the
performance

design

P1: high

area: domain

roadmap

enhancement
labels

bsakaguchi commented

2026-10-15 18:31:00 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

Measured checkpoint: draft PR #3428 contains immutable public UART/I2C corpus identities, bounded independent raw-sample oracles, existing-selector SDK/model workflows and curated findings.

Fourteen actual SDK workflows (22 calls) preserved bytes/sample intervals; focused replies decreased 74.8% / 63.0%. Eight actual Codex tasks requested gpt-5.6-terra / high: all eight factual answers passed; one extra generic discovery call failed the predeclared fixed single-call plan and its cost remains recorded. Eligible pairs show mixed total-input/time results, so no consistent whole-task efficiency gain is established and no production cache/default/API change is justified.

Repeated producer runs and separate window extraction measurements delimit what is observed: producer startup/decoding/formatting remain combined; window extraction is small; pages rerun decoding. Any future retained-output proposal needs demonstrated task benefit and explicit bounded ownership rather than a generic decoder cache.

The cost cohort remains pinned to ca7450be; latest-main (44d98de) build, 16 focused tests and two real MCP integrations are separate verification. Private binaries/raw transcripts remain uncommitted; the source and curated reproduction/results are in the PR. This is a measured no-change decision within this issue's scope.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): Measured checkpoint: draft PR #3428 contains immutable public UART/I2C corpus identities, bounded independent raw-sample oracles, existing-selector SDK/model workflows and curated findings. Fourteen actual SDK workflows (22 calls) preserved bytes/sample intervals; focused replies decreased 74.8% / 63.0%. Eight actual Codex tasks requested `gpt-5.6-terra` / `high`: all eight factual answers passed; one extra generic discovery call failed the predeclared fixed single-call plan and its cost remains recorded. Eligible pairs show mixed total-input/time results, so **no consistent whole-task efficiency gain is established and no production cache/default/API change is justified**. Repeated producer runs and separate window extraction measurements delimit what is observed: producer startup/decoding/formatting remain combined; window extraction is small; pages rerun decoding. Any future retained-output proposal needs demonstrated task benefit and explicit bounded ownership rather than a generic decoder cache. The cost cohort remains pinned to `ca7450be`; latest-main (`44d98de`) build, 16 focused tests and two real MCP integrations are separate verification. Private binaries/raw transcripts remain uncommitted; the source and [curated reproduction/results](https://github.com/morluto/rea/blob/perf/3131-sigrok-selection-20261014/experiments/analysis-efficiency/signal-selection-workflows.md) are in the PR. This is a measured no-change decision within this issue's scope.

bsakaguchi referenced this issue

2026-10-16 21:09:19 +00:00

[GH-ISSUE #4307] [Feature] Inspect the dyld shared cache and resolve system library loads #1711

bsakaguchi referenced this issue

2026-10-17 21:08:29 +00:00

[PR #1090] [MERGED] fix(apple): admit only compatible dyld subcache mappings #2638

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:55 +00:00