[GH-ISSUE #3130] test(perf): evaluate larger and non-HTTP captures with existing field discovery #1091

New issue

Closed

opened 2026-10-15 18:30:59 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:30:59 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3130

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. Broader MCP performance remains owned by #1065; this is passive saved-capture measurement, not a new provider or API-surface feature.

Analyst question and outcome

For one fixed packet/transaction question, does discovery-guided exact field/filter selection or retained paging reduce actual model input and answer time on larger or non-HTTP saved captures?

Existing evidence and decision boundary

The two small HTTP recordings in #2640 are a negative control: summary/page returns more than twice the SDK JSON and increases total model input by 50.54–50.92%. Preserve complete defaults. Curated measurements remain pinned to their original runtime.

Existing TShark operations already expose fields, filter, decode-as, pass mode and output windows; protocol/field discovery is available. Output paging may still dissect the input. None of these facts establishes a repeated-work saving on another question.

Bounded work

Refresh and pin current main, installed Wireshark identity and actual advertised schemas. Select at most three digest-pinned public recordings: one larger HTTP/application recording and two non-HTTP protocol recordings with independently checkable facts. Do not acquire live traffic.
Write an exact question and oracle for each selected case. Compare complete/necessary baseline facts with existing discovery-assisted field selection and, where useful, retained pages. Count discovery, extra schemas/RPCs, repeated dissection and recovery calls. Preserve matching-row versus source coverage, original order and null/unknown fields; page exhaustion is not network completeness.
Run at least two interleaved SDK pairs per comparison. For viable equivalent workflows, run two actual Codex pairs for a representative question. Record negative/unavailable branches rather than replacing targets or providers to manufacture a win.
Measure first/follow-up work separately. Propose retained decoder-output reuse only if repeated dissection is demonstrably material and existing controls are insufficient; cache design/implementation is a separate decision.

Acceptance

Pin corpora, runtime/provider/client identities, questions, independent oracles and exact treatment arguments.
Complete repeated SDK and representative real-model comparisons with equivalent supported facts and all discovery/retry costs.
Quantify repeated dissection separately from output-window costs and state unavailable stage/resource measurements.
Record the question-specific decision; preserve the small-HTTP negative control and complete defaults. Any justified code change is separately reviewed with relevant CLI/MCP and old-Evidence checks.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3130 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. Broader MCP performance remains owned by #1065; this is passive saved-capture measurement, not a new provider or API-surface feature. ## Analyst question and outcome For one fixed packet/transaction question, does discovery-guided exact field/filter selection or retained paging reduce actual model input and answer time on larger or non-HTTP saved captures? ## Existing evidence and decision boundary The two small HTTP recordings in #2640 are a negative control: summary/page returns more than twice the SDK JSON and increases total model input by 50.54–50.92%. Preserve complete defaults. [Curated measurements](https://github.com/morluto/rea/blob/c7105d9723c835b5e0122f5fc6cfbea86265c349/experiments/analysis-efficiency/pcap-small-captures.md) remain pinned to their original runtime. Existing TShark operations already expose fields, filter, decode-as, pass mode and output windows; protocol/field discovery is available. Output paging may still dissect the input. None of these facts establishes a repeated-work saving on another question. ## Bounded work 1. Refresh and pin current main, installed Wireshark identity and actual advertised schemas. Select at most three digest-pinned public recordings: one larger HTTP/application recording and two non-HTTP protocol recordings with independently checkable facts. Do not acquire live traffic. 2. Write an exact question and oracle for each selected case. Compare complete/necessary baseline facts with existing discovery-assisted field selection and, where useful, retained pages. Count discovery, extra schemas/RPCs, repeated dissection and recovery calls. Preserve matching-row versus source coverage, original order and null/unknown fields; page exhaustion is not network completeness. 3. Run at least two interleaved SDK pairs per comparison. For viable equivalent workflows, run two actual Codex pairs for a representative question. Record negative/unavailable branches rather than replacing targets or providers to manufacture a win. 4. Measure first/follow-up work separately. Propose retained decoder-output reuse only if repeated dissection is demonstrably material and existing controls are insufficient; cache design/implementation is a separate decision. ## Acceptance - [ ] Pin corpora, runtime/provider/client identities, questions, independent oracles and exact treatment arguments. - [ ] Complete repeated SDK and representative real-model comparisons with equivalent supported facts and all discovery/retry costs. - [ ] Quantify repeated dissection separately from output-window costs and state unavailable stage/resource measurements. - [ ] Record the question-specific decision; preserve the small-HTTP negative control and complete defaults. Any justified code change is separately reviewed with relevant CLI/MCP and old-Evidence checks. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi

2026-10-15 18:30:59 +00:00

closed this issue
added the
performance

design

P1: high

area: domain

roadmap

enhancement
labels

bsakaguchi commented

2026-10-15 18:31:00 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

The larger/non-HTTP experiment is complete on pinned main ec67aebed02dbd20fa5c7703fcd183ad0b1c343c (Linux TShark/capinfos 4.0.17, Node 24.18.0). The implementation uses the existing public controls; production APIs/defaults/provider limits and Evidence/session ownership remain unchanged.

Corpus and question scope:

Official Wireshark PostgreSQL JDBC saved session: 590,093 compressed / 4,906,316 expanded bytes, 18,472 frames. The fixed question covers original frames 1–2,000, first twelve port-5432 TCP packets and the matching count. An independent raw-header oracle establishes 1,337 matches.
Official DNS sample: 38 frames, six nonzero-RCODE responses with exact IDs/names/types.
Official DHCP sample: four frames, preserving omitted server ID and the distinct Discover/Offer versus Request/ACK transaction IDs. These two small non-HTTP samples are controls; larger DNS/DHCP performance remains unverified.

Nineteen actual SDK workflows include two interleaved repetitions per arm/recording and a first/next/repeated page sequence. Focused reply JSON decreases 42.29% / 53.71% / 12.71% for PostgreSQL / DNS / DHCP. Discovery adds glossary work; the DHCP discovery+focused reply increases 44.08%. All requested SDK facts match the independent oracle, including original frame order, input/filter/window scope and omission semantics. Two pairs on each previous HTTP control remain negative summary/page controls on this new runtime; historic model results keep their original pin.

Six actual Codex CLI 0.161.0 trials use an isolated temporary MCP configuration and requested gpt-5.6-terra / high. The two eligible baseline/focused pairs show input changes -9.86% / +1.82% and time changes +37.70% / -19.30%, so no stable total-token or speed gain is established. The complete discovery trial adds 29.90% input and 58.63% elapsed time. The second discovery attempt returns all correct facts but omits the required discovery call: factual pass, workflow failure, actual costs retained. An earlier rejected-approval setup preflight and its cost are also retained separately.

Decision: document question-scoped input limits, fields, filters and output windows, and keep discovery available for unfamiliar installed fields. Do not force discovery or alter defaults, and do not introduce a generic cache on this evidence. The three-page diagnostic confirms repeated dissection, but this narrow cost alone is insufficient to justify more lifecycle/identity machinery.

Heavy jobs ran serially with one CPU, 3 GiB early stop / 4 GiB hard memory and no swap. Non-HTTP SDK/model payload-exit peaks were below 0.90 GiB, with verified payload cleanup and no recorded limit/OOM events in that observed phase. Final whole-scope counters remain unknown. Captures, raw results, credentials and configurations remain private. The independent large-error-context product concern is logged under /tmp for later repair, outside this issue.

The PR contains the reproducible driver, independent oracle, actual-result rubric, curated complete measurements and bounded-query guidance. This is an evidence-backed negative/mixed result, not a claim of universal performance gains.

Implementation and curated report: #3232.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): The larger/non-HTTP experiment is complete on pinned main `ec67aebed02dbd20fa5c7703fcd183ad0b1c343c` (Linux TShark/capinfos 4.0.17, Node 24.18.0). The implementation uses the existing public controls; production APIs/defaults/provider limits and Evidence/session ownership remain unchanged. Corpus and question scope: - Official Wireshark PostgreSQL JDBC saved session: 590,093 compressed / 4,906,316 expanded bytes, 18,472 frames. The fixed question covers original frames 1–2,000, first twelve port-5432 TCP packets and the matching count. An independent raw-header oracle establishes 1,337 matches. - Official DNS sample: 38 frames, six nonzero-RCODE responses with exact IDs/names/types. - Official DHCP sample: four frames, preserving omitted server ID and the distinct Discover/Offer versus Request/ACK transaction IDs. These two small non-HTTP samples are controls; larger DNS/DHCP performance remains unverified. Nineteen actual SDK workflows include two interleaved repetitions per arm/recording and a first/next/repeated page sequence. Focused reply JSON decreases 42.29% / 53.71% / 12.71% for PostgreSQL / DNS / DHCP. Discovery adds glossary work; the DHCP discovery+focused reply increases 44.08%. All requested SDK facts match the independent oracle, including original frame order, input/filter/window scope and omission semantics. Two pairs on each previous HTTP control remain negative summary/page controls on this new runtime; historic model results keep their original pin. Six actual Codex CLI 0.161.0 trials use an isolated temporary MCP configuration and requested `gpt-5.6-terra` / high. The two eligible baseline/focused pairs show input changes -9.86% / +1.82% and time changes +37.70% / -19.30%, so no stable total-token or speed gain is established. The complete discovery trial adds 29.90% input and 58.63% elapsed time. The second discovery attempt returns all correct facts but omits the required discovery call: factual pass, workflow failure, actual costs retained. An earlier rejected-approval setup preflight and its cost are also retained separately. Decision: document question-scoped input limits, fields, filters and output windows, and keep discovery available for unfamiliar installed fields. Do not force discovery or alter defaults, and do not introduce a generic cache on this evidence. The three-page diagnostic confirms repeated dissection, but this narrow cost alone is insufficient to justify more lifecycle/identity machinery. Heavy jobs ran serially with one CPU, 3 GiB early stop / 4 GiB hard memory and no swap. Non-HTTP SDK/model payload-exit peaks were below 0.90 GiB, with verified payload cleanup and no recorded limit/OOM events in that observed phase. Final whole-scope counters remain unknown. Captures, raw results, credentials and configurations remain private. The independent large-error-context product concern is logged under `/tmp` for later repair, outside this issue. The PR contains the reproducible driver, independent oracle, actual-result rubric, curated complete measurements and bounded-query guidance. This is an evidence-backed negative/mixed result, not a claim of universal performance gains. Implementation and curated report: #3232.

bsakaguchi referenced this issue

2026-10-17 21:08:28 +00:00

[PR #1087] [MERGED] fix(javascript): retain NodeNext artifact source facts #2635

bsakaguchi referenced this issue

2026-10-17 21:08:28 +00:00

[PR #1088] [MERGED] fix(mcp): retain complete application results across response limits #2636

bsakaguchi referenced this issue

2026-10-17 21:08:29 +00:00

[PR #1091] [MERGED] fix(ci): restore generated shard inputs and cleanup assertions #2639

bsakaguchi referenced this issue

2026-10-17 21:08:32 +00:00

[PR #1092] [MERGED] fix(javascript): preserve HTML carriage return source ranges #2640