[PR #2416] [MERGED] test(perf): distinguish Codex tool context and token accounting #5211

New issue

Closed

opened 2026-10-31 01:04:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:04:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2416
Author: @N0zoM1z0
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/model-context-receipts-20261012

📝 Commits (4)

02bdbf4 test(perf): capture owned Codex tool context and per-response usage
b49cf0a test(perf): add opt-in exact tool discovery experiment
fb33ffd test(perf): verify guided discovery facts and repeated token outcomes
42f02c6 docs(perf): record mixed real Android discovery results

📊 Changes

7 files changed (+582 additions, -28 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+5 -0)
📝 experiments/analysis-efficiency/agent-trial.py (+53 -26)
➕ experiments/analysis-efficiency/codex_context.py (+117 -0)
➕ experiments/analysis-efficiency/model-context.md (+209 -0)
📝 experiments/analysis-efficiency/review-model.py (+22 -1)
➕ experiments/analysis-efficiency/test_codex_context.py (+175 -0)
📝 package.json (+1 -1)

📄 Description

Raw MCP reply bytes and public MCP call counts do not describe actual client discovery, projection and model rounds. Add optional --capture-tool-context to retain owned client tool items and per-response usage. Add separate optional --guide-enabled-tools to supply exact selected function names for narrowly scoped discovery experiments. Existing trials stay ephemeral and their default instructions remain unchanged; REA public APIs/providers/defaults do not change.

Capture filters owned rollouts to tool calls/replies and numeric usage, preserving original IDs/caller values/digests. It excludes account/session metadata, world state, messages, encrypted reasoning, internal passthrough, auth/config and ambient environment. Missing/malformed/symlink records are unavailable/partial, explicit null context-window values remain unknown, and capture failure cannot bypass profile/credential cleanup. Validate guidance prerequisites before acquiring a profile and close the public-event reader explicitly.

Real findings are qualified rather than converted from bytes to tokens:

A committed control/candidate capture pair uses the unchanged ShareX question and two original MCP operations. Both pass selected facts, but candidate input remains higher (56,655 versus 52,656). The control prints about 2 KB after client projection, the candidate about 23 KB; both discover broad truncated metadata. The requested direct label does not establish absence of an executor.
Four same-runtime guidance trials use unguided/guided/guided/unguided order, requested gpt-5.6-terra high, identical artifact/question/two selected tools/providers/capture mode. Both guided metadata outputs are 12,336 bytes without truncation versus 41,709 truncated bytes. All trials preserve source bytes, remove profiles and make two successful dependent MCP calls.
Guided input/uncached input is lower in both observations and time is shorter. The first unguided answer omits classification.container and is not equivalent completed work. The second comparison has two selected-factually acceptable answers after explicitly verifying two accurate extra returned-row counts: input 103,301 → 58,336 (~43.5%), uncached 30,597 → 20,192 (~34.0%), time 60.336 → 50.685 seconds (~16.0%). This is a small one-application signal; no general default recommendation.
The grader validates only known additional native counts against source arrays and integer types. Unknown extra claims, wrong counts, conflicting scopes, boolean/string counts and omitted original coverage still fail. Original failed receipts are retained. All full selected gates still fail unknown final physical resource counters, separately from factual checks.

Validation:

Six focused Python tests pass through the existing CI Python gate: filtering/caller-data preservation, partial/missing records, foreign symlinks, capture-failure cleanup, guidance prerequisites before acquisition and strict additional-count semantics. Simulated CLI tests establish cleanup behavior only.
Complete Python gate, full supported single-thread typecheck, dependency check and unchanged lint pass locally under the resource safeguards. Normal pre-commit and command-scoped checked pre-push run without changing repository hooks. Remote exact-head CI and final manual review remain before merge.
Real Codex capture cohorts are recorded with immutable source/runtime hashes, usage, retries/failed representations and cleanup. A fresh actual NewPipe/JADX SDK workflow and two real Codex Android trials pass selected facts. Guided total input is lower (99,913 → 89,220), but uncached input is higher (19,273 → 26,756) and time is longer (66.797 → 76.936 seconds). This counterexample prevents a general guidance/default recommendation.

Starts from merged main 82cfcee19f18b0eeae9a330ba04e0411cdf85225 in a new isolated worktree; guidance cohort checkpoint e20bd0e2a905f06bc8c56fe04b0e61f3da88636a, current review head 42f02c6f8d6c50265396a2ea99857329c3518d37. Curated findings: experiments/analysis-efficiency/model-context.md. Raw data/profiles/runtimes stay private. One CPU, unchanged 3 GiB early-stop/4 GiB hard memory, no swap and bounded tasks apply; no guard abort is observed, final counters remain unknown.

Continues #2016. Keep draft until available live checks, exact-head CI and manual review complete. No release publication/deployment or plugin SDK.

Historical CI attribution: the old head's Linux shard 4 and Windows lane reproduced the same three Ghidra ELF64 fixture failures on independent main c462; original logs remain in the requested /tmp handoff. Another PR independently repaired those fixtures in main 37b7550ecc916acc8619fa72e5c05af45702733f. This PR is rebased onto that main without changing any of its seven own files (byte-identical comparison verified). Its six Python tests and full checked pre-push pass again. New exact-head CI is running; no historical failure is relabeled as a pass. Measured old runtime/source commits remain pinned and are not relabeled after rebase.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2416 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/model-context-receipts-20261012` --- ### 📝 Commits (4) - [`02bdbf4`](https://github.com/morluto/rea/commit/02bdbf49d1349a34dfc08d4611ffa9ea8fe74805) test(perf): capture owned Codex tool context and per-response usage - [`b49cf0a`](https://github.com/morluto/rea/commit/b49cf0a4c6b5c918eacadcdd723db91049e2f5af) test(perf): add opt-in exact tool discovery experiment - [`fb33ffd`](https://github.com/morluto/rea/commit/fb33ffdfd19d66b1e3eda7cc5f3ac32785ef9fcf) test(perf): verify guided discovery facts and repeated token outcomes - [`42f02c6`](https://github.com/morluto/rea/commit/42f02c6f8d6c50265396a2ea99857329c3518d37) docs(perf): record mixed real Android discovery results ### 📊 Changes **7 files changed** (+582 additions, -28 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+5 -0) 📝 `experiments/analysis-efficiency/agent-trial.py` (+53 -26) ➕ `experiments/analysis-efficiency/codex_context.py` (+117 -0) ➕ `experiments/analysis-efficiency/model-context.md` (+209 -0) 📝 `experiments/analysis-efficiency/review-model.py` (+22 -1) ➕ `experiments/analysis-efficiency/test_codex_context.py` (+175 -0) 📝 `package.json` (+1 -1) </details> ### 📄 Description Raw MCP reply bytes and public MCP call counts do not describe actual client discovery, projection and model rounds. Add optional `--capture-tool-context` to retain owned client tool items and per-response usage. Add separate optional `--guide-enabled-tools` to supply exact selected function names for narrowly scoped discovery experiments. Existing trials stay ephemeral and their default instructions remain unchanged; REA public APIs/providers/defaults do not change. Capture filters owned rollouts to tool calls/replies and numeric usage, preserving original IDs/caller values/digests. It excludes account/session metadata, world state, messages, encrypted reasoning, internal passthrough, auth/config and ambient environment. Missing/malformed/symlink records are unavailable/partial, explicit null context-window values remain unknown, and capture failure cannot bypass profile/credential cleanup. Validate guidance prerequisites before acquiring a profile and close the public-event reader explicitly. Real findings are qualified rather than converted from bytes to tokens: - A committed control/candidate capture pair uses the unchanged ShareX question and two original MCP operations. Both pass selected facts, but candidate input remains higher (56,655 versus 52,656). The control prints about 2 KB after client projection, the candidate about 23 KB; both discover broad truncated metadata. The requested direct label does not establish absence of an executor. - Four same-runtime guidance trials use unguided/guided/guided/unguided order, requested `gpt-5.6-terra` high, identical artifact/question/two selected tools/providers/capture mode. Both guided metadata outputs are 12,336 bytes without truncation versus 41,709 truncated bytes. All trials preserve source bytes, remove profiles and make two successful dependent MCP calls. - Guided input/uncached input is lower in both observations and time is shorter. The first unguided answer omits classification.container and is not equivalent completed work. The second comparison has two selected-factually acceptable answers after explicitly verifying two accurate extra returned-row counts: input 103,301 → 58,336 (~43.5%), uncached 30,597 → 20,192 (~34.0%), time 60.336 → 50.685 seconds (~16.0%). This is a small one-application signal; no general default recommendation. - The grader validates only known additional native counts against source arrays and integer types. Unknown extra claims, wrong counts, conflicting scopes, boolean/string counts and omitted original coverage still fail. Original failed receipts are retained. All full selected gates still fail unknown final physical resource counters, separately from factual checks. Validation: - Six focused Python tests pass through the existing CI Python gate: filtering/caller-data preservation, partial/missing records, foreign symlinks, capture-failure cleanup, guidance prerequisites before acquisition and strict additional-count semantics. Simulated CLI tests establish cleanup behavior only. - Complete Python gate, full supported single-thread typecheck, dependency check and unchanged lint pass locally under the resource safeguards. Normal pre-commit and command-scoped checked pre-push run without changing repository hooks. Remote exact-head CI and final manual review remain before merge. - Real Codex capture cohorts are recorded with immutable source/runtime hashes, usage, retries/failed representations and cleanup. A fresh actual NewPipe/JADX SDK workflow and two real Codex Android trials pass selected facts. Guided total input is lower (99,913 → 89,220), but uncached input is higher (19,273 → 26,756) and time is longer (66.797 → 76.936 seconds). This counterexample prevents a general guidance/default recommendation. Starts from merged main `82cfcee19f18b0eeae9a330ba04e0411cdf85225` in a new isolated worktree; guidance cohort checkpoint `e20bd0e2a905f06bc8c56fe04b0e61f3da88636a`, current review head `42f02c6f8d6c50265396a2ea99857329c3518d37`. Curated findings: `experiments/analysis-efficiency/model-context.md`. Raw data/profiles/runtimes stay private. One CPU, unchanged 3 GiB early-stop/4 GiB hard memory, no swap and bounded tasks apply; no guard abort is observed, final counters remain unknown. Continues #2016. Keep draft until available live checks, exact-head CI and manual review complete. No release publication/deployment or plugin SDK. Historical CI attribution: the old head's Linux shard 4 and Windows lane reproduced the same three Ghidra ELF64 fixture failures on independent main c462; original logs remain in the requested /tmp handoff. Another PR independently repaired those fixtures in main `37b7550ecc916acc8619fa72e5c05af45702733f`. This PR is rebased onto that main without changing any of its seven own files (byte-identical comparison verified). Its six Python tests and full checked pre-push pass again. New exact-head CI is running; no historical failure is relabeled as a pass. Measured old runtime/source commits remain pinned and are not relabeled after rebase. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:04:51 +00:00