morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5598] [Bug] Teach recovery before client deadlines hide REA errors (RepoPrompt CE comparison) #3397

New issue

Closed

opened 2026-10-31 00:39:55 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 00:39:55 +00:00

Owner

Originally created by @gensokyo-dev on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5598

Area

MCP or agent integration; transport or result projection.

Observed behavior

Follow-up to #5431 / #5470. REA's canonical errors now recover well in actual Codex trials, but a client-side deadline can expire before any REA recovery message arrives. In two real Codex runs with mcp_servers.rea.tool_timeout_sec=1, the agent repeated the exact failed analysis once and did not identify the adjustable client deadline. Neither run fabricated Evidence, but neither completed the task or gave the actionable configuration remedy. A 60-second control on the same real target succeeded.

Reproduction actually run

REA source eac296a7da3356f1c5342155d635975ddd89cba6; Linux x64, Node 24.18.0, codex-cli 0.161.0, model gpt-6.1-sol. Fresh standalone read-only Codex sessions, explicit approvals for the named static tools, no target execution or configuration writes.

Target: installed Commander 14.0.3 (12 relevant files; 164,187 source bytes). Call analyze_javascript_application with its absolute directory as input_path, detail:"summary", max_heap_mb:128, and bounded semantic analysis. Change only the MCP client's tool_timeout_sec from 60 to 1. Observed host diagnostic:

timed out awaiting tools/call after 1000ms

This is not REA's provider_timeout: no REA error/recovery/Evidence reaches the model. Client completion does not establish backend cancellation or cleanup. Prior server-error trials recovered 9/9; one server-timeout recovery also introduced an avoidable invalid source_paths:[] retry (omission, not an empty array, means no filter).

Expected behavior / impact

Teach transport-timeout recovery before the call: identify the client deadline, do not repeat unchanged work, preserve unknown execution/cleanup state, and name an actionable host setting or authorized fallback. Separately, a returned JavaScript server timeout should identify its actual analysis_timeout_ms control and valid work-reduction alternatives. Never suggest changing a client deadline as a fix for an already returned provider timeout. No permission broadening, engine installs, or target substitution.

Frequency: 2/2 client-deadline probes, with one unchanged retry each. Other hosts/models and real native engines: NOT_ENOUGH_INFO; not tested in these probes. Two historical pre-#5470 server-error controls also recovered, so these observations do not establish a causal improvement in general recovery success rate.

RepoPrompt CE: cloned-source comparison

Cloned https://github.com/repoprompt/repoprompt-ce.git, pinned fce93ca0ef4b460fc512bba7a4e8462837aabf97 (2026-10-20). Source-derived observations, not a runtime evaluation of its macOS application or a claim about the proprietary product:

Mechanism	Why it matters / REA comparison
State-specific error projection distinguishes admission timeout, settled cancellation, detached work and unresponsive cleanup; busy replies preserve origin identity and may expose `retry_after_ms`.	Recovery follows observed lifecycle state, not a generic timeout/retry slogan. REA already retains cleanup/partial Evidence; keep those semantics authoritative.
Lane failures return `next_call` where an executable remedy exists, and distinguish replaying a receipt from retrying creation.	A recovery hint should name an actually available operation/control and avoid duplicate effects. Do not invent a universal next tool or imply a session snapshot proves every target-free worker stopped.
Single-tool schema help, schema renderer, and filtered `tools --schema` use current tool definitions.	Repair from the connected contract rather than past release knowledge; do not return a full catalog on every error. REA already exposes complete MCP schemas and command help; prefer them over adding another discovery tool.
Timeout policy reserves execution, cancellation and response delivery separately; CLI deadline selection adds a delivery margin to semantic waits while honoring explicit caller timeouts.	The outer deadline must leave room for a useful server response. REA cannot change an arbitrary host's timeout from a response the host has already discarded.
Workflow prompts teach schema lookup and expensive-call timeout requirements upfront.	Recovery needs a pre-call teaching surface, not only better error prose. REA initialize instructions and its shipped skill are the appropriate existing surfaces. Do not copy RepoPrompt's task-specific 45-minute timeout into REA.

Additional limit: CE includes permissive argument repair and a Claude transcript filter; neither is evidence that unknown-tool recovery reaches an agent. Do not copy those mechanisms into REA's strict input contracts or hide failed calls in acceptance results.

Proposed focused change and acceptance

Add concise initialization guidance for input-contract recovery, returned provider deadlines versus host deadlines, and safe retries. Keep detailed host-specific instructions in the existing recovery skill reference.
At the JavaScript timeout producer, identify analysis_timeout_ms and valid source-selection semantics; share the existing tagged error projection across CLI/MCP. Preserve caller budgets, partial Evidence and failed-cleanup ownership.
Deterministic public-boundary tests: initialization guidance arrives before calls; actual JavaScript timeout has actionable remediation and retained Evidence remains usable; existing schema and error-budget guards still pass.
Actual Codex CLI A/B on the same base/target/model: 1-second host deadline, successful control, malformed/multiple fields, server timeout, stale Evidence and unavailable-provider safe stop. Verify actual calls and Evidence, not keyword success alone. For an immutable short client deadline, correct diagnosis/actionable stop is success, not pretending analysis completed.
Report whole-task tokens (including schemas/retries), wall/CPU time and sampled peak process-family RSS; serial trials, bounded worker heap. Separate cold startup from warm runs and label unmeasured costs.

Before submitting

Searched related open/closed issues; this is a follow-up, not a replacement for #5431, #1587 or #5273.
No credentials, private target data or binary artifacts included.

Originally created by @gensokyo-dev on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5598 ## Area MCP or agent integration; transport or result projection. ## Observed behavior Follow-up to #5431 / #5470. REA's canonical errors now recover well in actual Codex trials, but a **client-side deadline can expire before any REA recovery message arrives**. In two real Codex runs with `mcp_servers.rea.tool_timeout_sec=1`, the agent repeated the exact failed analysis once and did not identify the adjustable client deadline. Neither run fabricated Evidence, but neither completed the task or gave the actionable configuration remedy. A 60-second control on the same real target succeeded. ## Reproduction actually run REA source `eac296a7da3356f1c5342155d635975ddd89cba6`; Linux x64, Node 24.18.0, codex-cli 0.161.0, model `gpt-6.1-sol`. Fresh standalone read-only Codex sessions, explicit approvals for the named static tools, no target execution or configuration writes. Target: installed Commander 14.0.3 (12 relevant files; 164,187 source bytes). Call `analyze_javascript_application` with its absolute directory as `input_path`, `detail:"summary"`, `max_heap_mb:128`, and bounded semantic analysis. Change only the MCP client's `tool_timeout_sec` from 60 to 1. Observed host diagnostic: ```text timed out awaiting tools/call after 1000ms ``` This is **not** REA's `provider_timeout`: no REA error/recovery/Evidence reaches the model. Client completion does not establish backend cancellation or cleanup. Prior server-error trials recovered 9/9; one server-timeout recovery also introduced an avoidable invalid `source_paths:[]` retry (omission, not an empty array, means no filter). ## Expected behavior / impact Teach transport-timeout recovery before the call: identify the client deadline, do not repeat unchanged work, preserve unknown execution/cleanup state, and name an actionable host setting or authorized fallback. Separately, a returned JavaScript server timeout should identify its actual `analysis_timeout_ms` control and valid work-reduction alternatives. Never suggest changing a client deadline as a fix for an already returned provider timeout. No permission broadening, engine installs, or target substitution. Frequency: 2/2 client-deadline probes, with one unchanged retry each. Other hosts/models and real native engines: NOT_ENOUGH_INFO; not tested in these probes. Two historical pre-#5470 server-error controls also recovered, so these observations do not establish a causal improvement in general recovery success rate. ## RepoPrompt CE: cloned-source comparison Cloned `https://github.com/repoprompt/repoprompt-ce.git`, pinned `fce93ca0ef4b460fc512bba7a4e8462837aabf97` (2026-10-20). Source-derived observations, **not** a runtime evaluation of its macOS application or a claim about the proprietary product: | Mechanism | Why it matters / REA comparison | | --- | --- | | [State-specific error projection](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPrompt/Infrastructure/MCP/MCPConnectionManager.swift#L15465) distinguishes admission timeout, settled cancellation, detached work and unresponsive cleanup; busy replies preserve origin identity and may expose `retry_after_ms`. | Recovery follows observed lifecycle state, not a generic timeout/retry slogan. REA already retains cleanup/partial Evidence; keep those semantics authoritative. | | [Lane failures](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPrompt/Infrastructure/MCP/Agent/AgentSessionLaneMCPToolService.swift#L115) return `next_call` where an executable remedy exists, and distinguish replaying a receipt from retrying creation. | A recovery hint should name an actually available operation/control and avoid duplicate effects. Do not invent a universal next tool or imply a session snapshot proves every target-free worker stopped. | | [Single-tool schema help](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPromptMCPCore/CommandRunner/MCPCommandRunner.swift#L312), [schema renderer](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPromptMCPCore/CommandRunner/ToolSchemaRenderer.swift#L357), and filtered `tools --schema` use current tool definitions. | Repair from the connected contract rather than past release knowledge; do not return a full catalog on every error. REA already exposes complete MCP schemas and command help; prefer them over adding another discovery tool. | | [Timeout policy](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPromptShared/MCP/MCPTimeoutPolicy.swift#L59) reserves execution, cancellation and response delivery separately; [CLI deadline selection](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPromptMCPCore/Interactive/InteractiveMCPClientSession.swift#L1519) adds a delivery margin to semantic waits while honoring explicit caller timeouts. | The outer deadline must leave room for a useful server response. REA cannot change an arbitrary host's timeout from a response the host has already discarded. | | [Workflow prompts](https://github.com/repoprompt/repoprompt-ce/blob/fce93ca0ef4b460fc512bba7a4e8462837aabf97/Sources/RepoPromptShared/Workflows/WorkflowPromptVariant.swift#L42) teach schema lookup and expensive-call timeout requirements upfront. | Recovery needs a pre-call teaching surface, not only better error prose. REA initialize instructions and its shipped skill are the appropriate existing surfaces. Do not copy RepoPrompt's task-specific 45-minute timeout into REA. | Additional limit: CE includes permissive argument repair and a Claude transcript filter; neither is evidence that unknown-tool recovery reaches an agent. Do not copy those mechanisms into REA's strict input contracts or hide failed calls in acceptance results. ## Proposed focused change and acceptance - Add concise initialization guidance for input-contract recovery, returned provider deadlines versus host deadlines, and safe retries. Keep detailed host-specific instructions in the existing recovery skill reference. - At the JavaScript timeout producer, identify `analysis_timeout_ms` and valid source-selection semantics; share the existing tagged error projection across CLI/MCP. Preserve caller budgets, partial Evidence and failed-cleanup ownership. - Deterministic public-boundary tests: initialization guidance arrives before calls; actual JavaScript timeout has actionable remediation and retained Evidence remains usable; existing schema and error-budget guards still pass. - Actual Codex CLI A/B on the same base/target/model: 1-second host deadline, successful control, malformed/multiple fields, server timeout, stale Evidence and unavailable-provider safe stop. Verify actual calls and Evidence, not keyword success alone. For an immutable short client deadline, correct diagnosis/actionable stop is success, not pretending analysis completed. - Report whole-task tokens (including schemas/retries), wall/CPU time and sampled peak process-family RSS; serial trials, bounded worker heap. Separate cold startup from warm runs and label unmeasured costs. ## Before submitting - [x] Searched related open/closed issues; this is a follow-up, not a replacement for #5431, #1587 or #5273. - [x] No credentials, private target data or binary artifacts included.

bsakaguchi

2026-10-31 00:39:55 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-31 00:39:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 20, 2026):

Cloned-source comparison and implemented outcome

RepoPrompt CE remains pinned at fce93ca0ef4b460fc512bba7a4e8462837aabf97. The useful pattern is state-aware, executable recovery plus pre-call teaching, not longer generic errors. Its actual code paths and limits are linked in the issue body. No RepoPrompt macOS runtime or proprietary-product equivalence is claimed.

PR #5608 implements that pattern using REA's existing initialization, named tool contract, producer-owned error remediation, and shipped skill. No extra discovery tool or full schema on every failure. Generic errors do not assume an adjustable budget; only a knowledgeable producer names a control.

Actual Codex CLI trials

Baseline 0ba4d6f43ad610dacc861a1548be1a5c4712cd61; layered implementation 24e32b26d5d5d57eb5dd411d0d48b2034a9e2790. Same real Commander 14.0.3 target, codex-cli 0.161.0, gpt-6.1-sol, Linux x64, Node 24.18.0. Static-only, fresh Codex/MCP processes, explicitly approved named tools, semantic/heap bounds, sequential trials; no target execution, installs or config writes.

Final layered cases	Observed outcome
1-second immutable client deadline, 4 trials	4/4 correctly stopped after one call; named `mcp_servers.rea.tool_timeout_sec`; no unchanged retry, invented Evidence, or assumed cleanup completion. This is correct recovery diagnosis, not completed analysis under an impossible fixed deadline.
Returned 1ms JavaScript server timeout, 2 trials	2/2 recovered in two calls by changing only `analysis_timeout_ms`; preserved path, summary, heap and semantic limits. No extra validation error or empty `source_paths`.
Normal control, 2 trials	2/2 succeeded in one call, preserving actual partial/unknown coverage.

Input repairs were independently checked against the live SDK-advertised Draft 2020-12 schemas; returned Evidence and citations, caller bounds, forbidden actions and object-order-independent failed-call repeats were checked. Raw answers/calls were reviewed, not just success keywords. Earlier iterations additionally exercised multiple invalid fields, stale Evidence and a deliberate missing-Ghidra-install preflight with honest stop/close (not real Ghidra qualification).

Counterexamples retained, not discarded: the baseline client trial repeated unchanged work; the first candidate stopped twice because it incorrectly made Evidence inspection mandatory when the reader was hidden; its two client stops did not name the exact setting. An initialization-only candidate still made one unchanged client retry. These motivated optional Evidence reuse and the targeted tool-description warning. All 28 initial trials, including intermediate failures, are retained. Small stochastic samples do not prove universal compliance or a causal recovery-rate/latency improvement.

Review follow-up af29cc3ef81d3b983930ffe1b8b162827fc590f7 also fixes generic advice for non-adjustable budgets and explicitly names rea analyze-javascript-application rather than suggesting a flag unsupported by the analyze shorthand. Both new regression assertions failed before the correction. The final head passed all 79 focused tests, npm run check, generated-doc checks and skill validation. Two additional actual Codex trials on that head both passed: client timeout stopped without retry and named its setting; server timeout corrected its MCP budget and completed in two calls. 30 total trials are retained; the final layered eight plus post-review two all satisfy the stated oracle.

Measured costs

Final 8 trials: 741,148 input tokens (587,136 cached), 4,825 output tokens; 255.62s summed wall time; 112.50s CPU; sampled peak family RSS 1,255.25 MiB.
Initial 28 trials: 3,083,977 input tokens (2,550,912 cached), 17,793 output tokens; 948.59s summed wall time; 412.44s CPU; sampled peak family RSS 1,290.05 MiB. No trial process hit OOM.
Including the two post-review trials: 3,370,627 input tokens (2,792,064 cached), 19,291 output; 1,032.06s wall sum; 444.47s CPU; the same 1,290.05 MiB sampled peak. Additional two alone: 83.47s wall, 32.03s CPU, 979.09 MiB sampled peak.
Whole-task token usage includes selected schemas, follow-ups and retries. Startup guidance adds context: initialize instructions are 1,817 bytes; the complete catalog is 3,404,253 bytes. These byte counts are not token savings. Control-task token/latency variability prevents a general speedup claim.
Fresh processes are not cold caches: filesystem/build/model caches were warm or uncontrolled. CPU is GNU time child-family CPU, excluding sampling/research/build/review overhead. Memory is descendant RSS sampled every 500ms, not cgroup peak; shared pages may be double-counted and short spikes missed.
The temporary sampler could skip its final refresh if an interval sample was pending; nonzero immediate process counts are therefore retained as inconclusive. Separate post-run task-specific command/cwd audits at 16:00:34 UTC and 16:10:21 UTC found no matches. This is not retroactive immediate-ownership proof.

Deterministic coverage: initialization delivery fails on the unmodified runtime; CLI/MCP timeout, retained Evidence and continued-work tests pass; existing error budgets/provenance preserved. Review additionally caught and corrected generic advice implying nonexistent timeout controls on fixed-budget operations. No new cache, provider, target authority or analysis semantics.

@gensokyo-dev commented on GitHub (Oct 20, 2026): ## Cloned-source comparison and implemented outcome RepoPrompt CE remains pinned at `fce93ca0ef4b460fc512bba7a4e8462837aabf97`. The useful pattern is **state-aware, executable recovery plus pre-call teaching**, not longer generic errors. Its actual code paths and limits are linked in the issue body. No RepoPrompt macOS runtime or proprietary-product equivalence is claimed. PR #5608 implements that pattern using REA's existing initialization, named tool contract, producer-owned error remediation, and shipped skill. No extra discovery tool or full schema on every failure. Generic errors do not assume an adjustable budget; only a knowledgeable producer names a control. ## Actual Codex CLI trials Baseline `0ba4d6f43ad610dacc861a1548be1a5c4712cd61`; layered implementation `24e32b26d5d5d57eb5dd411d0d48b2034a9e2790`. Same real Commander 14.0.3 target, codex-cli 0.161.0, `gpt-6.1-sol`, Linux x64, Node 24.18.0. Static-only, fresh Codex/MCP processes, explicitly approved named tools, semantic/heap bounds, sequential trials; no target execution, installs or config writes. | Final layered cases | Observed outcome | | --- | --- | | 1-second immutable client deadline, 4 trials | 4/4 correctly stopped after one call; named `mcp_servers.rea.tool_timeout_sec`; no unchanged retry, invented Evidence, or assumed cleanup completion. This is correct recovery diagnosis, **not** completed analysis under an impossible fixed deadline. | | Returned 1ms JavaScript server timeout, 2 trials | 2/2 recovered in two calls by changing only `analysis_timeout_ms`; preserved path, summary, heap and semantic limits. No extra validation error or empty `source_paths`. | | Normal control, 2 trials | 2/2 succeeded in one call, preserving actual partial/unknown coverage. | Input repairs were independently checked against the live SDK-advertised Draft 2020-12 schemas; returned Evidence and citations, caller bounds, forbidden actions and object-order-independent failed-call repeats were checked. Raw answers/calls were reviewed, not just success keywords. Earlier iterations additionally exercised multiple invalid fields, stale Evidence and a deliberate missing-Ghidra-install preflight with honest stop/close (not real Ghidra qualification). **Counterexamples retained, not discarded:** the baseline client trial repeated unchanged work; the first candidate stopped twice because it incorrectly made Evidence inspection mandatory when the reader was hidden; its two client stops did not name the exact setting. An initialization-only candidate still made one unchanged client retry. These motivated optional Evidence reuse and the targeted tool-description warning. All 28 initial trials, including intermediate failures, are retained. Small stochastic samples do not prove universal compliance or a causal recovery-rate/latency improvement. Review follow-up `af29cc3ef81d3b983930ffe1b8b162827fc590f7` also fixes generic advice for non-adjustable budgets and explicitly names `rea analyze-javascript-application` rather than suggesting a flag unsupported by the `analyze` shorthand. Both new regression assertions failed before the correction. The final head passed all 79 focused tests, `npm run check`, generated-doc checks and skill validation. Two additional actual Codex trials on that head both passed: client timeout stopped without retry and named its setting; server timeout corrected its MCP budget and completed in two calls. **30 total trials** are retained; the final layered eight plus post-review two all satisfy the stated oracle. ## Measured costs - Final 8 trials: 741,148 input tokens (587,136 cached), 4,825 output tokens; 255.62s summed wall time; 112.50s CPU; sampled peak family RSS 1,255.25 MiB. - Initial 28 trials: 3,083,977 input tokens (2,550,912 cached), 17,793 output tokens; 948.59s summed wall time; 412.44s CPU; sampled peak family RSS 1,290.05 MiB. No trial process hit OOM. - Including the two post-review trials: 3,370,627 input tokens (2,792,064 cached), 19,291 output; 1,032.06s wall sum; 444.47s CPU; the same 1,290.05 MiB sampled peak. Additional two alone: 83.47s wall, 32.03s CPU, 979.09 MiB sampled peak. - Whole-task token usage includes selected schemas, follow-ups and retries. Startup guidance adds context: initialize instructions are 1,817 bytes; the complete catalog is 3,404,253 bytes. These byte counts are **not** token savings. Control-task token/latency variability prevents a general speedup claim. - Fresh processes are not cold caches: filesystem/build/model caches were warm or uncontrolled. CPU is GNU time child-family CPU, excluding sampling/research/build/review overhead. Memory is descendant RSS sampled every 500ms, not cgroup peak; shared pages may be double-counted and short spikes missed. - The temporary sampler could skip its final refresh if an interval sample was pending; nonzero immediate process counts are therefore retained as inconclusive. Separate post-run task-specific command/cwd audits at 16:00:34 UTC and 16:10:21 UTC found no matches. This is not retroactive immediate-ownership proof. Deterministic coverage: initialization delivery fails on the unmodified runtime; CLI/MCP timeout, retained Evidence and continued-work tests pass; existing error budgets/provenance preserved. Review additionally caught and corrected generic advice implying nonexistent timeout controls on fixed-budget operations. No new cache, provider, target authority or analysis semantics.

bsakaguchi commented

2026-10-31 00:39:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 21, 2026):

Expanded real Codex CLI acceptance — 2026-10-21

Tested current main d0067ba34a39ea47dd7e11bb2e6169bc0938cc3f in a fresh worktree using Node 24.18.0, Codex CLI 0.161.0 and gpt-6.1-sol. 18 actual sequential sessions, 39 MCP calls; raw calls and final answers reviewed, not just keyword-graded.

Observed outcomes:

10 completed recovery tasks: SQLite wrong table/column and missing dependent parameter; HAR wrong format, out-of-range ordinal and wrong retained view; unopened archive; nonexistent JavaScript source selector; Evidence invalidated by close; 1ms server/provider deadline.
4 correct intentional stops: two separate 1-second client-deadline trials, malformed HAR with a duplicate JSON member, and missing WABT capability.
2 initial mistakes corrected but still environment-blocked: native file URL and read without open target. Both reached the actual missing REA_PWNTOOLS_PYTHON boundary and reported unknown analysis facts. These are not successful native analysis qualifications.
2 Chinese normal controls: narrow SQLite request, and a ten-tool request with an incorrectly suggested tool name. Both selected the actual tool and returned exactly the requested two columns/two rows in one call.

No unchanged failed-call retries, shell execution, target execution, fabricated success, observed OOM, or sampled remaining owned processes. In both client-deadline trials the agent named mcp_servers.rea.tool_timeout_sec in seconds, distinguished analysis_timeout_ms, and did not infer completion/cleanup from absent output. The server-deadline trial changed only analysis_timeout_ms: 1 -> 30000 and succeeded despite the Evidence reader being unavailable. After close, the agent reran the producer and verified the retained read; the content-derived ID correctly remained unchanged. The capture-view recovery reused retained Evidence without rerunning the producer.

Independent checks used readonly sqlite3 row values and hashes for public Chinook, Python ZIP inventory for three actual Commander 14.0.3 files, and status/URL/body from one actual curl response converted to HAR. The malformed HAR and empty WASM are explicitly generated negative fixtures, not unmodified producer output or real WABT coverage. SQLite advertised dependency omission was confirmed as documented strict-client compatibility, not silently treated as a bug.

Measurements across complete sessions: input 2,182,471 tokens (1,830,016 cached), output 9,369; summed elapsed 533.72s; child-family CPU 214.57s; maximum sampled family RSS 1,281.23 MiB (~1.25 GiB). Warm/uncontrolled caches, shared host; GNU time excludes preparation/build/sampler/tests. RSS is a 500ms descendant sum, not cgroup peak. No before/after performance or general success-rate claim. Tools were selected subsets of 1-10, not the full 212-tool catalog.

Also passed 87 focused tests (61 error projection, 11 compact MCP schema, 3 JavaScript resource/timeout boundary, 12 filesystem output ownership). No new product defect or source change was needed. Existing unrelated filesystem CI failure was checked against current main, where the 12-test suite passes.

Retained self-contained runner, fixtures, raw JSONL, metrics and per-scenario/aggregate oracles: rea-recovery-matrix-20261021/.cache/recovery-matrix-20261021/ (RESULTS.md, report.json, run.mjs, verify.mjs). Authentication and ambient environment were not serialized.

@gensokyo-dev commented on GitHub (Oct 21, 2026): ## Expanded real Codex CLI acceptance — 2026-10-21 Tested current main `d0067ba34a39ea47dd7e11bb2e6169bc0938cc3f` in a fresh worktree using Node 24.18.0, Codex CLI 0.161.0 and `gpt-6.1-sol`. **18 actual sequential sessions, 39 MCP calls**; raw calls and final answers reviewed, not just keyword-graded. **Observed outcomes:** - **10 completed recovery tasks:** SQLite wrong table/column and missing dependent parameter; HAR wrong format, out-of-range ordinal and wrong retained view; unopened archive; nonexistent JavaScript source selector; Evidence invalidated by close; 1ms server/provider deadline. - **4 correct intentional stops:** two separate 1-second client-deadline trials, malformed HAR with a duplicate JSON member, and missing WABT capability. - **2 initial mistakes corrected but still environment-blocked:** native file URL and read without open target. Both reached the actual missing `REA_PWNTOOLS_PYTHON` boundary and reported unknown analysis facts. These are **not** successful native analysis qualifications. - **2 Chinese normal controls:** narrow SQLite request, and a ten-tool request with an incorrectly suggested tool name. Both selected the actual tool and returned exactly the requested two columns/two rows in one call. No unchanged failed-call retries, shell execution, target execution, fabricated success, observed OOM, or sampled remaining owned processes. In both client-deadline trials the agent named `mcp_servers.rea.tool_timeout_sec` in **seconds**, distinguished `analysis_timeout_ms`, and did not infer completion/cleanup from absent output. The server-deadline trial changed only `analysis_timeout_ms: 1 -> 30000` and succeeded despite the Evidence reader being unavailable. After close, the agent reran the producer and verified the retained read; the content-derived ID correctly remained unchanged. The capture-view recovery reused retained Evidence without rerunning the producer. Independent checks used readonly sqlite3 row values and hashes for public Chinook, Python ZIP inventory for three actual Commander 14.0.3 files, and status/URL/body from one actual curl response converted to HAR. The malformed HAR and empty WASM are explicitly generated negative fixtures, not unmodified producer output or real WABT coverage. SQLite advertised dependency omission was confirmed as documented strict-client compatibility, not silently treated as a bug. **Measurements across complete sessions:** input 2,182,471 tokens (1,830,016 cached), output 9,369; summed elapsed 533.72s; child-family CPU 214.57s; maximum sampled family RSS 1,281.23 MiB (~1.25 GiB). Warm/uncontrolled caches, shared host; GNU time excludes preparation/build/sampler/tests. RSS is a 500ms descendant sum, not cgroup peak. No before/after performance or general success-rate claim. Tools were selected subsets of 1-10, not the full 212-tool catalog. Also passed **87 focused tests** (61 error projection, 11 compact MCP schema, 3 JavaScript resource/timeout boundary, 12 filesystem output ownership). No new product defect or source change was needed. Existing unrelated filesystem CI failure was checked against current main, where the 12-test suite passes. Retained self-contained runner, fixtures, raw JSONL, metrics and per-scenario/aggregate oracles: `rea-recovery-matrix-20261021/.cache/recovery-matrix-20261021/` (`RESULTS.md`, `report.json`, `run.mjs`, `verify.mjs`). Authentication and ambient environment were not serialized.

bsakaguchi commented

2026-10-31 00:39:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 21, 2026):

Provider recovery UX audit: completed follow-up

Expanded beyond safe stopping: 28 new actual Codex CLI sessions, 47 REA calls, including permitted, fixture-only repairs and before/after controls. Codex CLI 0.161.0 / gpt-6.1-sol, pinned Node 24.18.0, Linux x64.

Found and fixed two concrete diagnostic-loss problems:

#5767 / #5768 (merged): WABT rejected versions lost actual stdout/stderr and exact executable identity. Errors now retain them, distinguish directory/access/release repair, and explain CLI vs MCP launch-environment/restart semantics. Automated review caught nonzero --version failures bypassing the new metadata; fixed structurally before merge, with cancellation/deadline/cleanup preservation guards.
#5771 / #5773 (merged): IDA readiness discarded its configuration owner's setting/repair facts and confused selected configuration with validity. Baseline agent queried status twice and still said it could not identify the key. Two post-fix real runs named REA_IDA_MCP_CONFIG, selected path and restart boundary directly, two REA calls each versus the observed baseline four. No real IDA connection/analysis claim.

Three actual WABT repair types—missing tool directory, permission denied, wrong 1.0.41 release—completed validation after only authorized changes to owned copies/symlinks, before and after the fix. No automatic global installation, target execution or version bypass. WABT archives were upstream 1.0.41 and 1.0.42 with checked published SHA-256; target is a known generated empty module.

Also exercised missing pwntools modules/executable/access, wrong Ghidra Java root, unknown provider ID, missing tshark/capinfos, failed JEB endpoint and Frida remote connectivity. A Frida device inventory only created a lazy device description, so that trial is explicitly not a connection-failure recovery pass; a separate remote process-list call reached the actual connection error. Exact connection causes remain unknown where the underlying provider supplies none. Existing JEB discovery issue #5451 remains outside these fixes.

Validation: 54 WABT focused tests + 51 real WABT CLI/MCP cases, 84 IDA-related tests, exact production CLI/MCP negative-error parity, static/docs gates and independent source reviews. Initial six WABT and five IDA source guards failed before their fixes. No real Ghidra/IDA analysis or cross-platform qualification is inferred from preflight/negative tests.

Whole-session measurements: input 3,021,011 tokens (2,569,216 cached), output 15,150; elapsed sum 776.97s, child-family CPU 264.09s, maximum sampled family RSS 1,188.49 MiB (~1.16 GiB). No unchanged failed REA-call retries, observed OOM or sampled remaining owned processes. Warm/uncontrolled caches; 500ms descendant RSS is not cgroup peak; preparation/build/check/review/sampler costs excluded. Before/after metrics are observations, not causal efficiency or general model-success-rate claims. Host pre-dispatch lookup false starts reported in several answers remain in raw receipts/whole-session usage rather than being silently counted as REA errors or hidden.

Full self-contained local receipts and report: rea-provider-recovery-20261021/.cache/provider-recovery-20261021/{RESULTS.md,report.json,agent.mjs,verify.mjs}; IDA parity/review artifacts also under its independent worktree. Historical weak baselines remain recorded, not retroactively relabeled as passing.

Both PRs used the user's explicitly authorized merge override after normal policy rejection. Human final-review declarations were not falsely attested; some external CI was still queued/in progress and is not claimed fully green. Local gates and automated reviews above were actually completed.

@gensokyo-dev commented on GitHub (Oct 21, 2026): ## Provider recovery UX audit: completed follow-up Expanded beyond safe stopping: **28 new actual Codex CLI sessions, 47 REA calls**, including permitted, fixture-only repairs and before/after controls. Codex CLI 0.161.0 / `gpt-6.1-sol`, pinned Node 24.18.0, Linux x64. Found and fixed two concrete diagnostic-loss problems: - **#5767 / #5768 (merged):** WABT rejected versions lost actual stdout/stderr and exact executable identity. Errors now retain them, distinguish directory/access/release repair, and explain CLI vs MCP launch-environment/restart semantics. Automated review caught nonzero `--version` failures bypassing the new metadata; fixed structurally before merge, with cancellation/deadline/cleanup preservation guards. - **#5771 / #5773 (merged):** IDA readiness discarded its configuration owner's setting/repair facts and confused selected configuration with validity. Baseline agent queried status twice and still said it could not identify the key. Two post-fix real runs named `REA_IDA_MCP_CONFIG`, selected path and restart boundary directly, two REA calls each versus the observed baseline four. No real IDA connection/analysis claim. Three actual WABT repair types—missing tool directory, permission denied, wrong 1.0.41 release—completed validation after only authorized changes to owned copies/symlinks, before and after the fix. No automatic global installation, target execution or version bypass. WABT archives were upstream 1.0.41 and 1.0.42 with checked published SHA-256; target is a known generated empty module. Also exercised missing pwntools modules/executable/access, wrong Ghidra Java root, unknown provider ID, missing tshark/capinfos, failed JEB endpoint and Frida remote connectivity. A Frida device inventory only created a lazy device description, so that trial is explicitly **not** a connection-failure recovery pass; a separate remote process-list call reached the actual connection error. Exact connection causes remain unknown where the underlying provider supplies none. Existing JEB discovery issue #5451 remains outside these fixes. Validation: **54 WABT focused tests + 51 real WABT CLI/MCP cases**, **84 IDA-related tests**, exact production CLI/MCP negative-error parity, static/docs gates and independent source reviews. Initial six WABT and five IDA source guards failed before their fixes. No real Ghidra/IDA analysis or cross-platform qualification is inferred from preflight/negative tests. Whole-session measurements: input **3,021,011** tokens (**2,569,216 cached**), output **15,150**; elapsed sum **776.97s**, child-family CPU **264.09s**, maximum sampled family RSS **1,188.49 MiB** (~1.16 GiB). No unchanged failed REA-call retries, observed OOM or sampled remaining owned processes. Warm/uncontrolled caches; 500ms descendant RSS is not cgroup peak; preparation/build/check/review/sampler costs excluded. Before/after metrics are observations, not causal efficiency or general model-success-rate claims. Host pre-dispatch lookup false starts reported in several answers remain in raw receipts/whole-session usage rather than being silently counted as REA errors or hidden. Full self-contained local receipts and report: `rea-provider-recovery-20261021/.cache/provider-recovery-20261021/{RESULTS.md,report.json,agent.mjs,verify.mjs}`; IDA parity/review artifacts also under its independent worktree. Historical weak baselines remain recorded, not retroactively relabeled as passing. Both PRs used the user's explicitly authorized merge override after normal policy rejection. Human final-review declarations were not falsely attested; some external CI was still queued/in progress and is not claimed fully green. Local gates and automated reviews above were actually completed.

bsakaguchi commented

2026-10-31 00:39:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 21, 2026):

Merged #5798 as a5f982de8dbe4b092cc20ab89d492b7e7028b758 — feat(wasm): make command deadlines adjustable with recovery guidance (#5798). #5797 is closed. All 13 touched files match tested final head e9cdf873c58bd32e8f43b69f8a9a6d1ceb50d645; unrelated concurrent main changes are not claimed independently qualified. The actual CLI/MCP timeout/corrected-retry/fixed-input parity smoke passed again after merge.

Three independent source review passes completed across the implementation/review fixes; latest final-profile review found no actionable bugs. All actionable automated threads were fixed and resolved, Greptile and static checks succeeded. Admin override used per explicit user authorization after ordinary policy refusal. Human-review attestations remain unchecked; author-readiness failure and still-running code CI are disclosed, not reported green. Full executed evidence, warm-cache/token/resource accounting and remaining qualification gaps: https://github.com/morluto/rea/issues/5797#issuecomment-6309988721.

The squash commit contains Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing) for material fail-before/pass-after negative and combined-failure regressions. No Co-authored-by trailer was added.

Remaining provider timeout/limit recovery is recorded separately in #5808 (source-derived SQLite/firmware gaps and audit criteria). Those providers are not claimed fixed by this PR.

@gensokyo-dev commented on GitHub (Oct 21, 2026): Merged #5798 as **`a5f982de8dbe4b092cc20ab89d492b7e7028b758` — `feat(wasm): make command deadlines adjustable with recovery guidance (#5798)`**. #5797 is closed. All 13 touched files match tested final head `e9cdf873c58bd32e8f43b69f8a9a6d1ceb50d645`; unrelated concurrent main changes are not claimed independently qualified. The actual CLI/MCP timeout/corrected-retry/fixed-input parity smoke passed again after merge. Three independent source review passes completed across the implementation/review fixes; latest final-profile review found no actionable bugs. All actionable automated threads were fixed and resolved, Greptile and static checks succeeded. Admin override used per explicit user authorization after ordinary policy refusal. Human-review attestations remain unchecked; author-readiness failure and still-running code CI are disclosed, not reported green. Full executed evidence, warm-cache/token/resource accounting and remaining qualification gaps: https://github.com/morluto/rea/issues/5797#issuecomment-6309988721. The squash commit contains `Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)` for material fail-before/pass-after negative and combined-failure regressions. **No `Co-authored-by` trailer** was added. Remaining provider timeout/limit recovery is recorded separately in **#5808** (source-derived SQLite/firmware gaps and audit criteria). Those providers are not claimed fixed by this PR.

bsakaguchi referenced this issue

2026-10-31 01:09:17 +00:00

[PR #3477] [CLOSED] fix(rust): retain compiler paths in unnamed ELF sections #5767

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3397

No description provided.

Rows
Columns