morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1587] [Feature] Improve task-centered REA onboarding and recovery in Codex skills #535

New issue

Open

opened 2026-10-15 18:25:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:25:58 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 10, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1587

Before submitting

I searched open and closed issues and pull requests for related work.
I removed credentials, private paths, and sensitive target data.

Area

MCP or agent integration; setup and documentation.

Current limitation

A fresh agent should move from a user-supplied artifact to useful evidence with minimal setup friction. An isolated Linux audit of the published rea-agents@6.3.0 skill and repository-main skill found that core routing and Evidence handling work, but several recoverable conditions still cause unnecessary retries or unclear next steps:

Codex can reject an advertised MCP operation with MCP tool call requires approval, but approval policy is never. This is a client approval decision, not a missing REA engine or registration problem. Switching to workspace-write alone did not resolve it. Archive and HAR operations worked through the actual MCP boundary and with narrowly scoped, explicitly authorized temporary Codex tool approvals.
A JavaScript module drill-down invented view.kind = "module"; the correct contract is kind = "item", collection = "modules". A subsequent path selector matched multiple module roles. The runtime returned useful node-ID candidates, but the skill lacks a concrete recipe for selecting them.
Summary analysis produced substantially less inline data, but a broad trace_application_feature request with direction = "both" expanded through containment relationships into unrelated modules. Codex truncated the displayed result inside its textual envelope. This client truncation is distinct from REA's structured transport-limit error.
The JavaScript reference still foregrounds complete inline results rather than the summary-first workflow supported by current main.
A skill-only CLI trial encountered a login-shell PATH reset in the isolated installation. Repeating with the correct public CLI and existing pinned Node on the disposable shell PATH succeeded. Guidance should diagnose the actual command environment without unnecessary global installation or MCP repair.

Use cases

A user asks, in English or Chinese, how a packaged Electron application exposes one renderer API.
A user asks for a ZIP member inventory or an offline HAR request without needing a native engine.
A native analysis operation reports an unavailable engine; the agent continues any useful supported static work and identifies the remaining limitation.
A skill is installed without MCP registration, or the active session has not reconnected after setup.
A large application requires a focused module view rather than a broad graph dump.
Ordinary source analysis and missing-target requests should avoid unnecessary REA onboarding.

Desired outcome and success criteria

Start directly when the required operation is available; diagnose only an observed blocker.
Distinguish client policy, registration/connection, command PATH, and task-specific provider availability. Give the smallest useful next step, then resume the original task when possible.
Preserve existing setup planning, backups, idempotence and authorization boundaries; do not install unrelated prerequisites or silently weaken client approvals.
Use the connected server's actual schemas, valid retained parent Evidence IDs, bounded module pages and unambiguous node-ID selectors. Avoid repeated identical failing calls.
Recognize client-truncated output as incomplete display evidence; recover through retained views or an explicitly selected export rather than parsing a truncated preview as canonical JSON.
Validate the updated installed skill through fresh, isolated Codex CLI sessions, with real tool results and explicit failure classification. Exit code zero alone is not a successful workflow oracle.
Keep model tests sequential and resource bounded.

Authority boundaries and non-goals

This first implementation targets the authored REA skill, focused references and the existing Codex evaluation lane. It does not introduce an installer for Node, Java or native engines, broaden target execution, change IDA/Ghidra mutation authority, or mark resource-producing operations falsely read-only to avoid client approval. Local Evidence preserves caller-selected ordinary values and redacts transport credentials.

Graph traversal/containment semantics and resource handling belong to the producing tool. Skill guidance can choose a narrower useful query, but should not claim to repair a broad trace's graph semantics or transport behavior. Native engine integration and other clients require their own verification.

Alternatives considered

Re-running setup or broad diagnostics on every operation adds friction and does not solve an active client's approval policy.
Globally allowing all tools or changing MCP annotations hides the actual authority boundary.
A second evaluation framework would duplicate the existing verify:agent lane; extend representative scenarios and use its evaluator where applicable.

Possible direction

Keep task routing concise in SKILL.md, place conditional connection/client recovery guidance in a focused reference, synchronize JavaScript summary and view examples, and strengthen fresh-profile Codex testing. Borrow question-driven stage exits and concise failure records from reverse-skill without importing its bulk prerequisites or mandatory runtime workflow.

Verification scope and follow-up TODOs

The initial audit ran 19 sequential real Codex CLI trials using Codex 0.162.1 and gpt-6.1-sol, against release 6.3.0 and main at c19e15a2. It also checked 139 actual post-SDK MCP input schemas. Trials included successful workflows, four client approval blocks, a genuine provider-unavailable response, and two invalid/ambiguous view requests; these are diagnostic observations, not 19 uniformly passing workflows. Model runs used one CPU core and a 3 GiB memory ceiling; peak sampled aggregate trial memory was about 724 MiB. These are small representative fixtures, not a general performance benchmark or isolated causal A/B test.

Implementation and retest (first phase: Linux / Codex only):

Skill version 37 keeps the initial route concise and moves conditional connection/recovery guidance into an installed reference. It distinguishes registration, active-session connection, client approval, shell PATH and provider availability, then continues the original task through an authorized workflow.
JavaScript guidance now gives concrete summary → retained parent → module page → exact node-ID item recipes, limits claims to the selected view, and distinguishes client preview truncation from a REA transport error. A focused outgoing trace avoids unrelated containment expansion in the tested fixture; broad graph semantics remain a separate follow-up.
Actual public setup exposed a packaging integration omission: the new reference was generated but absent from the setup installer's managed file list. The list and existing byte-equality/idempotence boundary checks now include it.
A real polluted-workspace test initially selected a desktop fixture from an unrelated saved report when the user said only “my desktop app.” The skill now asks for an app name/path before artifact searching unless a target was already selected for this task. Both a fresh-profile negative scenario and the original polluted-workspace request passed on retest without any REA calls.
verify:agent now installs through public rea setup into independent HOME/CODEX_HOME/XDG directories, copies authentication only, disables shell snapshots and additional agents, and removes authentication even when fixtures are retained. Codex helper aliases were refused when the profile lived under the OS temporary directory; the default disposable account now lives in a private cache and retains the sandbox.
Evaluator fixes count complete text-only REA input errors and SDK-rejected invalid schema arguments, preserve provider/client distinctions, and allow useful session-status verification after successful open/close. Report schema 4 supports expectedFirstTool: null and a separately validated targetClarificationPassed result; report consumers must allow this negative routing outcome. Normal analysis gates still require their tools.

The final representative set contains 13 limited workflow outcomes using Codex CLI 0.162.1, model gpt-6.1-sol, isolated REA package 6.3.0 and installed skill 37:

Coverage	Outcome / oracle
Native missing-provider workflow	Public MCP session operations reached real unavailable-provider diagnostics; static fallback and session closure were reviewed. One independent inspection call remained client-policy blocked. No real engine pass is claimed.
Chinese ASAR factual scenario	Passed the closed rubric for four fixture claims with authenticated complete producer Evidence.
Large module-view workflow	Summary, module pages and node-ID item calls passed routing/heuristic gates with zero invalid or repeated calls; transcript review confirmed limited module-view claims.
Fresh missing-target workflow	Strict structured target clarification and zero REA calls passed.
Nine manually reviewed outcomes	Chinese ASAR, larger focused trace, ZIP and offline HAR CLI recovery after client blocks, skill-only CLI use, ordinary-source negative routing, older-runtime schema compatibility, normal Codex client mode, and the polluted-workspace missing-target retest.

Earlier failing verifier attempts and the original wrong-target result are retained as diagnostic evidence. Positive transcript heuristics and manual workflow review do not establish unrestricted factual correctness; only the configured ASAR scenario has a closed factual oracle. The closed rubric also required full Evidence and an extra Evidence pull, so these tests do not demonstrate universal token savings.

All model runs were sequential under a one-core CPU allocation and 3 GiB process-tree memory ceiling. Peak sampled aggregate memory in the final model runs was about 930 MiB, with no model-run OOM or timeout. An initial full static typecheck was stopped at the owned scope's memory ceiling; retrying with compiler concurrency and memory controls succeeded without increasing the ceiling. Validation passed:

64 focused tests across four existing filesystem/evaluation test files; the regrouped evaluator file subsequently passed its 11 tests again.
npm run check (typecheck, lint, formatting, unused-code, metadata, module boundaries and architecture guards).
npm run docs:check and the skill frontmatter validator.
npm run verify:package: repository-root Skills 1.7.2 discovery installed all seven canonical skill files, including the recovery reference; packaged skill/setup, backup/readback, read-only plan, idempotence, CLI and public MCP checks passed. The packaged MCP server listed 170 tools. Config/provider simulations in this lane do not establish another real agent client's behavior or real engine correctness.
npm pack --dry-run: the package includes the new reference and generated skill version 37.

Remaining friction is explicit: some module lookups still need several identity pages; a client's independent approval policy can still block an operation; full Evidence required by a factual rubric can be expensive; broad trace expansion and transport behavior require producing-tool work if that scope is pursued.

Implement the focused skill/onboarding improvements.
Rebuild and install the updated skill into disposable Codex profiles; rerun real Codex CLI workflows and relevant deterministic checks.
Verify Pi, Hermes and DeepSeek Harness native skill/onboarding workflows separately; see the second-phase scope and limitations below.
Verify remaining agent clients separately (Claude Code, Gemini, Copilot, Qwen, OpenCode, OMP, Grok, and other supported integrations). Not tested in this investigation.
Verify real analysis providers separately (Hopper, Ghidra, IDA, managed/Android/firmware/runtime providers as applicable). No real-provider engine validation in this investigation.
Verify further model providers/models and macOS/Windows first-use onboarding. Limited gpt-6-sol client trials are now tested below; the live DeepSeek API and other model providers remain unverified.
Follow up on producing-tool trace expansion/client output limits with their own contract and coverage if a tool change is needed.

PR and final-head verification

Ready-for-review implementation: #1613, based on main at c1217e71; final PR head 47e72a0.

The final head passed 65 focused tests (including preservation of an unrecognized structured failure), npm run check, npm run docs:check, npm run verify:package and npm pack --dry-run. Four fresh-profile real Codex scenarios were repeated on that rebased head: native provider unavailability, Chinese ASAR, module views, and missing-target clarification. All passed their limited gates with zero repeated calls or input-validation failures. The ASAR four-claim factual oracle passed, and missing-target made zero REA calls. Sampled aggregate model-run memory peaked at about 860 MiB; copied authentication was removed.

Other clients, live engine providers and platform/model onboarding remain the follow-up scope listed above. CI is being monitored before merge.

References

Authored skill: https://github.com/morluto/rea/tree/main/.agents/skills
Earlier bootstrap work: #737; version-pinned packaging: #1338
Summary-view support: #1568
Separate JavaScript resource-limit work: #1563 and #1565
Codex configuration reference: https://learn.chatgpt.com/docs/config-file/config-reference
Prior-art review: https://github.com/zhaoxuya520/reverse-skill (read-only review; its scripts and providers were not executed)

Additional CI and model revalidation

The first CI run for #1613 found six inherited cleanup/uninstall boundary regressions, also present on an unrelated PR. They were reproduced twice locally, including on updated main. The separate prerequisite #1618 fixes late CDP cleanup cancellation and preserves the current error/resource/file-ownership rules in the regressions; 149 integrated tests and 64 repeated boundary tests passed, plus the full local check. Its real Chrome 154 CLI/stdio MCP and real Web runtime CI lanes passed.

At integration revision 82e01131, a four-scenario fresh Codex rerun retained correct routing, zero repeated calls, correct module views and a validated zero-REA-call missing-target clarification. The ASAR run answered all four factual claims correctly but omitted source.kind on one view call, recovered after the SDK rejection, and consequently failed the strict zero-invalid-input gate. Its record is retained. Two further independent ASAR profiles both passed the four-claim rubric with zero invalid/repeated calls. This has not established a repeatable REA design defect; no speculative extra fix or relaxed evaluation gate was added. Model behavior is not guaranteed by a single passing run. Authentication copies were removed after every run.

Final branch head bec59688: all four new-profile Codex scenarios passed their strict gates (zero invalid/repeated calls; ASAR four-claim factual rubric passed; missing-target clarified with zero REA calls). The run took about 158 seconds with sampled peak about 835 MiB. The earlier recovered invalid-call run remains above. #1618 passed all 21 checks and merged as 16d9c5ad; #1613 has refreshed CI on this final branch head.

PR #1613 merged as 2653b194 after all 19 checks passed on bec59688; #1618 merged after all 21 checks passed. Follow-up client and first-time npm-runner audits now start in a separate worktree. The npm release remains 6.3.0; merging does not publish the new skill.

Second phase: Pi, Hermes, DeepSeek Harness and cold npx onboarding

This phase used separate worktrees and private client installations on Linux x64 with Node 24.18.0/npm 11.16.0. Pi 1.1.0 and DeepSeek Harness 0.2.0-rc.2 were installed locally; Hermes source 66605471 used its official minimal Python/MCP installation in a private runtime (Python 3.14.7). No global client/toolchain upgrade or native engine installation was performed.

Release/source identity matters: the actual npm 6.3.0 artifact has skill 34, 139 tools and gitHead 74cf9e14. Source builds used skill 37 and 170 tools while package.json still says 6.3.0. The registry still reports 6.3.0 at 2026-10-10 20:33 UTC. Automatic Pi and Hermes setup exists in these source builds but is absent from the released package; DeepSeek is a manual native-MCP preview, not an advertised automatic setup client.

Confirmed REA findings and changes:

#1625: native verifiers constructed replacement environments with USERPROFILE but no HOME. Two reduced Node probes fell back to the real POSIX account home. Pi also had an explicit skill setting, and DeepSeek an agents-home override, so earlier passes did not establish default shared personal skill discovery. Bind HOME and USERPROFILE and test the actual defaults.
#1626: two released Pi and two released Hermes setup attempts rejected the client before writes. Installation guidance mixed current-main capability with published-package behavior. Document the checked release and require a build containing these integrations.
#1630: the source/tests trees in #1627 were identical to its base, but main failed the unchanged duplicate-code gate at 2.52%. Refreshed main also reproduced it. The separate #1631 shares identical semantic graph fixture builders, keeps all assertions and identities, and passes the same 38 tests before/after. Duplication fell to 2.47% with the existing threshold. #1631 merged as be44b509; all 20 applicable checks subsequently passed.

#1627 contains only the confirmed HOME/discovery verifier and release-guidance changes. Final head a7ba07ba is based on bbd01923, including #1631. Build:cached, check:ci, docs:check and all four native client call lanes passed on that integrated head. All 19 CI checks passed on a7ba07ba; #1627 merged at 2026-10-10 20:37:32 UTC as b74d9a1b. #1625 and #1626 closed; the compatibility follow-ups remain open.

Native deterministic-loopback verification: 23 passing call-lane records in total (7 baseline records with the isolation/discovery limitations above; 16 after the HOME fix). Every corrected native lane discovered all 170 actual SDK tools, checked the advertised schemas, loaded the installed skill through the tested discovery path and returned complete named-schema Unicode-path JavaScript Evidence. The 16 corrected runs include twice each Pi/Hermes/Hermes sticky/DeepSeek, then each repeated at 1bba5545 and a7ba07ba. These establish native integration with fixture model responses, not live-provider reasoning or engine coverage.

Real Codex-backed model trials used gpt-6-sol, fresh homes/Git roots and a two-module inert JavaScript bundle. Positive answers were manually checked against four known fixture facts, not scored by a general factual-correctness benchmark.

Native workflow	Independent repeats	Outcome
Pi bundle analysis	2	Correct routes/backend/retries/static-only answers. Seven REA calls in one run; seventeen in the other, including one invalid seed followed by recovery.
Hermes bundle analysis	2 native, plus 1 earlier imported-entrypoint trial	Correct fixture answers and static-vs-runtime limitations; native runs used nine and ten REA calls and exited successfully. The final native repeat used integrated `a7ba07ba`.
DeepSeek bundle analysis	2	Correct final answers with eight and ten REA calls, but both processes exceeded the 120-second ceiling after completion.
Missing target, each of Pi/Hermes/DeepSeek	2 per client	All six asked for the app name/path and made zero REA operations, despite a nearby bundle fixture. Pi/Hermes exited; DeepSeek did not.
DeepSeek with no REA/skill/MCP	2	Final READY, zero tool calls, still alive at the 45-second ceiling.

The initial imported Pi trial recovered from two invalid selectors. Separate authentication/configuration experiments failed before any REA work (one Pi exit-zero transcript contained a model error); they are not positive outcomes. A wrong external sticky-test argument was corrected to the existing environment selector, without a REA code change.

Client-owned or uncertain observations are recorded without speculative production fixes:

#1632: Pi code-mode declarations project a valid union selector to unknown; two transcripts exposed the lost type shape. This does not prove that type loss alone caused model guesses or that REA's schema is invalid. Investigate the client projection first; retain strict REA selectors.
#1633: Hermes emits an ancient-config warning for a fresh unversioned config. Four empty-map/empty-file controls reproduce it without REA; two missing-file controls do not. Boot maintenance projects an absent stamp to zero while the migration owner distinguishes absent from explicitly old. Do not hardcode a client schema version in REA.
#1636: DeepSeek native headless completion does not end the process on the tested Codex adapter; no-REA controls reproduce it. Investigate client/provider lifecycle ownership rather than adding a REA forced-exit workaround.

DeepSeek headless tool-result previews were truncated. The follow-up reused the installed producer's frame scanner to decode every complete frame of canonical session logs, with no torn tails. All 18 REA inputs and complete outputs passed their named contracts; final cited Evidence IDs matched returned Evidence. Complete logs contained no transport credential. A preliminary one-frame decoder was recognized as incomplete and replaced; its partial record is not a complete-session pass.

Two independent empty npm caches verified the actual first-use npx --yes rea-agents@6.3.0 Codex workflow. Cold acquisition took about 49 seconds each. Unapproved setup reported needs_confirmation/exit 1 with no client/skill writes; dry-run showed the plan; approved setup installed six canonical release skill files and pinned MCP registration; repeat had no actions or config changes; doctor was healthy. Both actual CLI analysis and the registered npx-launched SDK MCP server returned Unicode-path fixture Evidence; all 139 release schemas were checked. Two additional native PTY runs reached the approval prompt and cancelled, preserving the original config without backups or skill writes. A preliminary unsupported CLI spelling and a missing pexpect in a private Python were harness errors, corrected using the public MCP path and existing system Python.

Every heavy local command/model run was sequential under one CPU and a 3 GiB process-tree ceiling. The largest sampled second-phase model scope was about 720 MiB; complete static checking peaked at about 2.61 GiB. No second-phase host OOM occurred. Original OAuth stores remained unchanged and temporary access-only files were removed. Private audit receipts/transcripts are retained outside the original development checkout.

Remaining TODOs stay open: other named clients, real DeepSeek/other model providers, native Windows/macOS first-use onboarding, and real deep-analysis engines. Passing native fixtures or package/platform CI does not establish those workflows. Merging these changes does not publish a new npm package.

Originally created by @N0zoM1z0 on GitHub (Oct 10, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1587 ### Before submitting - [x] I searched open and closed issues and pull requests for related work. - [x] I removed credentials, private paths, and sensitive target data. ### Area MCP or agent integration; setup and documentation. ### Current limitation A fresh agent should move from a user-supplied artifact to useful evidence with minimal setup friction. An isolated Linux audit of the published `rea-agents@6.3.0` skill and repository-main skill found that core routing and Evidence handling work, but several recoverable conditions still cause unnecessary retries or unclear next steps: - Codex can reject an advertised MCP operation with `MCP tool call requires approval, but approval policy is never`. This is a client approval decision, not a missing REA engine or registration problem. Switching to `workspace-write` alone did not resolve it. Archive and HAR operations worked through the actual MCP boundary and with narrowly scoped, explicitly authorized temporary Codex tool approvals. - A JavaScript module drill-down invented `view.kind = "module"`; the correct contract is `kind = "item"`, `collection = "modules"`. A subsequent path selector matched multiple module roles. The runtime returned useful node-ID candidates, but the skill lacks a concrete recipe for selecting them. - Summary analysis produced substantially less inline data, but a broad `trace_application_feature` request with `direction = "both"` expanded through containment relationships into unrelated modules. Codex truncated the displayed result inside its textual envelope. This client truncation is distinct from REA's structured transport-limit error. - The JavaScript reference still foregrounds complete inline results rather than the summary-first workflow supported by current main. - A skill-only CLI trial encountered a login-shell PATH reset in the isolated installation. Repeating with the correct public CLI and existing pinned Node on the disposable shell PATH succeeded. Guidance should diagnose the actual command environment without unnecessary global installation or MCP repair. ### Use cases - A user asks, in English or Chinese, how a packaged Electron application exposes one renderer API. - A user asks for a ZIP member inventory or an offline HAR request without needing a native engine. - A native analysis operation reports an unavailable engine; the agent continues any useful supported static work and identifies the remaining limitation. - A skill is installed without MCP registration, or the active session has not reconnected after setup. - A large application requires a focused module view rather than a broad graph dump. - Ordinary source analysis and missing-target requests should avoid unnecessary REA onboarding. ### Desired outcome and success criteria - Start directly when the required operation is available; diagnose only an observed blocker. - Distinguish client policy, registration/connection, command PATH, and task-specific provider availability. Give the smallest useful next step, then resume the original task when possible. - Preserve existing setup planning, backups, idempotence and authorization boundaries; do not install unrelated prerequisites or silently weaken client approvals. - Use the connected server's actual schemas, valid retained parent Evidence IDs, bounded module pages and unambiguous node-ID selectors. Avoid repeated identical failing calls. - Recognize client-truncated output as incomplete display evidence; recover through retained views or an explicitly selected export rather than parsing a truncated preview as canonical JSON. - Validate the updated installed skill through fresh, isolated Codex CLI sessions, with real tool results and explicit failure classification. Exit code zero alone is not a successful workflow oracle. - Keep model tests sequential and resource bounded. ### Authority boundaries and non-goals This first implementation targets the authored REA skill, focused references and the existing Codex evaluation lane. It does not introduce an installer for Node, Java or native engines, broaden target execution, change IDA/Ghidra mutation authority, or mark resource-producing operations falsely read-only to avoid client approval. Local Evidence preserves caller-selected ordinary values and redacts transport credentials. Graph traversal/containment semantics and resource handling belong to the producing tool. Skill guidance can choose a narrower useful query, but should not claim to repair a broad trace's graph semantics or transport behavior. Native engine integration and other clients require their own verification. ### Alternatives considered - Re-running setup or broad diagnostics on every operation adds friction and does not solve an active client's approval policy. - Globally allowing all tools or changing MCP annotations hides the actual authority boundary. - A second evaluation framework would duplicate the existing `verify:agent` lane; extend representative scenarios and use its evaluator where applicable. ### Possible direction Keep task routing concise in `SKILL.md`, place conditional connection/client recovery guidance in a focused reference, synchronize JavaScript summary and view examples, and strengthen fresh-profile Codex testing. Borrow question-driven stage exits and concise failure records from reverse-skill without importing its bulk prerequisites or mandatory runtime workflow. ### Verification scope and follow-up TODOs The initial audit ran 19 sequential real Codex CLI trials using Codex 0.162.1 and `gpt-6.1-sol`, against release 6.3.0 and main at `c19e15a2`. It also checked 139 actual post-SDK MCP input schemas. Trials included successful workflows, four client approval blocks, a genuine provider-unavailable response, and two invalid/ambiguous view requests; these are diagnostic observations, not 19 uniformly passing workflows. Model runs used one CPU core and a 3 GiB memory ceiling; peak sampled aggregate trial memory was about 724 MiB. These are small representative fixtures, not a general performance benchmark or isolated causal A/B test. Implementation and retest (first phase: Linux / Codex only): - Skill version 37 keeps the initial route concise and moves conditional connection/recovery guidance into an installed reference. It distinguishes registration, active-session connection, client approval, shell PATH and provider availability, then continues the original task through an authorized workflow. - JavaScript guidance now gives concrete summary → retained parent → module page → exact node-ID item recipes, limits claims to the selected view, and distinguishes client preview truncation from a REA transport error. A focused outgoing trace avoids unrelated containment expansion in the tested fixture; broad graph semantics remain a separate follow-up. - Actual public setup exposed a packaging integration omission: the new reference was generated but absent from the setup installer's managed file list. The list and existing byte-equality/idempotence boundary checks now include it. - A real polluted-workspace test initially selected a desktop fixture from an unrelated saved report when the user said only “my desktop app.” The skill now asks for an app name/path before artifact searching unless a target was already selected for this task. Both a fresh-profile negative scenario and the original polluted-workspace request passed on retest without any REA calls. - `verify:agent` now installs through public `rea setup` into independent HOME/CODEX_HOME/XDG directories, copies authentication only, disables shell snapshots and additional agents, and removes authentication even when fixtures are retained. Codex helper aliases were refused when the profile lived under the OS temporary directory; the default disposable account now lives in a private cache and retains the sandbox. - Evaluator fixes count complete text-only REA input errors and SDK-rejected invalid schema arguments, preserve provider/client distinctions, and allow useful session-status verification after successful open/close. Report schema **4** supports `expectedFirstTool: null` and a separately validated `targetClarificationPassed` result; report consumers must allow this negative routing outcome. Normal analysis gates still require their tools. The final representative set contains **13 limited workflow outcomes** using Codex CLI **0.162.1**, model **gpt-6.1-sol**, isolated REA package **6.3.0** and installed skill **37**: | Coverage | Outcome / oracle | | --- | --- | | Native missing-provider workflow | Public MCP session operations reached real unavailable-provider diagnostics; static fallback and session closure were reviewed. One independent inspection call remained client-policy blocked. No real engine pass is claimed. | | Chinese ASAR factual scenario | Passed the closed rubric for four fixture claims with authenticated complete producer Evidence. | | Large module-view workflow | Summary, module pages and node-ID item calls passed routing/heuristic gates with zero invalid or repeated calls; transcript review confirmed limited module-view claims. | | Fresh missing-target workflow | Strict structured target clarification and zero REA calls passed. | | Nine manually reviewed outcomes | Chinese ASAR, larger focused trace, ZIP and offline HAR CLI recovery after client blocks, skill-only CLI use, ordinary-source negative routing, older-runtime schema compatibility, normal Codex client mode, and the polluted-workspace missing-target retest. | Earlier failing verifier attempts and the original wrong-target result are retained as diagnostic evidence. Positive transcript heuristics and manual workflow review do not establish unrestricted factual correctness; only the configured ASAR scenario has a closed factual oracle. The closed rubric also required full Evidence and an extra Evidence pull, so these tests do **not** demonstrate universal token savings. All model runs were sequential under a one-core CPU allocation and 3 GiB process-tree memory ceiling. Peak sampled aggregate memory in the final model runs was about **930 MiB**, with no model-run OOM or timeout. An initial full static typecheck was stopped at the owned scope's memory ceiling; retrying with compiler concurrency and memory controls succeeded without increasing the ceiling. Validation passed: - 64 focused tests across four existing filesystem/evaluation test files; the regrouped evaluator file subsequently passed its 11 tests again. - `npm run check` (typecheck, lint, formatting, unused-code, metadata, module boundaries and architecture guards). - `npm run docs:check` and the skill frontmatter validator. - `npm run verify:package`: repository-root Skills 1.7.2 discovery installed all seven canonical skill files, including the recovery reference; packaged skill/setup, backup/readback, read-only plan, idempotence, CLI and public MCP checks passed. The packaged MCP server listed 170 tools. Config/provider simulations in this lane do not establish another real agent client's behavior or real engine correctness. - `npm pack --dry-run`: the package includes the new reference and generated skill version 37. Remaining friction is explicit: some module lookups still need several identity pages; a client's independent approval policy can still block an operation; full Evidence required by a factual rubric can be expensive; broad trace expansion and transport behavior require producing-tool work if that scope is pursued. - [x] Implement the focused skill/onboarding improvements. - [x] Rebuild and install the updated skill into disposable Codex profiles; rerun real Codex CLI workflows and relevant deterministic checks. - [x] Verify Pi, Hermes and DeepSeek Harness native skill/onboarding workflows separately; see the second-phase scope and limitations below. - [ ] Verify remaining agent clients separately (Claude Code, Gemini, Copilot, Qwen, OpenCode, OMP, Grok, and other supported integrations). **Not tested in this investigation.** - [ ] Verify real analysis providers separately (Hopper, Ghidra, IDA, managed/Android/firmware/runtime providers as applicable). **No real-provider engine validation in this investigation.** - [ ] Verify further model providers/models and macOS/Windows first-use onboarding. Limited gpt-6-sol client trials are now tested below; the live DeepSeek API and other model providers remain unverified. - [ ] Follow up on producing-tool trace expansion/client output limits with their own contract and coverage if a tool change is needed. ### PR and final-head verification Ready-for-review implementation: #1613, based on main at `c1217e71`; final PR head `47e72a0`. The final head passed **65** focused tests (including preservation of an unrecognized structured failure), `npm run check`, `npm run docs:check`, `npm run verify:package` and `npm pack --dry-run`. Four fresh-profile real Codex scenarios were repeated on that rebased head: native provider unavailability, Chinese ASAR, module views, and missing-target clarification. All passed their limited gates with zero repeated calls or input-validation failures. The ASAR four-claim factual oracle passed, and missing-target made zero REA calls. Sampled aggregate model-run memory peaked at about **860 MiB**; copied authentication was removed. Other clients, live engine providers and platform/model onboarding remain the follow-up scope listed above. CI is being monitored before merge. ### References - Authored skill: https://github.com/morluto/rea/tree/main/.agents/skills - Earlier bootstrap work: #737; version-pinned packaging: #1338 - Summary-view support: #1568 - Separate JavaScript resource-limit work: #1563 and #1565 - Codex configuration reference: https://learn.chatgpt.com/docs/config-file/config-reference - Prior-art review: https://github.com/zhaoxuya520/reverse-skill (read-only review; its scripts and providers were not executed) ### Additional CI and model revalidation The first CI run for #1613 found six inherited cleanup/uninstall boundary regressions, also present on an unrelated PR. They were reproduced twice locally, including on updated main. The separate prerequisite #1618 fixes late CDP cleanup cancellation and preserves the current error/resource/file-ownership rules in the regressions; 149 integrated tests and 64 repeated boundary tests passed, plus the full local check. Its real Chrome 154 CLI/stdio MCP and real Web runtime CI lanes passed. At integration revision `82e01131`, a four-scenario fresh Codex rerun retained correct routing, zero repeated calls, correct module views and a validated zero-REA-call missing-target clarification. The ASAR run answered all four factual claims correctly but omitted `source.kind` on one view call, recovered after the SDK rejection, and consequently **failed** the strict zero-invalid-input gate. Its record is retained. Two further independent ASAR profiles both passed the four-claim rubric with zero invalid/repeated calls. This has not established a repeatable REA design defect; no speculative extra fix or relaxed evaluation gate was added. Model behavior is not guaranteed by a single passing run. Authentication copies were removed after every run. Final branch head `bec59688`: all four new-profile Codex scenarios passed their strict gates (zero invalid/repeated calls; ASAR four-claim factual rubric passed; missing-target clarified with zero REA calls). The run took about 158 seconds with sampled peak about 835 MiB. The earlier recovered invalid-call run remains above. #1618 passed all 21 checks and merged as `16d9c5ad`; #1613 has refreshed CI on this final branch head. PR #1613 merged as `2653b194` after all 19 checks passed on `bec59688`; #1618 merged after all 21 checks passed. Follow-up client and first-time npm-runner audits now start in a separate worktree. The npm release remains `6.3.0`; merging does not publish the new skill. ### Second phase: Pi, Hermes, DeepSeek Harness and cold npx onboarding This phase used separate worktrees and private client installations on Linux x64 with Node 24.18.0/npm 11.16.0. Pi 1.1.0 and DeepSeek Harness 0.2.0-rc.2 were installed locally; Hermes source 66605471 used its official minimal Python/MCP installation in a private runtime (Python 3.14.7). No global client/toolchain upgrade or native engine installation was performed. Release/source identity matters: the actual npm 6.3.0 artifact has skill **34**, **139** tools and gitHead `74cf9e14`. Source builds used skill **37** and **170** tools while package.json still says 6.3.0. The registry still reports 6.3.0 at 2026-10-10 20:33 UTC. Automatic Pi and Hermes setup exists in these source builds but is absent from the released package; DeepSeek is a manual native-MCP preview, not an advertised automatic setup client. Confirmed REA findings and changes: - #1625: native verifiers constructed replacement environments with USERPROFILE but no HOME. Two reduced Node probes fell back to the real POSIX account home. Pi also had an explicit skill setting, and DeepSeek an agents-home override, so earlier passes did not establish default shared personal skill discovery. Bind HOME and USERPROFILE and test the actual defaults. - #1626: two released Pi and two released Hermes setup attempts rejected the client before writes. Installation guidance mixed current-main capability with published-package behavior. Document the checked release and require a build containing these integrations. - #1630: the source/tests trees in #1627 were identical to its base, but main failed the unchanged duplicate-code gate at 2.52%. Refreshed main also reproduced it. The separate #1631 shares identical semantic graph fixture builders, keeps all assertions and identities, and passes the same 38 tests before/after. Duplication fell to 2.47% with the existing threshold. #1631 merged as `be44b509`; all 20 applicable checks subsequently passed. #1627 contains only the confirmed HOME/discovery verifier and release-guidance changes. Final head `a7ba07ba` is based on `bbd01923`, including #1631. Build:cached, check:ci, docs:check and all four native client call lanes passed on that integrated head. All **19** CI checks passed on `a7ba07ba`; #1627 merged at 2026-10-10 20:37:32 UTC as `b74d9a1b`. #1625 and #1626 closed; the compatibility follow-ups remain open. Native deterministic-loopback verification: 23 passing call-lane records in total (7 baseline records with the isolation/discovery limitations above; 16 after the HOME fix). Every corrected native lane discovered all 170 actual SDK tools, checked the advertised schemas, loaded the installed skill through the tested discovery path and returned complete named-schema Unicode-path JavaScript Evidence. The 16 corrected runs include twice each Pi/Hermes/Hermes sticky/DeepSeek, then each repeated at 1bba5545 and a7ba07ba. These establish native integration with fixture model responses, not live-provider reasoning or engine coverage. Real Codex-backed model trials used gpt-6-sol, fresh homes/Git roots and a two-module inert JavaScript bundle. Positive answers were manually checked against four known fixture facts, not scored by a general factual-correctness benchmark. | Native workflow | Independent repeats | Outcome | | --- | --- | --- | | Pi bundle analysis | 2 | Correct routes/backend/retries/static-only answers. Seven REA calls in one run; seventeen in the other, including one invalid seed followed by recovery. | | Hermes bundle analysis | 2 native, plus 1 earlier imported-entrypoint trial | Correct fixture answers and static-vs-runtime limitations; native runs used nine and ten REA calls and exited successfully. The final native repeat used integrated a7ba07ba. | | DeepSeek bundle analysis | 2 | Correct final answers with eight and ten REA calls, but both processes exceeded the 120-second ceiling after completion. | | Missing target, each of Pi/Hermes/DeepSeek | 2 per client | All six asked for the app name/path and made zero REA operations, despite a nearby bundle fixture. Pi/Hermes exited; DeepSeek did not. | | DeepSeek with no REA/skill/MCP | 2 | Final READY, zero tool calls, still alive at the 45-second ceiling. | The initial imported Pi trial recovered from two invalid selectors. Separate authentication/configuration experiments failed before any REA work (one Pi exit-zero transcript contained a model error); they are not positive outcomes. A wrong external sticky-test argument was corrected to the existing environment selector, without a REA code change. Client-owned or uncertain observations are recorded without speculative production fixes: - #1632: Pi code-mode declarations project a valid union selector to `unknown`; two transcripts exposed the lost type shape. This does not prove that type loss alone caused model guesses or that REA's schema is invalid. Investigate the client projection first; retain strict REA selectors. - #1633: Hermes emits an ancient-config warning for a fresh unversioned config. Four empty-map/empty-file controls reproduce it without REA; two missing-file controls do not. Boot maintenance projects an absent stamp to zero while the migration owner distinguishes absent from explicitly old. Do not hardcode a client schema version in REA. - #1636: DeepSeek native headless completion does not end the process on the tested Codex adapter; no-REA controls reproduce it. Investigate client/provider lifecycle ownership rather than adding a REA forced-exit workaround. DeepSeek headless tool-result previews were truncated. The follow-up reused the installed producer's frame scanner to decode **every** complete frame of canonical session logs, with no torn tails. All 18 REA inputs and complete outputs passed their named contracts; final cited Evidence IDs matched returned Evidence. Complete logs contained no transport credential. A preliminary one-frame decoder was recognized as incomplete and replaced; its partial record is not a complete-session pass. Two independent **empty npm caches** verified the actual first-use `npx --yes rea-agents@6.3.0` Codex workflow. Cold acquisition took about 49 seconds each. Unapproved setup reported needs_confirmation/exit 1 with no client/skill writes; dry-run showed the plan; approved setup installed six canonical release skill files and pinned MCP registration; repeat had no actions or config changes; doctor was healthy. Both actual CLI analysis and the registered npx-launched SDK MCP server returned Unicode-path fixture Evidence; all 139 release schemas were checked. Two additional native PTY runs reached the approval prompt and cancelled, preserving the original config without backups or skill writes. A preliminary unsupported CLI spelling and a missing pexpect in a private Python were harness errors, corrected using the public MCP path and existing system Python. Every heavy local command/model run was sequential under one CPU and a 3 GiB process-tree ceiling. The largest sampled second-phase model scope was about 720 MiB; complete static checking peaked at about 2.61 GiB. No second-phase host OOM occurred. Original OAuth stores remained unchanged and temporary access-only files were removed. Private audit receipts/transcripts are retained outside the original development checkout. Remaining TODOs stay open: other named clients, real DeepSeek/other model providers, native Windows/macOS first-use onboarding, and real deep-analysis engines. Passing native fixtures or package/platform CI does not establish those workflows. Merging these changes does not publish a new npm package.

bsakaguchi added the

labels

2026-10-15 18:25:58 +00:00

bsakaguchi referenced this issue

2026-10-16 21:13:48 +00:00

[PR #535] [MERGED] fix(mcp): emit valid schemas for empty arrays #2055

bsakaguchi referenced this issue

2026-10-16 21:13:48 +00:00

[PR #536] [MERGED] fix(javascript): honor computed keys when naming callables, exports, requests and options #2057

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:17 +00:00

bsakaguchi commented

2026-11-03 21:43:16 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Nov 3, 2026):

TL;DR: Two commits landed on the trace TODO. I found none that claim the other three open TODOs. This is a read of main at c3647ade1 (#8824), and nothing ran, so it says what landed, not that it works.

Other agent clients: no commit I found claims a skill or onboarding run in them.
Real analysis providers: no commit I found claims this.
Other model providers, macOS and Windows first use: no commit I found claims this.
Trace expansion and client output limits: neither commit names this issue. One is 7c389cbefc (#7743). It keeps two-way traces from crossing containers. The other is 423059a4de (#8522). It pages retained traces through analysis views. The JS guide (lines 168-170) still says such a trace can reach other modules. I did not test whether it still does.

The skill is now version 59. The body says 37.

🤖 Written with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Nov 3, 2026): **TL;DR:** Two commits landed on the trace TODO. I found none that claim the other three open TODOs. This is a read of main at c3647ade1 (#8824), and nothing ran, so it says what landed, not that it works. - **Other agent clients:** no commit I found claims a skill or onboarding run in them. - **Real analysis providers:** no commit I found claims this. - **Other model providers, macOS and Windows first use:** no commit I found claims this. - **Trace expansion and client output limits:** neither commit names this issue. One is 7c389cbefc (#7743). It keeps two-way traces from crossing containers. The other is 423059a4de (#8522). It pages retained traces through analysis views. The JS guide (lines 168-170) still says such a trace can reach other modules. I did not test whether it still does. The skill is now version 59. The body says 37. 🤖 Written with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#535

No description provided.

Rows
Columns