[PR #1613] [MERGED] feat(skills)!: improve Codex onboarding and task recovery #4700

Closed
opened 2026-10-31 00:54:07 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1613
Author: @N0zoM1z0
Created: 10/10/2026
Status: ✅ Merged
Merged: 10/10/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/codex-skill-onboarding


📝 Commits (1)

  • bec5968 feat(skills)!: improve Codex onboarding and task recovery

📊 Changes

12 files changed (+721 additions, -127 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+34 -57)
➕ .agents/skills/reverse-engineer-anything/references/connection-and-recovery.md (+127 -0)
📝 .agents/skills/reverse-engineer-anything/references/javascript-applications.md (+76 -13)
📝 docs/testing.md (+50 -1)
📝 scripts/verify-agent-experience.mjs (+197 -32)
📝 src/application/SetupSkill.ts (+1 -0)
📝 src/evaluation/AgentEvaluationReport.ts (+12 -5)
📝 src/evaluation/CodexAgentEval.ts (+56 -16)
📝 src/generatedPackageMetadata.ts (+1 -1)
📝 tests/boundary/filesystem/setupSkill.test.ts (+2 -2)
📝 tests/evaluation/model-evals/agentEvaluationReport.test.ts (+23 -0)
📝 tests/evaluation/model-evals/codexAgentEval.test.ts (+142 -0)

📄 Description

Summary

Fresh Codex sessions could confuse client approval failures with REA setup/provider failures, invent invalid module-view inputs, or select an unnamed app from an unrelated saved report. Skill version 37 gives a task-centered recovery path, concrete JavaScript view recipes, and target clarification before artifact searching.

Refs #1587. The issue remains open for other clients, models, platforms and real analysis providers.

Problem and expected behavior

An advertised MCP tool can fail with MCP tool call requires approval, but approval policy is never before REA executes it. Repeating setup or installing an engine does not resolve that decision. The agent should identify the actual blocker and continue through an authorized supported workflow.

A module path can identify several graph roles. The agent should request a module page and select its returned node_id, rather than inventing view.kind = "module" or assuming path uniqueness. A request for “my app” without a selected target must ask for an app name/path instead of choosing a nearby fixture.

Change and scope

  • Keep routing concise; move conditional connection, approval, PATH and provider recovery into a focused installed reference. Preserve setup planning, backups, client approvals and target-execution scope.
  • Document summary → retained parent Evidence → module page → node-ID item inspection, limited module-view claims, directed traces, and client preview truncation separately from REA transport errors.
  • Include the new reference in the setup transaction's managed file list. A real installation caught that packaging integration omission; existing byte-equality and idempotence checks cover it.
  • Exercise public setup in independent Codex HOME/CODEX_HOME/XDG directories. Copy authentication only, disable shell snapshots/additional agents, and remove authentication even when retaining fixtures. Keep the disposable account outside the OS temporary directory so current Codex can create its helper aliases without weakening the sandbox.
  • Extend the existing evaluator for Chinese ASAR, module views and missing-target routing. Count complete text-only REA errors and invalid SDK inputs, while distinguishing status verification after successful target open/close from unchanged retries.

Contract and boundary impact

  • Semantic owner and earliest changed stage: authored skill target selection/recovery; setup's managed skill transaction; Codex transcript normalization and evaluation gate.
  • CLI and MCP/tool-catalog contract: no production command/tool schema changes. The optional verify:agent diagnostic report moves from schema 3 to 4.
  • Provider, bridge, target-format, or platform compatibility: connected runtime schemas remain authoritative, including older releases; no engine/bridge changes. Local model testing covers Linux/Codex only.
  • Evidence, artifact, provenance, or reconstruction contract: unchanged. Guidance preserves parent/view identity, coverage, caller-selected values, failed attempts, and explicit runtime unknowns.
  • Process execution, authorization, cleanup, or containment impact: disposable Codex configuration; read-only sandbox; native fixture preapproves only named open_binary/close_binary operations. No global toolchain installs or target execution. Copied authentication is removed in finally.
  • Generated metadata, package, or installation impact: skill version 37 and tracked package metadata; setup installs all seven canonical skill files. Ignored runtime/skill/catalog outputs are regenerated and not committed.

Evidence and regression coverage

  • Tests added or updated: filesystem setup installs and compares the recovery reference; evaluator tests cover text-only ambiguity, SDK discriminator rejection, incomplete JSON, unrecognized structured failures, successful/failed lifecycle verification, nullable first-tool routing, and strict negative-route gating.
  • Base reproduction or other evidence: real Codex transcripts showed invalid/ambiguous view requests and wrong-target selection despite exit code zero. The old installer omitted the new reference, and the old evaluator undercounted text-only/SDK input failures. These observations drove the corrections.
  • User-visible CLI/MCP output: production envelopes are unchanged. A recovered ZIP/HAR task used complete public CLI Evidence; provider_unavailable remained an unavailable outcome. The missing-target result is a validated needs_target clarification with zero REA calls.
  • Remaining proof gaps: most model scenarios use limited routing/heuristic gates or manual workflow review. Only the configured ASAR scenario has a closed four-claim factual oracle. Module lookup still takes multiple pages; this does not repair broad graph semantics or demonstrate universal token savings.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

CI follow-up: the inherited cleanup/uninstall regressions were isolated in PR #1618, whose 21 checks passed before squash merge 16d9c5ad. This PR is rebased onto that main. The prerequisite changes are not part of this diff.

  • npm run test:focused -- tests/boundary/filesystem/setupSkill.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts — 65 tests passed.

  • npm run check — passed, including typecheck/lint/format/metadata/module boundaries/architecture guards.

  • npm run docs:check — passed.

  • npm run verify:package — passed repository-root Skills discovery, exact installed files, package installation, setup planning/backups/idempotence and public CLI/MCP checks.

  • npm pack --dry-run — passed; includes the recovery reference.

  • Skill frontmatter validator — passed.

  • Initial implementation, real Codex CLI 0.162.1 / gpt-6.1-sol — 13 representative final workflow outcomes: native provider unavailability, Chinese ASAR factual rubric, large module views, fresh and polluted-workspace target clarification, ZIP/HAR client-block recovery, skill-only CLI, ordinary-source routing, older-runtime schemas and normal client mode. Earlier failing attempts remain recorded in #1587. Runs were sequential with one CPU core and a 3 GiB process-tree ceiling; final model-run sampled peak was about 930 MiB.

  • Final PR head 47e72a0: repeated the real fresh-profile Codex verifier after rebasing onto latest main. All four selected scenarios (native, asar-zh, javascript-module-view, missing-target) passed their limited gates with zero repeated calls or input-validation failures. The ASAR four-claim factual rubric passed; missing-target made zero REA calls and validated its clarification. Runs remained sequential under one core / 3 GiB, with sampled peak about 859 MiB; copied authentication was removed.

  • Integration revision 82e01131: a fresh four-scenario rerun preserved routing and the ASAR factual rubric, but the ASAR case recovered from one omitted source.kind and therefore failed its strict zero-invalid-input gate. Two additional independent ASAR profiles passed their four-claim rubrics with zero invalid/repeated calls. The failed run is retained in #1587; the evaluator gate was not relaxed and no speculative repair was added. Authentication copies were removed.

  • Final branch head bec59688: 65 focused tests, full npm run check, and npm run docs:check passed after rebasing. Authored skills, setup addition, evaluator and verifier files are byte-identical to the integration revision used for the documented model retests. CI is refreshed on this head.

  • Final head bec59688: repeated all four fresh Codex scenarios after the final rebase. All strict scenario gates passed: zero invalid/repeated calls, successful ASAR four-claim rubric, and validated missing-target clarification with zero REA calls. Selected module-view claims were manually reviewed against the item projection and explicit limits. Runs took about 158 seconds, sampled peak about 835 MiB, and authentication copies were removed. Earlier failed runs remain recorded in #1587.

Compatibility, safety, and release

  • Breaking changes or migration steps: optional evaluation reports now use schemaVersion 4. A schema-3 consumer expecting a string expectedFirstTool must accept null and targetClarificationPassed for missing-target scenarios. Existing analysis cases still expect named tools. Requiring a fabricated tool name would misrepresent the required absence of REA calls, so the report version is explicit; production CLI/MCP contracts are unchanged.
  • Real Hopper/Ghidra, browser, or OS coverage: no real engine validation or other agent/model/platform onboarding claims. Missing-provider diagnostics and package simulations are not engine passes; remaining scope stays in #1587.
  • Package or release metadata impact: bundled skill 36 → 37; package version unchanged. Conventional breaking marker applies to the optional evaluation-report format.
  • Security, privacy, process, or containment review: no blanket approval changes, invented read-only effects, unrelated installs, target execution or GUI mutation. Temporary authentication removed; only authored source and tests committed.

Review checklist

  • This PR addresses a concrete problem or an agreed enhancement.
  • The PR has one focused outcome and the title follows Conventional Commits.
  • Related issue is linked.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible outcomes include representative output above.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1613 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/10/2026 **Status:** ✅ Merged **Merged:** 10/10/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/codex-skill-onboarding` --- ### 📝 Commits (1) - [`bec5968`](https://github.com/morluto/rea/commit/bec59688a5e8faf24701bcc5787fc9283f0a92e2) feat(skills)!: improve Codex onboarding and task recovery ### 📊 Changes **12 files changed** (+721 additions, -127 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+34 -57) ➕ `.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md` (+127 -0) 📝 `.agents/skills/reverse-engineer-anything/references/javascript-applications.md` (+76 -13) 📝 `docs/testing.md` (+50 -1) 📝 `scripts/verify-agent-experience.mjs` (+197 -32) 📝 `src/application/SetupSkill.ts` (+1 -0) 📝 `src/evaluation/AgentEvaluationReport.ts` (+12 -5) 📝 `src/evaluation/CodexAgentEval.ts` (+56 -16) 📝 `src/generatedPackageMetadata.ts` (+1 -1) 📝 `tests/boundary/filesystem/setupSkill.test.ts` (+2 -2) 📝 `tests/evaluation/model-evals/agentEvaluationReport.test.ts` (+23 -0) 📝 `tests/evaluation/model-evals/codexAgentEval.test.ts` (+142 -0) </details> ### 📄 Description ## Summary Fresh Codex sessions could confuse client approval failures with REA setup/provider failures, invent invalid module-view inputs, or select an unnamed app from an unrelated saved report. Skill version 37 gives a task-centered recovery path, concrete JavaScript view recipes, and target clarification before artifact searching. Refs #1587. The issue remains open for other clients, models, platforms and real analysis providers. ## Problem and expected behavior An advertised MCP tool can fail with `MCP tool call requires approval, but approval policy is never` before REA executes it. Repeating setup or installing an engine does not resolve that decision. The agent should identify the actual blocker and continue through an authorized supported workflow. A module path can identify several graph roles. The agent should request a module page and select its returned `node_id`, rather than inventing `view.kind = "module"` or assuming path uniqueness. A request for “my app” without a selected target must ask for an app name/path instead of choosing a nearby fixture. ## Change and scope - Keep routing concise; move conditional connection, approval, PATH and provider recovery into a focused installed reference. Preserve setup planning, backups, client approvals and target-execution scope. - Document summary → retained parent Evidence → module page → node-ID item inspection, limited module-view claims, directed traces, and client preview truncation separately from REA transport errors. - Include the new reference in the setup transaction's managed file list. A real installation caught that packaging integration omission; existing byte-equality and idempotence checks cover it. - Exercise public setup in independent Codex HOME/CODEX_HOME/XDG directories. Copy authentication only, disable shell snapshots/additional agents, and remove authentication even when retaining fixtures. Keep the disposable account outside the OS temporary directory so current Codex can create its helper aliases without weakening the sandbox. - Extend the existing evaluator for Chinese ASAR, module views and missing-target routing. Count complete text-only REA errors and invalid SDK inputs, while distinguishing status verification after successful target open/close from unchanged retries. ## Contract and boundary impact - Semantic owner and earliest changed stage: authored skill target selection/recovery; setup's managed skill transaction; Codex transcript normalization and evaluation gate. - CLI and MCP/tool-catalog contract: no production command/tool schema changes. The optional `verify:agent` diagnostic report moves from schema 3 to 4. - Provider, bridge, target-format, or platform compatibility: connected runtime schemas remain authoritative, including older releases; no engine/bridge changes. Local model testing covers Linux/Codex only. - Evidence, artifact, provenance, or reconstruction contract: unchanged. Guidance preserves parent/view identity, coverage, caller-selected values, failed attempts, and explicit runtime unknowns. - Process execution, authorization, cleanup, or containment impact: disposable Codex configuration; read-only sandbox; native fixture preapproves only named `open_binary`/`close_binary` operations. No global toolchain installs or target execution. Copied authentication is removed in `finally`. - Generated metadata, package, or installation impact: skill version 37 and tracked package metadata; setup installs all seven canonical skill files. Ignored runtime/skill/catalog outputs are regenerated and not committed. ## Evidence and regression coverage - Tests added or updated: filesystem setup installs and compares the recovery reference; evaluator tests cover text-only ambiguity, SDK discriminator rejection, incomplete JSON, unrecognized structured failures, successful/failed lifecycle verification, nullable first-tool routing, and strict negative-route gating. - Base reproduction or other evidence: real Codex transcripts showed invalid/ambiguous view requests and wrong-target selection despite exit code zero. The old installer omitted the new reference, and the old evaluator undercounted text-only/SDK input failures. These observations drove the corrections. - User-visible CLI/MCP output: production envelopes are unchanged. A recovered ZIP/HAR task used complete public CLI Evidence; `provider_unavailable` remained an unavailable outcome. The missing-target result is a validated `needs_target` clarification with zero REA calls. - Remaining proof gaps: most model scenarios use limited routing/heuristic gates or manual workflow review. Only the configured ASAR scenario has a closed four-claim factual oracle. Module lookup still takes multiple pages; this does not repair broad graph semantics or demonstrate universal token savings. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed CI follow-up: the inherited cleanup/uninstall regressions were isolated in [PR #1618](https://github.com/morluto/rea/pull/1618), whose 21 checks passed before squash merge `16d9c5ad`. This PR is rebased onto that main. The prerequisite changes are not part of this diff. - `npm run test:focused -- tests/boundary/filesystem/setupSkill.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts` — 65 tests passed. - `npm run check` — passed, including typecheck/lint/format/metadata/module boundaries/architecture guards. - `npm run docs:check` — passed. - `npm run verify:package` — passed repository-root Skills discovery, exact installed files, package installation, setup planning/backups/idempotence and public CLI/MCP checks. - `npm pack --dry-run` — passed; includes the recovery reference. - Skill frontmatter validator — passed. - Initial implementation, real Codex CLI 0.162.1 / `gpt-6.1-sol` — 13 representative final workflow outcomes: native provider unavailability, Chinese ASAR factual rubric, large module views, fresh and polluted-workspace target clarification, ZIP/HAR client-block recovery, skill-only CLI, ordinary-source routing, older-runtime schemas and normal client mode. Earlier failing attempts remain recorded in #1587. Runs were sequential with one CPU core and a 3 GiB process-tree ceiling; final model-run sampled peak was about 930 MiB. - Final PR head `47e72a0`: repeated the real fresh-profile Codex verifier after rebasing onto latest main. All four selected scenarios (`native`, `asar-zh`, `javascript-module-view`, `missing-target`) passed their limited gates with zero repeated calls or input-validation failures. The ASAR four-claim factual rubric passed; missing-target made zero REA calls and validated its clarification. Runs remained sequential under one core / 3 GiB, with sampled peak about 859 MiB; copied authentication was removed. - Integration revision `82e01131`: a fresh four-scenario rerun preserved routing and the ASAR factual rubric, but the ASAR case recovered from one omitted `source.kind` and therefore failed its strict zero-invalid-input gate. Two additional independent ASAR profiles passed their four-claim rubrics with zero invalid/repeated calls. The failed run is retained in #1587; the evaluator gate was not relaxed and no speculative repair was added. Authentication copies were removed. - Final branch head `bec59688`: 65 focused tests, full `npm run check`, and `npm run docs:check` passed after rebasing. Authored skills, setup addition, evaluator and verifier files are byte-identical to the integration revision used for the documented model retests. CI is refreshed on this head. - Final head `bec59688`: repeated all four fresh Codex scenarios after the final rebase. All strict scenario gates passed: zero invalid/repeated calls, successful ASAR four-claim rubric, and validated missing-target clarification with zero REA calls. Selected module-view claims were manually reviewed against the item projection and explicit limits. Runs took about 158 seconds, sampled peak about 835 MiB, and authentication copies were removed. Earlier failed runs remain recorded in #1587. ## Compatibility, safety, and release - Breaking changes or migration steps: optional evaluation reports now use **schemaVersion 4**. A schema-3 consumer expecting a string `expectedFirstTool` must accept `null` and `targetClarificationPassed` for missing-target scenarios. Existing analysis cases still expect named tools. Requiring a fabricated tool name would misrepresent the required absence of REA calls, so the report version is explicit; production CLI/MCP contracts are unchanged. - Real Hopper/Ghidra, browser, or OS coverage: no real engine validation or other agent/model/platform onboarding claims. Missing-provider diagnostics and package simulations are not engine passes; remaining scope stays in #1587. - Package or release metadata impact: bundled skill 36 → 37; package version unchanged. Conventional breaking marker applies to the optional evaluation-report format. - Security, privacy, process, or containment review: no blanket approval changes, invented read-only effects, unrelated installs, target execution or GUI mutation. Temporary authentication removed; only authored source and tests committed. ## Review checklist - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] The PR has one focused outcome and the title follows Conventional Commits. - [x] Related issue is linked. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible outcomes include representative output above. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 00:54:07 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#4700
No description provided.