[PR #1778] [MERGED] fix(ida): wait out a transient busy worker health in headless sessions #4811

New issue

Closed

opened 2026-10-31 00:55:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:55:19 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1778
Author: @nikolay-dementiev
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @morluto

Base: main ← Head: fix/1766-ida-busy-health

📝 Commits (8)

7507c71 test(ida): expect a transient busy worker health to be waited out
454fdf5 fix(ida): wait out a transient busy worker health before reading identity
7118ed6 docs(ida): record the verified macOS arm64 headless workflow and the busy-health wait
5345091 test(ida): pin the busy wait deadline, cause and reported tool, and assert clean release
ff652e6 fix(ida): repeat only a tool-name-shaped busy tool and keep the wait in its own module
ca97e61 test(ida): drive the busy wait with a virtual clock and pin its bound and cancellation
813c4dd fix(ida): stop the busy wait at its deadline without another read and honour a late abort
2751dcf docs(ida): state when the busy wait times out, cancels or reports an unknown tool

📊 Changes

7 files changed (+524 additions, -12 deletions)

View changed files

📝 docs/ida-provider.md (+2 -2)
📝 docs/testing.md (+9 -3)
➕ src/ida/IdaBusyHealth.test.ts (+133 -0)
➕ src/ida/IdaBusyHealth.ts (+74 -0)
📝 src/ida/IdaProtocolValues.ts (+7 -0)
📝 src/ida/IdaSessionClient.test.ts (+292 -3)
📝 src/ida/IdaSessionClient.ts (+7 -4)

📄 Description

Summary

A freshly opened headless IDA worker can answer server_health with status: "busy" before it reports its identity. REA parsed that first answer strictly, so a transient state surfaced as the non-retryable unreadable_output. The headless session now re-reads the health for a bounded time instead.

Fixes #1766

Problem and expected behavior

Right after idb_open, server_health(database) can return {"status":"busy","uptime_sec":1.041,"busy_tool":"config_json_get","busy_sec":0.0,"queued_calls":0}; a later call returns the full {"status":"ok", input_path, idb_path, imagebase, ...}. modernHealthSchema requires status: "ok" and every identity field, so the first answer was rejected as malformed producer data and reported as unreadable_output with retryable: false. On macOS arm64 with IDA 9.1 the real verify:ida lane failed on this in 5 of 6 runs.

Expected: a busy worker is waited out within the registration's request deadline, cancellation ends the wait at once, and a worker that stays busy past the deadline is reported as a retryable timeout that names the busy tool the worker reported. Any other unexpected health payload stays unreadable_output.

Change and scope

src/ida/IdaBusyHealth.ts (new): readSettledHealth reads server_health and, only when the payload matches the narrow busy shape (status: "busy", optional string busy_tool, other fields passed through), waits 100 ms and reads again. The wait ends when the worker answers, the caller's signal aborts, or timeoutMs (the registration's request deadline, default 300000) has elapsed since the first read. The deadline is checked again after every pause, so no further read is started once it has passed, and the signal is checked before every retry read.
Past the deadline it throws AnalysisTimeoutError("server_health: headless worker still busy in <busy_tool>", timeoutMs, { cause: <last busy payload> }), the existing provider-neutral retryable timeout (provider_timeout). busy_tool is repeated only when it is tool-name-shaped ([A-Za-z0-9_.:-], at most 64 characters); an omitted or other value is reported as unknown, so producer text is never echoed into an error.
src/ida/IdaProtocolValues.ts: the narrow busyHealthSchema. modernHealthSchema is unchanged, so every other non-ok payload still fails as before; a payload with status: "busy" but an invalid shape (for example a non-string busy_tool) is rejected by the busy schema, so its unreadable_output detail names the busy-shape problem instead of listing the missing ok fields.
src/ida/IdaSessionClient.ts: the headless branch of #observe calls the new helper; #observe now receives the execution options so the caller's signal reaches the wait (private seam).
Docs: docs/ida-provider.md and docs/testing.md describe the wait and record the macOS arm64 headless workflow with IDA Professional 9.1 at upstream ida-pro-mcp c133c3853faa111a9b00ee615c013b720d0c4acd, after the real lane passed. The Windows statements are kept; Linux and other IDA releases stay unverified.

Intentionally not included:

The attached (GUI, legacy) profile is unchanged.
A structured busy state on the error (a provider_state: "busy" / retry_action: "wait" style field shared with the Hopper path) is out of scope here; the busy tool is carried in the existing operation text.
A missing procedure reported as unreadable_output is a separate issue (#1769).
Each upstream request still has its own timeoutMs (no per-call timeout is added to the connection), so a health read already in flight at the deadline can extend the total beyond one timeoutMs; a valid ok that arrives after the deadline is accepted.

Contract and boundary impact

Semantic owner and earliest changed stage: src/ida headless session identity read (#observe), before any operation result is produced.
CLI and MCP/tool-catalog contract: none. No tool, parameter, schema or error code changes; the new outcome reuses provider_timeout with its existing details.operation and details.timeout_ms.
Provider, bridge, target-format, or platform compatibility: IDA headless database-supervisor profile only; verified on macOS arm64 with IDA 9.1. Attached profile and Hopper/Ghidra unchanged.
Evidence, artifact, provenance, or reconstruction contract: none. Evidence for a settled worker is identical; only the previously unreachable busy-then-ok sequence now succeeds. The raw busy payload is kept as the error cause, not projected.
Process execution, authorization, cleanup, or containment impact: none to authorization. The wait honours the caller's abort and close() still releases the owned worker and private workspace after a timeout or cancellation (tested).
Generated metadata/package/installation impact: none.

Evidence and regression coverage

Tests added or updated: src/ida/IdaBusyHealth.test.ts drives the wait through an internal, defaulted clock seam with a virtual clock (no real sleeps): exact read count and pauses, the remaining-budget clamp, no read after expiry, an abort after the pause has fulfilled, and the last busy tool being the one reported. src/ida/IdaSessionClient.test.ts at the session-client boundary with the recording fake connection (no module mocks): busy then ok after re-reads, a busy observation taken after an operation and its cancellation, an invalid busy payload keeping its own diagnostic; busy forever gives provider_timeout, retryable, the exact operation naming the busy tool, the raw payload as cause, a wait of at least the deadline and a bounded number of reads; an omitted busy_tool reported as unknown; free-form or over-long busy_tool text not echoed; abort during the wait ends it promptly as AnalysisCancelledError; unknown status, non-string busy_tool, a payload without status, and busy followed by a malformed payload stay unreadable_output without further waiting; every case asserts a clean close().
Base reproduction or other evidence: the new busy tests fail on the unfixed code (4 failed on the original base: busy then ok, busy forever, unknown tool, abort during the wait; the malformed-payload cases pass there by design). Real lane before the fix on macOS 26.5 arm64, IDA 9.1, fixture built from tests/conformance/ida/inventory.c with clang -O0 -arch arm64, procedure _rea_fixture_add: 1 of 6 runs passed; the failures were in different MCP phases and carry unreadable_output, retryable: false, "status: Invalid input: expected "ok"" with the identity fields undefined. After the fix: 6 of 6, 5 of 5 and then 2 of 2 on the final code, each passed: true, mode headless, all five checks.

User-visible CLI/MCP output: the busy-forever case, sanitized:

{ "code": "provider_timeout", "retryable": true,
  "details": { "operation": "server_health: headless worker still busy in config_json_get", "timeout_ms": 300000 } }

Remaining proof gaps: only macOS arm64 with IDA 9.1 was run for real; Linux and other IDA releases are unverified. The busy window is intermittent, so a passing real run does not by itself prove the wait; the deterministic tests do.
Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run check:fast, npm run check, npm run docs:check, npm run verify:test-discovery (816 files) - green
npm run test:focused over the six src/ida test files - 49 passed
npm run test:fast - 377 files, 4292 passed, 7 skipped, 0 failed
REA_IDA_MCP_CONFIG=$HOME/ida-mcp.json npm run verify:ida -- --target <inventory> --procedure _rea_fixture_add --report <path> - before the fix 1 of 6 passed, after the fix 6 of 6, 5 of 5 and 2 of 2 on the final code passed; no IDA worker was left running
Mutation checks, each reverted: removing the busy branch fails 7 session-client tests; echoing busy_tool unfiltered fails 2; changing the pause 100 to 200 ms fails 3; removing the remaining-budget clamp fails 1; removing the post-pause deadline check fails 2; removing the pre-retry abort check fails 1; dropping the invalid-busy diagnostic fails 1

macOS arm64 only; Linux and Windows were not executed for this change. test:boundary and test:acceptance were not run because no boundary or acceptance file is touched.

Compatibility, safety, and release

Breaking changes or migration steps: none. A worker that answered ok first behaves as before; the only changed outcome is busy-then-ok now succeeding and busy-until-deadline becoming a retryable timeout instead of unreadable_output.
Real Hopper/Ghidra, browser, or OS coverage: real IDA 9.1 on macOS arm64 (headless); Hopper, Ghidra and browser not applicable.
Package or release metadata impact: none.
Security, privacy, process, or containment review: producer-supplied busy_tool is repeated only when tool-name-shaped, the raw payload stays in the non-projected cause, the wait is bounded by timeoutMs (no read starts after it) and by the caller's signal (checked before every retry read), and a timeout or cancellation leaves cleanup of the owned worker and private workspace to the existing close() path.

Review checklist

The change addresses a concrete problem, not an unscoped refactor.
The PR has one focused outcome, and the title follows type(scope): outcome.
Related issue is linked.
Tests cover changed behavior and failure paths.
Owning docs, contracts, and generated metadata are updated.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1778 **Author:** [@nikolay-dementiev](https://github.com/nikolay-dementiev) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/1766-ida-busy-health` --- ### 📝 Commits (8) - [`7507c71`](https://github.com/morluto/rea/commit/7507c71d5f7b28f0848e08e3008a2029a709145f) test(ida): expect a transient busy worker health to be waited out - [`454fdf5`](https://github.com/morluto/rea/commit/454fdf5f9f3922ac7a3a414dcbf91dccfba4dfed) fix(ida): wait out a transient busy worker health before reading identity - [`7118ed6`](https://github.com/morluto/rea/commit/7118ed6239008eb72cb1aad4eb4e3320a8de0b63) docs(ida): record the verified macOS arm64 headless workflow and the busy-health wait - [`5345091`](https://github.com/morluto/rea/commit/53450912e1108a57a8e20ff8db6e0192f99ea883) test(ida): pin the busy wait deadline, cause and reported tool, and assert clean release - [`ff652e6`](https://github.com/morluto/rea/commit/ff652e6bf7b7e2f828b2000ef551998584d18c5a) fix(ida): repeat only a tool-name-shaped busy tool and keep the wait in its own module - [`ca97e61`](https://github.com/morluto/rea/commit/ca97e615df757102eea1ade39433d8cf8c1424f5) test(ida): drive the busy wait with a virtual clock and pin its bound and cancellation - [`813c4dd`](https://github.com/morluto/rea/commit/813c4dd78923abde168b17b5324064f5f9854dd4) fix(ida): stop the busy wait at its deadline without another read and honour a late abort - [`2751dcf`](https://github.com/morluto/rea/commit/2751dcf5f8e05fa53b0521624b96befe1a8cc2c6) docs(ida): state when the busy wait times out, cancels or reports an unknown tool ### 📊 Changes **7 files changed** (+524 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `docs/ida-provider.md` (+2 -2) 📝 `docs/testing.md` (+9 -3) ➕ `src/ida/IdaBusyHealth.test.ts` (+133 -0) ➕ `src/ida/IdaBusyHealth.ts` (+74 -0) 📝 `src/ida/IdaProtocolValues.ts` (+7 -0) 📝 `src/ida/IdaSessionClient.test.ts` (+292 -3) 📝 `src/ida/IdaSessionClient.ts` (+7 -4) </details> ### 📄 Description ## Summary A freshly opened headless IDA worker can answer `server_health` with `status: "busy"` before it reports its identity. REA parsed that first answer strictly, so a transient state surfaced as the non-retryable `unreadable_output`. The headless session now re-reads the health for a bounded time instead. Fixes #1766 ## Problem and expected behavior Right after `idb_open`, `server_health(database)` can return `{"status":"busy","uptime_sec":1.041,"busy_tool":"config_json_get","busy_sec":0.0,"queued_calls":0}`; a later call returns the full `{"status":"ok", input_path, idb_path, imagebase, ...}`. `modernHealthSchema` requires `status: "ok"` and every identity field, so the first answer was rejected as malformed producer data and reported as `unreadable_output` with `retryable: false`. On macOS arm64 with IDA 9.1 the real `verify:ida` lane failed on this in 5 of 6 runs. Expected: a busy worker is waited out within the registration's request deadline, cancellation ends the wait at once, and a worker that stays busy past the deadline is reported as a retryable timeout that names the busy tool the worker reported. Any other unexpected health payload stays `unreadable_output`. ## Change and scope - `src/ida/IdaBusyHealth.ts` (new): `readSettledHealth` reads `server_health` and, only when the payload matches the narrow busy shape (`status: "busy"`, optional string `busy_tool`, other fields passed through), waits 100 ms and reads again. The wait ends when the worker answers, the caller's signal aborts, or `timeoutMs` (the registration's request deadline, default 300000) has elapsed since the first read. The deadline is checked again after every pause, so no further read is started once it has passed, and the signal is checked before every retry read. - Past the deadline it throws `AnalysisTimeoutError("server_health: headless worker still busy in <busy_tool>", timeoutMs, { cause: <last busy payload> })`, the existing provider-neutral retryable timeout (`provider_timeout`). `busy_tool` is repeated only when it is tool-name-shaped (`[A-Za-z0-9_.:-]`, at most 64 characters); an omitted or other value is reported as `unknown`, so producer text is never echoed into an error. - `src/ida/IdaProtocolValues.ts`: the narrow `busyHealthSchema`. `modernHealthSchema` is unchanged, so every other non-`ok` payload still fails as before; a payload with `status: "busy"` but an invalid shape (for example a non-string `busy_tool`) is rejected by the busy schema, so its `unreadable_output` detail names the busy-shape problem instead of listing the missing `ok` fields. - `src/ida/IdaSessionClient.ts`: the headless branch of `#observe` calls the new helper; `#observe` now receives the execution options so the caller's signal reaches the wait (private seam). - Docs: `docs/ida-provider.md` and `docs/testing.md` describe the wait and record the macOS arm64 headless workflow with IDA Professional 9.1 at upstream ida-pro-mcp `c133c3853faa111a9b00ee615c013b720d0c4acd`, after the real lane passed. The Windows statements are kept; Linux and other IDA releases stay unverified. Intentionally not included: - The attached (GUI, legacy) profile is unchanged. - A structured busy state on the error (a `provider_state: "busy"` / `retry_action: "wait"` style field shared with the Hopper path) is out of scope here; the busy tool is carried in the existing `operation` text. - A missing procedure reported as `unreadable_output` is a separate issue (#1769). - Each upstream request still has its own `timeoutMs` (no per-call timeout is added to the connection), so a health read already in flight at the deadline can extend the total beyond one `timeoutMs`; a valid `ok` that arrives after the deadline is accepted. ## Contract and boundary impact - Semantic owner and earliest changed stage: `src/ida` headless session identity read (`#observe`), before any operation result is produced. - CLI and MCP/tool-catalog contract: none. No tool, parameter, schema or error code changes; the new outcome reuses `provider_timeout` with its existing `details.operation` and `details.timeout_ms`. - Provider, bridge, target-format, or platform compatibility: IDA headless database-supervisor profile only; verified on macOS arm64 with IDA 9.1. Attached profile and Hopper/Ghidra unchanged. - Evidence, artifact, provenance, or reconstruction contract: none. Evidence for a settled worker is identical; only the previously unreachable busy-then-ok sequence now succeeds. The raw busy payload is kept as the error cause, not projected. - Process execution, authorization, cleanup, or containment impact: none to authorization. The wait honours the caller's abort and `close()` still releases the owned worker and private workspace after a timeout or cancellation (tested). - Generated metadata/package/installation impact: none. ## Evidence and regression coverage - Tests added or updated: `src/ida/IdaBusyHealth.test.ts` drives the wait through an internal, defaulted clock seam with a virtual clock (no real sleeps): exact read count and pauses, the remaining-budget clamp, no read after expiry, an abort after the pause has fulfilled, and the last busy tool being the one reported. `src/ida/IdaSessionClient.test.ts` at the session-client boundary with the recording fake connection (no module mocks): busy then ok after re-reads, a busy observation taken after an operation and its cancellation, an invalid busy payload keeping its own diagnostic; busy forever gives `provider_timeout`, retryable, the exact operation naming the busy tool, the raw payload as cause, a wait of at least the deadline and a bounded number of reads; an omitted `busy_tool` reported as `unknown`; free-form or over-long `busy_tool` text not echoed; abort during the wait ends it promptly as `AnalysisCancelledError`; unknown status, non-string `busy_tool`, a payload without status, and busy followed by a malformed payload stay `unreadable_output` without further waiting; every case asserts a clean `close()`. - Base reproduction or other evidence: the new busy tests fail on the unfixed code (4 failed on the original base: busy then ok, busy forever, unknown tool, abort during the wait; the malformed-payload cases pass there by design). Real lane before the fix on macOS 26.5 arm64, IDA 9.1, fixture built from `tests/conformance/ida/inventory.c` with `clang -O0 -arch arm64`, procedure `_rea_fixture_add`: 1 of 6 runs passed; the failures were in different MCP phases and carry `unreadable_output`, `retryable: false`, "status: Invalid input: expected \"ok\"" with the identity fields undefined. After the fix: 6 of 6, 5 of 5 and then 2 of 2 on the final code, each `passed: true`, mode `headless`, all five checks. - User-visible CLI/MCP output: the busy-forever case, sanitized: ```json { "code": "provider_timeout", "retryable": true, "details": { "operation": "server_health: headless worker still busy in config_json_get", "timeout_ms": 300000 } } ``` - Remaining proof gaps: only macOS arm64 with IDA 9.1 was run for real; Linux and other IDA releases are unverified. The busy window is intermittent, so a passing real run does not by itself prove the wait; the deterministic tests do. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run check:fast`, `npm run check`, `npm run docs:check`, `npm run verify:test-discovery` (816 files) - green - `npm run test:focused` over the six `src/ida` test files - 49 passed - `npm run test:fast` - 377 files, 4292 passed, 7 skipped, 0 failed - `REA_IDA_MCP_CONFIG=$HOME/ida-mcp.json npm run verify:ida -- --target <inventory> --procedure _rea_fixture_add --report <path>` - before the fix 1 of 6 passed, after the fix 6 of 6, 5 of 5 and 2 of 2 on the final code passed; no IDA worker was left running - Mutation checks, each reverted: removing the busy branch fails 7 session-client tests; echoing `busy_tool` unfiltered fails 2; changing the pause 100 to 200 ms fails 3; removing the remaining-budget clamp fails 1; removing the post-pause deadline check fails 2; removing the pre-retry abort check fails 1; dropping the invalid-busy diagnostic fails 1 macOS arm64 only; Linux and Windows were not executed for this change. `test:boundary` and `test:acceptance` were not run because no boundary or acceptance file is touched. ## Compatibility, safety, and release - Breaking changes or migration steps: none. A worker that answered `ok` first behaves as before; the only changed outcome is busy-then-ok now succeeding and busy-until-deadline becoming a retryable timeout instead of `unreadable_output`. - Real Hopper/Ghidra, browser, or OS coverage: real IDA 9.1 on macOS arm64 (headless); Hopper, Ghidra and browser not applicable. - Package or release metadata impact: none. - Security, privacy, process, or containment review: producer-supplied `busy_tool` is repeated only when tool-name-shaped, the raw payload stays in the non-projected cause, the wait is bounded by `timeoutMs` (no read starts after it) and by the caller's signal (checked before every retry read), and a timeout or cancellation leaves cleanup of the owned worker and private workspace to the existing `close()` path. ## Review checklist - [x] The change addresses a concrete problem, not an unscoped refactor. - [x] The PR has one focused outcome, and the title follows `type(scope): outcome`. - [x] Related issue is linked. - [x] Tests cover changed behavior and failure paths. - [x] Owning docs, contracts, and generated metadata are updated. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:55:19 +00:00