morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #8176] [Design] Report verified Kuna export progress when supported #6463

New issue

Open

opened 2026-11-01 21:11:20 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-11-01 21:11:20 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8176

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Can REA report useful, explicitly bounded progress for a shared Kuna project export when the installed version supports it, while preserving the existing atomic export and cancellation semantics?

Context and evidence

REA at 47fa6ab3795d547207001ffb01643c0aa39dcad8 runs the ordinary project export without upstream progress flags. Waiting MCP requests receive the REA phase and elapsed time every ten seconds (src/kuna/KunaSessionClient.ts); those messages do not claim a completed-function count.

On macOS arm64, I compiled an unexecuted 81-function fixture. With direct official Kuna 1.986 decompile-project --progress --jobs 1, I observed .streaming while the process was running, a growing index.jsonl (23 rows during one sample), and no .c file at that sample. The process completed with 81 successes and removed .streaming. A direct --stream run showed a .c file and 45 index rows before completion. These short runs do not test crash recovery, live-feed failures, multiple waiters, or cross-platform behavior.

At the original baseline, Kuna 1.833 was REA's verified release and rejected --progress as unknown. Its --stream is different: it writes incremental artifacts and rejects --assert; upstream documents ordering and other tradeoffs. REA now recommends 1.986 as its verified release; other releases remain usable but unverified. No REA progress integration was exercised in this experiment.

Expected interaction with targets

Read the selected local binary, launch a caller-supplied Kuna process, and observe files inside the owned export staging directory; the exported artifacts remain subject to existing cache publication and cleanup rules. No target execution is proposed.

Boundary details (optional)

Do not equate a progress event with a verified final function or a published export. A missing, malformed, incomplete, or failed feed must not imply completion, and raw feed bytes must be bounded. Consider version/capability discovery and preserve the current elapsed-time fallback for older or unsupported releases. Shared waiters, request cancellation, producer ownership, cleanup, result identity, and MCP timeout behavior all need explicit design.

Options and tradeoffs

Keep the existing elapsed-time messages; use a version-gated progress-only feed when its protocol and version support are verified; or separately consider partial results where useful. Avoid changing to --stream solely for a percentage: it changes artifact behavior and can preclude assertions.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

If pursued, use real-provider tests for progress, unsupported versions, malformed or missing feed, success and mixed failures; verify shared-waiter cancellation, finalization and cleanup, CLI/MCP behavior and atomic publication. Decide what total/completion claims upstream's feed actually permits.

References

REA provider guide: github.com/morluto/rea@47fa6ab379/docs/kuna-provider.md
Upstream CLI: https://github.com/Noelo-Lab/kuna/blob/v1.986/docs/cli.md

Follow-up: interrupted progress and shared waiters

On macOS arm64 with real Kuna 1.986, an owned 1,200-function fixture and --progress, terminating the producer after two live index rows left .streaming and a partial index, with no C file. Retrying a copy completed; its index, C, header and assembly digests matched a separate fresh export. This is one controlled timing, not a general upstream resume guarantee.

On REA main e5e0dc554, a controlled launcher added --progress to the real Kuna command. In a public compact-schema stdio MCP control, cancellation after the live index appeared was followed by successful close, empty owned staging and no published project; a fresh open and assembly request succeeded. In a separate shared-export control, two assembly requests were active, the survivor was still pending when the first was canceled, and one launched project process completed the survivor's request; close left one published project and empty staging. The fixture digest was unchanged. This does not test REA consuming or displaying the upstream progress feed, a naturally occurring cancellation, malformed feed, or cancellation during upstream finalization.

Kuna 1.989 feed consistency control

On macOS arm64 with Kuna 1.989, a source-built 901-function Mach-O (SHA-256 e0c639fbb540ab04c008b0d0c81ec86ffff18ce35e6809901c4e5c8b884fcc46, never executed) completed a direct decompile-project --progress --jobs 1 run. During one live sample, .streaming read both before and after index.jsonl reported schema 1, phase: decompiling, functions_total: 901, functions_done: 0, and functions_failed: 0, while the index read between those marker reads contained 696 lines. Rows had null C offsets and lengths. At completion the index had 901 rows and .streaming had been removed. This is one short run and sequential file reads, not an atomic snapshot or a general timing bound; it demonstrates that simultaneously visible marker counters and index rows must not be combined as one consistent count.

On the same fixture, --stream --jobs 1 exposed 481 rows in one sample, with non-null C offsets and lengths; its marker still reported zero completed in that sample. The final .c and .asm bytes differed from the plain export. In contrast, the final .c and .asm bytes of the progress-only and plain --jobs 1 exports matched. These are observed byte comparisons, not a claim that all results or environments match. The target digest was unchanged. No REA progress-feed integration was tested in this control.

For a future REA design, report any marker counters as last-observed upstream values with their own timestamp and treat index rows as a separately bounded feed; neither proves final publication, successful function analysis, or a consistent cross-file snapshot. Continue to distinguish the existing shared export's final verified outcome from intermediate progress.

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8176 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Can REA report useful, explicitly bounded progress for a shared Kuna project export when the installed version supports it, while preserving the existing atomic export and cancellation semantics? ### Context and evidence REA at `47fa6ab3795d547207001ffb01643c0aa39dcad8` runs the ordinary project export without upstream progress flags. Waiting MCP requests receive the REA phase and elapsed time every ten seconds (`src/kuna/KunaSessionClient.ts`); those messages do not claim a completed-function count. On macOS arm64, I compiled an unexecuted 81-function fixture. With direct official Kuna 1.986 `decompile-project --progress --jobs 1`, I observed `.streaming` while the process was running, a growing `index.jsonl` (23 rows during one sample), and no `.c` file at that sample. The process completed with 81 successes and removed `.streaming`. A direct `--stream` run showed a `.c` file and 45 index rows before completion. These short runs do not test crash recovery, live-feed failures, multiple waiters, or cross-platform behavior. At the original baseline, Kuna 1.833 was REA's verified release and rejected `--progress` as unknown. Its `--stream` is different: it writes incremental artifacts and rejects `--assert`; upstream documents ordering and other tradeoffs. REA now recommends 1.986 as its verified release; other releases remain usable but unverified. No REA progress integration was exercised in this experiment. ### Expected interaction with targets Read the selected local binary, launch a caller-supplied Kuna process, and observe files inside the owned export staging directory; the exported artifacts remain subject to existing cache publication and cleanup rules. No target execution is proposed. ### Boundary details (optional) Do not equate a progress event with a verified final function or a published export. A missing, malformed, incomplete, or failed feed must not imply completion, and raw feed bytes must be bounded. Consider version/capability discovery and preserve the current elapsed-time fallback for older or unsupported releases. Shared waiters, request cancellation, producer ownership, cleanup, result identity, and MCP timeout behavior all need explicit design. ### Options and tradeoffs Keep the existing elapsed-time messages; use a version-gated progress-only feed when its protocol and version support are verified; or separately consider partial results where useful. Avoid changing to `--stream` solely for a percentage: it changes artifact behavior and can preclude assertions. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions If pursued, use real-provider tests for progress, unsupported versions, malformed or missing feed, success and mixed failures; verify shared-waiter cancellation, finalization and cleanup, CLI/MCP behavior and atomic publication. Decide what total/completion claims upstream's feed actually permits. ### References REA provider guide: https://github.com/morluto/rea/blob/47fa6ab3795d547207001ffb01643c0aa39dcad8/docs/kuna-provider.md Upstream CLI: https://github.com/Noelo-Lab/kuna/blob/v1.986/docs/cli.md ### Follow-up: interrupted progress and shared waiters On macOS arm64 with real Kuna 1.986, an owned 1,200-function fixture and `--progress`, terminating the producer after two live index rows left `.streaming` and a partial index, with no C file. Retrying a copy completed; its index, C, header and assembly digests matched a separate fresh export. This is one controlled timing, not a general upstream resume guarantee. On REA main `e5e0dc554`, a controlled launcher added `--progress` to the real Kuna command. In a public compact-schema stdio MCP control, cancellation after the live index appeared was followed by successful close, empty owned staging and no published project; a fresh open and assembly request succeeded. In a separate shared-export control, two assembly requests were active, the survivor was still pending when the first was canceled, and one launched project process completed the survivor's request; close left one published project and empty staging. The fixture digest was unchanged. This does not test REA consuming or displaying the upstream progress feed, a naturally occurring cancellation, malformed feed, or cancellation during upstream finalization. ### Kuna 1.989 feed consistency control On macOS arm64 with Kuna 1.989, a source-built 901-function Mach-O (SHA-256 `e0c639fbb540ab04c008b0d0c81ec86ffff18ce35e6809901c4e5c8b884fcc46`, never executed) completed a direct `decompile-project --progress --jobs 1` run. During one live sample, `.streaming` read both before and after `index.jsonl` reported schema 1, `phase: decompiling`, `functions_total: 901`, `functions_done: 0`, and `functions_failed: 0`, while the index read between those marker reads contained 696 lines. Rows had null C offsets and lengths. At completion the index had 901 rows and `.streaming` had been removed. This is one short run and sequential file reads, not an atomic snapshot or a general timing bound; it demonstrates that simultaneously visible marker counters and index rows must not be combined as one consistent count. On the same fixture, `--stream --jobs 1` exposed 481 rows in one sample, with non-null C offsets and lengths; its marker still reported zero completed in that sample. The final `.c` and `.asm` bytes differed from the plain export. In contrast, the final `.c` and `.asm` bytes of the progress-only and plain `--jobs 1` exports matched. These are observed byte comparisons, not a claim that all results or environments match. The target digest was unchanged. No REA progress-feed integration was tested in this control. For a future REA design, report any marker counters as last-observed upstream values with their own timestamp and treat index rows as a separately bounded feed; neither proves final publication, successful function analysis, or a consistent cross-file snapshot. Continue to distinguish the existing shared export's final verified outcome from intermediate progress.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:11:20 +00:00

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

Follow-up on the PR #8187 branch, macOS arm64, Kuna 1.986, MCP stdio: I built an unexecuted 1,200-function Mach-O and issued two concurrent procedure_assembly calls against a cold, isolated cache. With the normal executable, both completed from the same published export before the 10-second wait notification or the scheduled cancellation; this does not test cancellation during a slow natural export.

For a controlled wait, I used a caller-owned launcher that sleeps 12 seconds before executing the same real Kuna decompile-project command, forwarding all other commands directly. Canceling one call after three seconds produced a client cancellation; the other caller received REA's kuna_export elapsed-wait notification at ten seconds and then successfully returned project-export assembly after the real Kuna command completed. A later call and a separate reopened MCP server read the same export key. Both closes succeeded, the target SHA-256 was unchanged, and the isolated staging directory was empty after completion. The delay is injected before Kuna runs: this is not evidence of native per-function progress, a naturally slow export, or cancellation during final publication/cleanup. The notice's decompiling phase described REA's build phase during the artificial pre-launch delay; it should not be read as a measured count or proof of current worker activity.

The original issue text describes the earlier baseline: #8187 proposes 1.986 as the verified release but remains open. Given the requested preference to avoid compatibility code, a future design need not assume maintaining separate per-version progress paths: consider the selected verified baseline and explicit unverified-release behavior first, while preserving the current atomic publication and honest unknowns. The existing wait notice is useful today; the remaining question is whether bounded upstream progress supplies enough additional value to justify integrating and verifying its feed.

@morluto commented on GitHub (Oct 30, 2026): Follow-up on the PR #8187 branch, macOS arm64, Kuna 1.986, MCP stdio: I built an unexecuted 1,200-function Mach-O and issued two concurrent `procedure_assembly` calls against a cold, isolated cache. With the normal executable, both completed from the same published export before the 10-second wait notification or the scheduled cancellation; this does not test cancellation during a slow natural export. For a controlled wait, I used a caller-owned launcher that sleeps 12 seconds **before** executing the same real Kuna `decompile-project` command, forwarding all other commands directly. Canceling one call after three seconds produced a client cancellation; the other caller received REA's `kuna_export` elapsed-wait notification at ten seconds and then successfully returned project-export assembly after the real Kuna command completed. A later call and a separate reopened MCP server read the same export key. Both closes succeeded, the target SHA-256 was unchanged, and the isolated staging directory was empty after completion. The delay is injected before Kuna runs: this is not evidence of native per-function progress, a naturally slow export, or cancellation during final publication/cleanup. The notice's `decompiling` phase described REA's build phase during the artificial pre-launch delay; it should not be read as a measured count or proof of current worker activity. The original issue text describes the earlier baseline: #8187 proposes 1.986 as the verified release but remains open. Given the requested preference to avoid compatibility code, a future design need not assume maintaining separate per-version progress paths: consider the selected verified baseline and explicit unverified-release behavior first, while preserving the current atomic publication and honest unknowns. The existing wait notice is useful today; the remaining question is whether bounded upstream progress supplies enough additional value to justify integrating and verifying its feed.

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A follow-up controlled MCP trace gives a concrete existing alternative to a new progress operation. While the same artificial pre-launch delay held a cold export open, procedure_pseudo_code for a different function completed through Kuna's single-function path. Its Evidence said raw_result.source.kind: "single-function" and recorded project_export: {state: "building", phase: "decompiling", inventory_functions: 1200}, plus the limitation that a later export can produce different text. The 1,200 count is the inventory, not the number decompiled; the recorded export state is an observation at that call, not a live status subscription. In this same interval, binary_session.analysis_activity was explicitly not_observed, not idle.

This does not show upstream progress integration or a natural long-running Kuna export. It suggests any improvement should assess the existing Evidence and wait notification together, avoid making a caller run an otherwise unwanted decompile just to obtain status, and keep inventory size, worker activity, completion, publication, and Evidence time distinct. #8187's 1.986 baseline remains an open PR.

@morluto commented on GitHub (Oct 30, 2026): A follow-up controlled MCP trace gives a concrete existing alternative to a new progress operation. While the same artificial pre-launch delay held a cold export open, `procedure_pseudo_code` for a different function completed through Kuna's single-function path. Its Evidence said `raw_result.source.kind: "single-function"` and recorded `project_export: {state: "building", phase: "decompiling", inventory_functions: 1200}`, plus the limitation that a later export can produce different text. The 1,200 count is the inventory, not the number decompiled; the recorded export state is an observation at that call, not a live status subscription. In this same interval, `binary_session.analysis_activity` was explicitly `not_observed`, not `idle`. This does not show upstream progress integration or a natural long-running Kuna export. It suggests any improvement should assess the existing Evidence and wait notification together, avoid making a caller run an otherwise unwanted decompile just to obtain status, and keep inventory size, worker activity, completion, publication, and Evidence time distinct. #8187's 1.986 baseline remains an open PR.

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

Real-provider sealing-boundary follow-up (macOS arm64, Kuna 1.986, PR #8187 branch): using the production KunaProjectCache callback as the synchronization point, I observed phase: sealing only after real Kuna had returned and its .c, .h, .asm, README and inventory files existed in isolated staging. No target was executed.

Non-cancelled control published one export and emptied staging.
Cancelling the only cache waiter at that callback returned cancellation. After cache.settled(key), no project was published and staging was empty; a subsequent build succeeded and published normally.
Cancelling one of two cache waiters at the same callback left the other to receive a published project; staging emptied. The target hash stayed unchanged, cache closes succeeded, and a final token-verified process-lineage observation found no descendants.

This is a real Kuna export with cancellation injected at REA's production-boundary callback, not a public MCP finalization-cancellation test and not a cancellation during manifest rename or staging retirement. At the instant the direct cache wait returned, staging still existed; it disappeared only after settlement in the sole-waiter case. The generic projected cancellation contained cleanup: complete at that instant. Whether that field describes only the departed waiter or could mislead an external caller about the still-owned background export needs a public-path/contract check; this observation alone does not establish an MCP bug. A first harness attempt failed on its own result-summary code after the control; two corrected fresh-cache runs produced the observations above.

@morluto commented on GitHub (Oct 30, 2026): Real-provider sealing-boundary follow-up (macOS arm64, Kuna 1.986, PR #8187 branch): using the production `KunaProjectCache` callback as the synchronization point, I observed `phase: sealing` only after real Kuna had returned and its `.c`, `.h`, `.asm`, README and inventory files existed in isolated staging. No target was executed. - Non-cancelled control published one export and emptied staging. - Cancelling the only cache waiter at that callback returned cancellation. After `cache.settled(key)`, no project was published and staging was empty; a subsequent build succeeded and published normally. - Cancelling one of two cache waiters at the same callback left the other to receive a published project; staging emptied. The target hash stayed unchanged, cache closes succeeded, and a final token-verified process-lineage observation found no descendants. This is a real Kuna export with cancellation injected at REA's production-boundary callback, not a public MCP finalization-cancellation test and not a cancellation during manifest rename or staging retirement. At the instant the direct cache wait returned, staging still existed; it disappeared only after settlement in the sole-waiter case. The generic projected cancellation contained `cleanup: complete` at that instant. Whether that field describes only the departed waiter or could mislead an external caller about the still-owned background export needs a public-path/contract check; this observation alone does not establish an MCP bug. A first harness attempt failed on its own result-summary code after the control; two corrected fresh-cache runs produced the observations above.

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A direct Kuna 1.986 assertion-file control is relevant to any future progress integration: serial decompile-project --assert @file --progress --jobs 1 succeeded on one fixture, and the final C/header/assembly digests matched the corresponding serial non-progress assertion export. The same release explicitly rejected --assert with --stream and with --jobs 2. A nonfatal rejected assertion was reported on stderr, and strict mode exited nonzero while still writing artifacts. This is not REA integration or a general guarantee; #8166 records the full assertion experiment.

@morluto commented on GitHub (Oct 30, 2026): A direct Kuna 1.986 assertion-file control is relevant to any future progress integration: serial `decompile-project --assert @file --progress --jobs 1` succeeded on one fixture, and the final C/header/assembly digests matched the corresponding serial non-progress assertion export. The same release explicitly rejected `--assert` with `--stream` and with `--jobs 2`. A nonfatal rejected assertion was reported on stderr, and strict mode exited nonzero while still writing artifacts. This is not REA integration or a general guarantee; #8166 records the full assertion experiment.

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Direct Kuna 1.986 follow-up on macOS arm64, using the existing unexecuted three-function Mach-O and decompile-project --progress --jobs 2 --max-fn-seconds 1. A normal control produced three bodies and no failures. A second run deliberately injected the upstream test-only worker stall for _adjust; this is fault instrumentation, not a natural timeout or a frequency claim.

In the injected run, sampled live index.jsonl rows first reported _main and _evaluate with seq 0 and 1, null c_offset/c_len, and no C file. The third row then reported _adjust with a watchdog error, still before the C file appeared. Later, while .streaming still existed, the index was rewritten in final address order: _adjust became seq 0 and the rows acquired file spans. The failed row's span covered an error comment, not a decompiled body. The process exited 0 and stderr reported 3/3 functions (100.0%), while final output said two succeeded and one failed. The source digest stayed unchanged.

For any REA progress design, these observations mean a completed-attempt count should not be presented as successful bodies; seq in the live feed is not a stable identity across finalization; null spans and the marker do not establish published artifacts; and nonzero final spans alone do not establish a body. The final error field and ordinary REA cache verification remain authoritative for their respective facts. This is a small sampled run of one version/fixture, not proof of every ordering, protocol behavior, or an upstream defect.

@morluto commented on GitHub (Oct 31, 2026): Direct Kuna 1.986 follow-up on macOS arm64, using the existing unexecuted three-function Mach-O and `decompile-project --progress --jobs 2 --max-fn-seconds 1`. A normal control produced three bodies and no failures. A second run deliberately injected the upstream test-only worker stall for `_adjust`; this is fault instrumentation, not a natural timeout or a frequency claim. In the injected run, sampled live `index.jsonl` rows first reported `_main` and `_evaluate` with `seq` 0 and 1, null `c_offset`/`c_len`, and no C file. The third row then reported `_adjust` with a watchdog error, still before the C file appeared. Later, while `.streaming` still existed, the index was rewritten in final address order: `_adjust` became `seq` 0 and the rows acquired file spans. The failed row's span covered an error comment, not a decompiled body. The process exited 0 and stderr reported `3/3 functions (100.0%)`, while final output said two succeeded and one failed. The source digest stayed unchanged. For any REA progress design, these observations mean a completed-attempt count should not be presented as successful bodies; `seq` in the live feed is not a stable identity across finalization; null spans and the marker do not establish published artifacts; and nonzero final spans alone do not establish a body. The final error field and ordinary REA cache verification remain authoritative for their respective facts. This is a small sampled run of one version/fixture, not proof of every ordering, protocol behavior, or an upstream defect.

bsakaguchi referenced this issue

2026-11-01 21:47:18 +00:00

[PR #6464] [MERGED] docs(firmware): document the Windows Unblob submodule checkout #7358

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 13:27:02 +00:00