morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7142] [Feature] Unicorn-backed function-level emulation provider (building block for #4972) #4285

New issue

Open

opened 2026-10-31 00:48:31 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 00:48:31 +00:00

Owner

Originally created by @reynold-hu on GitHub (Oct 25, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7142

[Feature] Unicorn-backed function-level emulation provider (building block for #4972)

Problem

REA can statically inspect native binaries (Ghidra, Hopper, IDA) and observe
live processes (Frida, debugger adapters), but it has no provider that
executes one native function in isolation under a declared harness. There is
currently no way to ask: "with these registers, this memory image, and these
stubbed imports, what does this function observably do?"

This gap is load-bearing for #4972 (semantic-equivalence verification for
reconstructed native code). Its step 4 requires a differential harness that
runs "the original function and candidate with equivalent initial state and
inputs" and states: "For pointer/ABI-sensitive functions, require an
appropriate native harness or dual-emulation strategy." No provider today
supplies the emulation side of that strategy.

Precision about the current state: Unicorn is not absent from this project.
Unicorn 2.1.2 is pinned inside the pwntools binary-diagnostics profile
(pwntools@4.15.0; pyelftools@0.33; unicorn@2.1.2), where it is used only for
static PLT-map derivation. The contract text states explicitly that those
outputs "do not establish target runtime behavior." This proposal elevates
Unicorn from that hidden sub-dependency into a first-class,
Evidence-producing execution provider. It does not introduce Unicorn to the
project, and it does not claim the pwntools lane already executes code.

Proposed scope (Phase 1 only)

One operation, working name emulate_function (name to be confirmed), that
runs a single native function in isolation:

Target: a function inside the currently bound target (symbol or
address), resolved through the existing provider's inventory; or raw
function bytes with an explicitly declared architecture and entry offset.
Phase 1 starts with x86_64 only.
Declared inputs: initial register values, an initial memory image
(address → bytes), and a stack setup. Nothing is read from the host
process or filesystem at run time.
Import/syscall policy: imports and syscalls are stubbed and
recorded, never executed. The harness records each stubbed call (name or
address, arguments as observed) and returns a caller-declared value or a
default zero value, stated in the result. No host syscall is performed.
Budgets: instruction-count cap, wall-time timeout, and address-space
cap, with the pwntools lane's limits as the starting proposal (input
32 MiB, output 64 MiB, address space 3 GiB, CPU 30 s, default timeout
30 s) — to be confirmed. Exceeding any budget stops the run and reports
the stop reason; it is never a partial success.
Stop conditions: normal return, unmapped memory access, unsupported
instruction, stubbed import/syscall, or budget exhaustion. Each is
reported explicitly.

Result model

One run returns, as Evidence:

return_value and final register state (as observed);
memory_writes: a diff of the memory image (address, bytes before, bytes
after), bounded by the output budget;
calls_hit: imports/syscalls reached, each marked executed: false and
stubbed: true, with observed arguments and the value returned by the
stub;
instructions_executed, stop_reason, and the harness declaration
(architecture, entry, initial state digest, budgets, engine version).

Evidence mapping (proposed): confidence: "observed" for effects the run
actually produced; authority: "controlled-replay";
environment.isolation: "process" (the run happens in an owned bridge
subprocess, never in the REA process and never on the host). The result
states the engine version and the harness declaration so a later run can be
compared against the same declaration.

Limitations (Phase 1)

No operating system is modeled: no real syscalls, no threading, no
signals, no filesystem, no network. Anything the function needs beyond
the declared memory and stubs is reported as unobserved, not assumed.
No self-modifying-code or JIT-emitted-code support in Phase 1; code
outside the declared function bytes stops the run with an explicit reason.
x86_64 only. Other architectures (AArch64 next) are later phases, each
needing its own verification.
A finite run establishes observed behavior for the declared inputs only.
It is not a proof of equivalence and must never be reported as one.

Non-goals

Not the #4972 equivalence verdict. This provider does not return PROVEN,
MISMATCH, or TESTED_ONLY; the differential harness and verdict belong to
a later phase built on top of it.
No packer/unpacking workflow, no anti-debugging or anti-instrumentation
evasion, no DRM/license/anti-cheat bypass, no exploit or payload
automation. The harness runs caller-declared bytes under caller-declared
state; it does not defeat protections.
No live-process injection or attachment (Frida and debugger adapters
already cover live observation).
No automatic installation, download, or redistribution of Unicorn or any
other tool. No new npm dependency. No in-process native binding.
No requirement that every REA user install the emulation engine for
ordinary usage; availability is optional and explicit, like every other
engine provider.

Dependency and licensing

Proposed route: reuse the already-pinned pwntools Python bridge
profile (Unicorn 2.1.2) behind the existing caller-supplied-Python
pattern (REA_PWNTOOLS_PYTHON). REA would not install Python or
packages, would not add an npm dependency, and would not need a new
configuration variable for Phase 1. The bridge script would be a new,
separately owned script (not an extension of the diagnostics scripts),
speaking the same owned-subprocess pattern as the existing bridges.
Considered and not proposed: a Node.js native binding. No maintained
npm binding for Unicorn 2.x was found (the unicorn package on npm is an
unrelated string-coloring library; the only JavaScript binding is a
community project with no evidence of Unicorn 2.x support or maintenance).
A node-ffi route would additionally require users to install libunicorn
separately and would add the project's first native dependency. If the
maintainers prefer an in-process route, that is a separate scope decision.
License note for review: Unicorn's core is GPLv2. The proposed
caller-supplied-subprocess pattern does not distribute Unicorn with REA,
matching how the pwntools lane already handles it, but this needs the
license review that #4972's non-goals require ("without explicit approval
and license review").
Host coverage, stated honestly: the pwntools lane is currently gated
to Linux x64 hosts (pwntoolsHostSupported), even though Unicorn wheels
exist for Windows and macOS. Phase 1 inherits that gate. Lifting it for
other hosts is a later phase requiring its own verification; it is not
claimed here.

Relationship to #4972

This provider is step 4's building block, not the epic. It supplies the
"dual-emulation strategy" input (a controlled run of the original function
under a declared harness) on which a later differential harness can run the
original and a candidate side by side. It deliberately does not implement
#4972's symbolic/concolic exploration, formal checks, or verdict model.
Per #4972's own acceptance criteria, it also must not be presented as "a
custom replay/execution subsystem": it composes an existing, already-pinned
engine through a provider adapter, which is exactly the integration
direction #4972 requests ("Integrate existing analysis/verification engines
through provider adapters").

Open questions for the maintainer

Operation naming: is emulate_function acceptable, or should it live
under a different operation family?
Placement: deep provider (registry selection, like Hopper/Ghidra) or an
auxiliary operation set attached to a bound session? Phase 1 is small
either way; the choice affects src/composition/binary.ts wiring.
Target binding: must the emulated function always come from the
currently bound target's inventory, or is raw-bytes-plus-architecture an
acceptable Phase 1 input?
Budgets: are the pwntools-lane limits an acceptable starting point, or
should emulation have its own budget table?
Stub policy: is stub-and-record with a caller-declared return value the
right default, or should unknown imports stop the run by default?
Doctor surface: should emulation availability be its own doctor check,
or ride on the existing pwntools profile verification?

Scope decision request

Per CONTRIBUTING.md ("Fail-closed integration"), this issue requests an
explicit maintainer scope decision on the Phase 1 behavior above before
substantial implementation intended for integration. I am happy to build a
small exploratory prototype and/or a draft PR to inform that decision; I
understand neither authorizes integration, and that a scope expansion needs
a new decision.

Originally created by @reynold-hu on GitHub (Oct 25, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7142 # [Feature] Unicorn-backed function-level emulation provider (building block for #4972) ## Problem REA can statically inspect native binaries (Ghidra, Hopper, IDA) and observe live processes (Frida, debugger adapters), but it has no provider that executes one native function in isolation under a declared harness. There is currently no way to ask: "with these registers, this memory image, and these stubbed imports, what does this function observably do?" This gap is load-bearing for #4972 (semantic-equivalence verification for reconstructed native code). Its step 4 requires a differential harness that runs "the original function and candidate with equivalent initial state and inputs" and states: "For pointer/ABI-sensitive functions, require an appropriate native harness or dual-emulation strategy." No provider today supplies the emulation side of that strategy. Precision about the current state: Unicorn is not absent from this project. Unicorn 2.1.2 is pinned inside the pwntools binary-diagnostics profile (`pwntools@4.15.0; pyelftools@0.33; unicorn@2.1.2`), where it is used only for static PLT-map derivation. The contract text states explicitly that those outputs "do not establish target runtime behavior." This proposal elevates Unicorn from that hidden sub-dependency into a first-class, Evidence-producing execution provider. It does not introduce Unicorn to the project, and it does not claim the pwntools lane already executes code. ## Proposed scope (Phase 1 only) One operation, working name `emulate_function` (name to be confirmed), that runs a single native function in isolation: - **Target**: a function inside the currently bound target (symbol or address), resolved through the existing provider's inventory; or raw function bytes with an explicitly declared architecture and entry offset. Phase 1 starts with **x86_64** only. - **Declared inputs**: initial register values, an initial memory image (address → bytes), and a stack setup. Nothing is read from the host process or filesystem at run time. - **Import/syscall policy**: imports and syscalls are **stubbed and recorded, never executed**. The harness records each stubbed call (name or address, arguments as observed) and returns a caller-declared value or a default zero value, stated in the result. No host syscall is performed. - **Budgets**: instruction-count cap, wall-time timeout, and address-space cap, with the pwntools lane's limits as the starting proposal (input 32 MiB, output 64 MiB, address space 3 GiB, CPU 30 s, default timeout 30 s) — to be confirmed. Exceeding any budget stops the run and reports the stop reason; it is never a partial success. - **Stop conditions**: normal return, unmapped memory access, unsupported instruction, stubbed import/syscall, or budget exhaustion. Each is reported explicitly. ## Result model One run returns, as Evidence: - `return_value` and final register state (as observed); - `memory_writes`: a diff of the memory image (address, bytes before, bytes after), bounded by the output budget; - `calls_hit`: imports/syscalls reached, each marked `executed: false` and `stubbed: true`, with observed arguments and the value returned by the stub; - `instructions_executed`, `stop_reason`, and the harness declaration (architecture, entry, initial state digest, budgets, engine version). Evidence mapping (proposed): `confidence: "observed"` for effects the run actually produced; `authority: "controlled-replay"`; `environment.isolation: "process"` (the run happens in an owned bridge subprocess, never in the REA process and never on the host). The result states the engine version and the harness declaration so a later run can be compared against the same declaration. ## Limitations (Phase 1) - No operating system is modeled: no real syscalls, no threading, no signals, no filesystem, no network. Anything the function needs beyond the declared memory and stubs is reported as unobserved, not assumed. - No self-modifying-code or JIT-emitted-code support in Phase 1; code outside the declared function bytes stops the run with an explicit reason. - x86_64 only. Other architectures (AArch64 next) are later phases, each needing its own verification. - A finite run establishes observed behavior for the declared inputs only. It is not a proof of equivalence and must never be reported as one. ## Non-goals - Not the #4972 equivalence verdict. This provider does not return PROVEN, MISMATCH, or TESTED_ONLY; the differential harness and verdict belong to a later phase built on top of it. - No packer/unpacking workflow, no anti-debugging or anti-instrumentation evasion, no DRM/license/anti-cheat bypass, no exploit or payload automation. The harness runs caller-declared bytes under caller-declared state; it does not defeat protections. - No live-process injection or attachment (Frida and debugger adapters already cover live observation). - No automatic installation, download, or redistribution of Unicorn or any other tool. No new npm dependency. No in-process native binding. - No requirement that every REA user install the emulation engine for ordinary usage; availability is optional and explicit, like every other engine provider. ## Dependency and licensing - **Proposed route**: reuse the already-pinned pwntools Python bridge profile (Unicorn 2.1.2) behind the existing caller-supplied-Python pattern (`REA_PWNTOOLS_PYTHON`). REA would not install Python or packages, would not add an npm dependency, and would not need a new configuration variable for Phase 1. The bridge script would be a new, separately owned script (not an extension of the diagnostics scripts), speaking the same owned-subprocess pattern as the existing bridges. - **Considered and not proposed**: a Node.js native binding. No maintained npm binding for Unicorn 2.x was found (the `unicorn` package on npm is an unrelated string-coloring library; the only JavaScript binding is a community project with no evidence of Unicorn 2.x support or maintenance). A node-ffi route would additionally require users to install libunicorn separately and would add the project's first native dependency. If the maintainers prefer an in-process route, that is a separate scope decision. - **License note for review**: Unicorn's core is GPLv2. The proposed caller-supplied-subprocess pattern does not distribute Unicorn with REA, matching how the pwntools lane already handles it, but this needs the license review that #4972's non-goals require ("without explicit approval and license review"). - **Host coverage, stated honestly**: the pwntools lane is currently gated to Linux x64 hosts (`pwntoolsHostSupported`), even though Unicorn wheels exist for Windows and macOS. Phase 1 inherits that gate. Lifting it for other hosts is a later phase requiring its own verification; it is not claimed here. ## Relationship to #4972 This provider is step 4's building block, not the epic. It supplies the "dual-emulation strategy" input (a controlled run of the original function under a declared harness) on which a later differential harness can run the original and a candidate side by side. It deliberately does not implement #4972's symbolic/concolic exploration, formal checks, or verdict model. Per #4972's own acceptance criteria, it also must not be presented as "a custom replay/execution subsystem": it composes an existing, already-pinned engine through a provider adapter, which is exactly the integration direction #4972 requests ("Integrate existing analysis/verification engines through provider adapters"). ## Open questions for the maintainer 1. Operation naming: is `emulate_function` acceptable, or should it live under a different operation family? 2. Placement: deep provider (registry selection, like Hopper/Ghidra) or an auxiliary operation set attached to a bound session? Phase 1 is small either way; the choice affects `src/composition/binary.ts` wiring. 3. Target binding: must the emulated function always come from the currently bound target's inventory, or is raw-bytes-plus-architecture an acceptable Phase 1 input? 4. Budgets: are the pwntools-lane limits an acceptable starting point, or should emulation have its own budget table? 5. Stub policy: is stub-and-record with a caller-declared return value the right default, or should unknown imports stop the run by default? 6. Doctor surface: should emulation availability be its own doctor check, or ride on the existing pwntools profile verification? ## Scope decision request Per `CONTRIBUTING.md` ("Fail-closed integration"), this issue requests an explicit maintainer scope decision on the Phase 1 behavior above before substantial implementation intended for integration. I am happy to build a small exploratory prototype and/or a draft PR to inform that decision; I understand neither authorizes integration, and that a scope expansion needs a new decision.

bsakaguchi commented

2026-10-31 00:48:32 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 25, 2026):

This would fill a real gap: a function-level emulation provider is exactly the differential harness step 4 of #4972 needs, and doing it behind a provider abstraction keeps the comparison logic independent of the emulator. Suggestions from the design side: make the harness contract explicit up front (register file, memory image, import stubbing, and the observable side effects you compare — return value, memory deltas, syscall/import calls), and consider starting with a minimal x86/x64 slice of Unicorn so the feature lands incrementally instead of waiting for full ISA coverage.

@XinwangTao commented on GitHub (Oct 25, 2026): This would fill a real gap: a function-level emulation provider is exactly the differential harness step 4 of #4972 needs, and doing it behind a provider abstraction keeps the comparison logic independent of the emulator. Suggestions from the design side: make the harness contract explicit up front (register file, memory image, import stubbing, and the observable side effects you compare — return value, memory deltas, syscall/import calls), and consider starting with a minimal x86/x64 slice of Unicorn so the feature lands incrementally instead of waiting for full ISA coverage.

bsakaguchi commented

2026-10-31 00:48:32 +00:00

Author

Owner

@ashishgit4 commented on GitHub (Oct 25, 2026):

/assign

@ashishgit4 commented on GitHub (Oct 25, 2026): /assign

bsakaguchi commented

2026-10-31 00:48:33 +00:00

Author

Owner

@reynold-hu commented on GitHub (Oct 26, 2026):

Thanks for the feedback — both suggestions are already the plan: Phase 1 is a minimal x86_64 slice with an explicit harness contract (registers, memory image, stub policy, and the compared observables: return value, memory deltas, syscall/import calls). I have a working Phase 1 implementation with tests passing locally, and I'm happy to open a draft PR to inform the scope decision once maintainers confirm the direction.

@reynold-hu commented on GitHub (Oct 26, 2026): Thanks for the feedback — both suggestions are already the plan: Phase 1 is a minimal x86_64 slice with an explicit harness contract (registers, memory image, stub policy, and the compared observables: return value, memory deltas, syscall/import calls). I have a working Phase 1 implementation with tests passing locally, and I'm happy to open a draft PR to inform the scope decision once maintainers confirm the direction.

bsakaguchi commented

2026-10-31 00:48:33 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 26, 2026):

Great to hear the Phase 1 plan and that you already have a working implementation with tests passing — opening a draft PR sounds like the right next step so the scope decision can be made against actual code. Two small things that would help the review: (1) spell out the default stub policy for undeclared imports/syscalls (e.g. return 0, log-and-continue, or abort) so the harness contract is unambiguous, and (2) clarify how the initial memory image is sourced — explicit user-provided map, or exported from an existing provider (Ghidra segments / Frida dump). That would make the differential-harness semantics for #4972 easy to verify.

@XinwangTao commented on GitHub (Oct 26, 2026): Great to hear the Phase 1 plan and that you already have a working implementation with tests passing — opening a draft PR sounds like the right next step so the scope decision can be made against actual code. Two small things that would help the review: (1) spell out the default stub policy for undeclared imports/syscalls (e.g. return 0, log-and-continue, or abort) so the harness contract is unambiguous, and (2) clarify how the initial memory image is sourced — explicit user-provided map, or exported from an existing provider (Ghidra segments / Frida dump). That would make the differential-harness semantics for #4972 easy to verify.

bsakaguchi referenced this issue

2026-10-31 01:12:18 +00:00

[PR #4285] feat(native): decode Delphi/C++Builder TPF0 form payloads in PE resources #6157

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425