morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3917] [Feature] Allow bounded larger Mach-O inputs for Apple dispatch metadata inspection #1530

New issue

Closed

opened 2026-10-16 21:06:44 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-16 21:06:44 +00:00

Owner

Originally created by @jaychou813 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3917

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration.
This request includes no credentials, private samples, private paths, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

inspect-native-dispatch-metadata rejects an otherwise readable arm64 Mach-O once the input exceeds the fixed 64 MiB snapshot budget. The rejection happens before metadata decoding. Increasing --max-records cannot change this input-byte limit.

This is a request for a supported, explicitly bounded way to inspect larger files, not a request to remove resource accounting. The current typed rejection is preferable to an incomplete result being reported as complete.

Public reproduction

I reproduced the limit through the published REA 6.4.0 CLI using only a self-authored C fixture on macOS arm64. Neither executable was run.

cat > budget-fixture.c <<'C'
#ifndef PAD_BYTES
#define PAD_BYTES 1024
#endif
const unsigned char fixture_payload[PAD_BYTES] = {1};
int main(void) { return fixture_payload[0]; }
C

clang -arch arm64 -O0 -DPAD_BYTES=1024 budget-fixture.c -o budget-small
clang -arch arm64 -O0 -DPAD_BYTES=68157440 budget-fixture.c -o budget-large

rea inspect-native-dispatch-metadata "$PWD/budget-small" --max-records 50 --json
rea inspect-native-dispatch-metadata "$PWD/budget-large" --max-records 50 --json

Observed with the installed package entrypoint, using Node 24.19.0:

Input	Size on this toolchain	CLI outcome
`budget-small`	16,912 bytes	Exit 0; metadata Evidence returned
`budget-large`	68,706,832 bytes	Exit 1; `resource_constraint`

Exact output projection for the large control:

{
  "code": "resource_constraint",
  "category": "resource_constraint",
  "message": "Apple static metadata input exceeds its 64 MiB snapshot budget",
  "retryable": false,
  "details": {
    "operation": "inspect_native_dispatch_metadata",
    "resource": "memory",
    "reported_limits": { "input_bytes": 67108864 }
  }
}

These fixtures intentionally contain no Objective-C classes; the small control checks the CLI/read/parse path and the large control isolates admission before metadata decoding. They do not establish large class-graph coverage. The fixture contains no code or data from any analyzed application.

Desired outcome and success criteria

Provide a supported way to select a larger finite input budget, or a bounded reader that can inspect selected metadata without retaining the entire image. Keep the 64 MiB default if appropriate.

The selected limit should be discoverable and reported in Evidence/errors. Oversized, malformed, cancelled, or changed inputs must continue to fail explicitly; this request does not ask to bypass those checks.

The current remediation suggests retrying with more memory, but that alone cannot change a fixed pre-read file-size check. If larger inputs remain unsupported, the recovery text should identify that limitation and an applicable alternate provider/workflow.

Expected interaction with targets

Read existing files or captured data. REA may use its own temporary analysis workspace; no selected-target execution or modification is requested.

Verification available so far

I have a prototype or observed results: the two self-authored CLI controls above, plus source inspection. MCP parity and a changed reader with real large Objective-C/Swift metadata have not been tested.

Alternatives considered

Independent Mach-O/Objective-C metadata parsing and selected LLVM disassembly can answer narrow questions, but do not provide the same REA metadata Evidence workflow. A different deep-analysis provider is a separate setup and analysis path.

Possible direction

Consider a validated maximum-input option with a finite ceiling and explicit memory accounting, or range-based reading. Please treat this as a proposal; the appropriate representation and limits are a maintainer design decision.

Version and references

Tested package: rea-agents@6.4.0.
Runtime: Node 24.19.0; macOS arm64; Apple clang/Command Line Tools.
Source inspected separately at a32b4b6ac54a02a8fe9a7cb62861e54473ba5eb0; no CLI rerun of that source revision is claimed.
Fixed maximumBytes and pre-read rejection.
#484 covers the metadata capability. #1783/#1913 concern Ghidra analyzer behavior, which is a different limit owner.

Originally created by @jaychou813 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3917 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched open and closed issues and pull requests for a related proposal. - [x] I understand the contribution scope must be reviewed before integration. - [x] This request includes no credentials, private samples, private paths, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation `inspect-native-dispatch-metadata` rejects an otherwise readable arm64 Mach-O once the input exceeds the fixed 64 MiB snapshot budget. The rejection happens before metadata decoding. Increasing `--max-records` cannot change this input-byte limit. This is a request for a supported, explicitly bounded way to inspect larger files, not a request to remove resource accounting. The current typed rejection is preferable to an incomplete result being reported as complete. ### Public reproduction I reproduced the limit through the published REA 6.4.0 CLI using only a self-authored C fixture on macOS arm64. Neither executable was run. ```sh cat > budget-fixture.c <<'C' #ifndef PAD_BYTES #define PAD_BYTES 1024 #endif const unsigned char fixture_payload[PAD_BYTES] = {1}; int main(void) { return fixture_payload[0]; } C clang -arch arm64 -O0 -DPAD_BYTES=1024 budget-fixture.c -o budget-small clang -arch arm64 -O0 -DPAD_BYTES=68157440 budget-fixture.c -o budget-large rea inspect-native-dispatch-metadata "$PWD/budget-small" --max-records 50 --json rea inspect-native-dispatch-metadata "$PWD/budget-large" --max-records 50 --json ``` Observed with the installed package entrypoint, using Node 24.19.0: | Input | Size on this toolchain | CLI outcome | |---|---:|---| | `budget-small` | 16,912 bytes | Exit 0; metadata Evidence returned | | `budget-large` | 68,706,832 bytes | Exit 1; `resource_constraint` | Exact output projection for the large control: ```json { "code": "resource_constraint", "category": "resource_constraint", "message": "Apple static metadata input exceeds its 64 MiB snapshot budget", "retryable": false, "details": { "operation": "inspect_native_dispatch_metadata", "resource": "memory", "reported_limits": { "input_bytes": 67108864 } } } ``` These fixtures intentionally contain no Objective-C classes; the small control checks the CLI/read/parse path and the large control isolates admission before metadata decoding. They do not establish large class-graph coverage. The fixture contains no code or data from any analyzed application. ### Desired outcome and success criteria Provide a supported way to select a larger finite input budget, or a bounded reader that can inspect selected metadata without retaining the entire image. Keep the 64 MiB default if appropriate. The selected limit should be discoverable and reported in Evidence/errors. Oversized, malformed, cancelled, or changed inputs must continue to fail explicitly; this request does not ask to bypass those checks. The current remediation suggests retrying with more memory, but that alone cannot change a fixed pre-read file-size check. If larger inputs remain unsupported, the recovery text should identify that limitation and an applicable alternate provider/workflow. ### Expected interaction with targets Read existing files or captured data. REA may use its own temporary analysis workspace; no selected-target execution or modification is requested. ### Verification available so far I have a prototype or observed results: the two self-authored CLI controls above, plus source inspection. MCP parity and a changed reader with real large Objective-C/Swift metadata have not been tested. ### Alternatives considered Independent Mach-O/Objective-C metadata parsing and selected LLVM disassembly can answer narrow questions, but do not provide the same REA metadata Evidence workflow. A different deep-analysis provider is a separate setup and analysis path. ### Possible direction Consider a validated maximum-input option with a finite ceiling and explicit memory accounting, or range-based reading. Please treat this as a proposal; the appropriate representation and limits are a maintainer design decision. ### Version and references - Tested package: `rea-agents@6.4.0`. - Runtime: Node 24.19.0; macOS arm64; Apple clang/Command Line Tools. - Source inspected separately at `a32b4b6ac54a02a8fe9a7cb62861e54473ba5eb0`; no CLI rerun of that source revision is claimed. - [Fixed `maximumBytes` and pre-read rejection](https://github.com/morluto/rea/blob/a32b4b6ac54a02a8fe9a7cb62861e54473ba5eb0/src/native/AppleDispatchMetadata.ts#L191). - #484 covers the metadata capability. #1783/#1913 concern Ghidra analyzer behavior, which is a different limit owner.

bsakaguchi commented

2026-10-16 21:06:45 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Agree with the framing: keep the 64 MiB budget as the default, but allow an explicitly bounded opt-in (e.g. a documented --max-input-bytes or config knob) so readable larger Mach-Os can be inspected without weakening the resource accounting for everyone else. Rejecting before metadata decode is the right default — an opt-in larger budget only trades memory/time for the caller who asks for it.

@XinwangTao commented on GitHub (Oct 15, 2026): Agree with the framing: keep the 64 MiB budget as the default, but allow an explicitly bounded opt-in (e.g. a documented --max-input-bytes or config knob) so readable larger Mach-Os can be inspected without weakening the resource accounting for everyone else. Rejecting before metadata decode is the right default — an opt-in larger budget only trades memory/time for the caller who asks for it.

bsakaguchi referenced this issue

2026-10-17 21:12:08 +00:00

[PR #1530] [CLOSED] fix(ci): avoid load-sensitive escape regression timing #2947

bsakaguchi closed this issue

2026-10-31 00:31:15 +00:00

bsakaguchi commented

2026-10-31 00:31:16 +00:00

Author

Owner

@ramchander7834-cloud commented on GitHub (Oct 19, 2026):

OPPO A59

@ramchander7834-cloud commented on GitHub (Oct 19, 2026): OPPO A59

bsakaguchi commented

2026-10-31 00:31:16 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026):

Proposal (implementing now), keeping the 64 MiB default and every integrity/cancellation check:

Budget the bytes actually retained. For a universal target, read only the session-selected slice into memory; stream the whole file through SHA-256 in bounded chunks so a changed file still fails the digest check. Most large macOS binaries are universal, so a 100 MB two-slice file whose selected slice fits now works with no opt-in. Thin files are unchanged.
Explicit opt-in: max_input_bytes (CLI --max-input-bytes), default 67108864, maximum 1073741824 (1 GiB), validated as a positive integer. It bounds the retained image, not the streamed digest.
Evidence records the effective budget and whether the retained range was the whole file or a slice. A rejection names the observed size, the effective budget, its source (default or caller) and the maximum. It no longer says "no runtime override" or suggests more memory.

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026): Proposal (implementing now), keeping the 64 MiB default and every integrity/cancellation check: 1. **Budget the bytes actually retained.** For a universal target, read only the session-selected slice into memory; stream the whole file through SHA-256 in bounded chunks so a changed file still fails the digest check. Most large macOS binaries are universal, so a 100 MB two-slice file whose selected slice fits now works with no opt-in. Thin files are unchanged. 2. **Explicit opt-in:** `max_input_bytes` (CLI `--max-input-bytes`), default 67108864, maximum 1073741824 (1 GiB), validated as a positive integer. It bounds the retained image, not the streamed digest. 3. Evidence records the effective budget and whether the retained range was the whole file or a slice. A rejection names the observed size, the effective budget, its source (default or caller) and the maximum. It no longer says "no runtime override" or suggests more memory.

bsakaguchi commented

2026-10-31 00:31:16 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026):

Scope update: the PR delivers points 2 and 3 of the proposal (max_input_bytes up to 1 GiB, recorded input_admission, actionable refusal). Point 1 (keeping only the selected universal slice in memory) is deferred: the decoder reports whole-file offsets from ~46 sites that index the full buffer, so it needs its own refactor. Until then the budget counts the whole file, as documented.

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026): Scope update: the PR delivers points 2 and 3 of the proposal (`max_input_bytes` up to 1 GiB, recorded `input_admission`, actionable refusal). Point 1 (keeping only the selected universal slice in memory) is deferred: the decoder reports whole-file offsets from ~46 sites that index the full buffer, so it needs its own refactor. Until then the budget counts the whole file, as documented.

bsakaguchi commented

2026-10-31 00:31:16 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

同意 defer point 1 的取舍：decoder 从 ~46 处索引 whole-file buffer，slice-only 保留必须先把 offset 语义改成相对 slice 的，独立成后续 PR 是对的，不阻塞 max_input_bytes 落地。小建议：在 point 1 落地前，refusal 文案里把 "budget 按整个文件计算（slice-only retention 在路线图）" 写明，避免用户以为选 slice 后预算会自动按 slice 计算。

@XinwangTao commented on GitHub (Oct 22, 2026): 同意 defer point 1 的取舍：decoder 从 ~46 处索引 whole-file buffer，slice-only 保留必须先把 offset 语义改成相对 slice 的，独立成后续 PR 是对的，不阻塞 max_input_bytes 落地。小建议：在 point 1 落地前，refusal 文案里把 "budget 按整个文件计算（slice-only retention 在路线图）" 写明，避免用户以为选 slice 后预算会自动按 slice 计算。

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1530

No description provided.

Rows
Columns