[GH-ISSUE #2473] [Design] inspect_macho returns 80k–230k tokens for ordinary apps because every symbol is inlined #825

New issue

Closed

opened 2026-10-15 18:28:46 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:28:46 +00:00

Owner

Originally created by @kaoru0822-kitauji on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2473

Summary

inspect_macho always returns every import and export inline, with three nullable fields per symbol. On ordinary macOS apps, one call is 80k–230k estimated tokens. That is far above common MCP client result budgets (Claude Code's default MCP output limit, for example, is 25,000 tokens), so agents get a truncated result or none for the overview they asked for. Every fix I can see changes the output contract, so I'm opening an issue with data instead of a PR, as CONTRIBUTING asks.

Measurements

Main at a2d8b94dd, macOS 26 (Apple silicon), Node 26.5.0, local build. Each target: an MCP stdio client calls open_binary and then inspect_macho with no arguments. Character counts are of the result text (the wire carries it twice, as structuredContent and as text). Tokens are tokenx estimates from the repo's own dependency, not a model tokenizer.

Target (selected slice)	Result chars	Est. tokens	Imports	Exports	Import + export chars	Chars in `null` symbol fields
Ghostty.app `ghostty`	780,760	228,377	2,472	3,529	745,343 (95%)	197,186 (25%)
Keynote.app `Keynote`	756,914	221,514	5,997	1	697,996 (92%)	257,021 (34%)
Calculator.app `Calculator`	293,620	83,423	2,059	1	259,733 (88%)	88,539 (30%)
Transmission.app `Transmission`	111,433	35,237	704	119	88,164 (79%)	31,926 (29%)
`/bin/ls`	20,894	6,843	90	1	8,081 (39%)	3,884 (19%)
Safari.app `Safari` (stub)	12,038	3,915	1	1	451	69

A typical import item from Calculator:

{"name":"_$s9Calculate0A10ExpressionC04RichB0C6isZeroSbvgTj","address":null,"weak":null,"reexport":null,"source":"Calculate"}

On these targets, address, weak and reexport are null on every import. That is 19–34% of the whole result. The "source" key and value repeat on every import (for example, 133,563 chars of Keynote's result).

Everything else (header, load commands, dependencies, segments, provenance, limitations) is much smaller: 12k (Safari) to 59k (Keynote) chars.

Options (for maintainers to choose)

Facet selection. Add an input such as symbols: "all" | "counts" | {offset, limit}, or a facet list. The default stays the complete result, so existing callers are unaffected. Agents that want an overview ask for counts and page through symbols. This follows the "supported selection, summaries or views" guidance in docs/tool-design.md. It would also fit inspect_analysis_view, which already projects views of completed Evidence.
Facet-level unknowns instead of per-item nulls. When a source never observes a field for a whole facet (dyld_info imports never carry an address here), state that once in the facet (for example unobserved_fields: ["address", "weak", "reexport"]) and omit it per item. This keeps "unknown" explicit without repeating it per symbol. It cuts 19–34% on its own but still leaves Keynote above 140k tokens.
Group imports by source. Return {source, names[]} rows. That saves roughly another 15–18% on import-heavy targets, at the cost of a different shape from exports.

Option 1 is the only one that brings large apps under typical client budgets. Options 2 and 3 reduce the complete-result case.

Not measured

Real clients' truncation behavior. I only measured REA's output.
Linux/ELF and other native inspection tools, which may have the same pattern.
binary_session is a separate, smaller case (about 45k chars here, mostly tool_availability and analysis_provider_candidates). I haven't proposed anything for it.

AI agent note

An AI agent (Claude Code) collected these measurements and wrote this issue during a scheduled maintenance loop. It made no code changes for this issue.

Originally created by @kaoru0822-kitauji on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2473 ### Summary `inspect_macho` always returns every import and export inline, with three nullable fields per symbol. On ordinary macOS apps, one call is 80k–230k estimated tokens. That is far above common MCP client result budgets (Claude Code's default MCP output limit, for example, is 25,000 tokens), so agents get a truncated result or none for the overview they asked for. Every fix I can see changes the output contract, so I'm opening an issue with data instead of a PR, as CONTRIBUTING asks. ### Measurements Main at a2d8b94dd, macOS 26 (Apple silicon), Node 26.5.0, local build. Each target: an MCP stdio client calls `open_binary` and then `inspect_macho` with no arguments. Character counts are of the result text (the wire carries it twice, as `structuredContent` and as text). Tokens are `tokenx` estimates from the repo's own dependency, not a model tokenizer. | Target (selected slice) | Result chars | Est. tokens | Imports | Exports | Import + export chars | Chars in `null` symbol fields | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Ghostty.app `ghostty` | 780,760 | 228,377 | 2,472 | 3,529 | 745,343 (95%) | 197,186 (25%) | | Keynote.app `Keynote` | 756,914 | 221,514 | 5,997 | 1 | 697,996 (92%) | 257,021 (34%) | | Calculator.app `Calculator` | 293,620 | 83,423 | 2,059 | 1 | 259,733 (88%) | 88,539 (30%) | | Transmission.app `Transmission` | 111,433 | 35,237 | 704 | 119 | 88,164 (79%) | 31,926 (29%) | | `/bin/ls` | 20,894 | 6,843 | 90 | 1 | 8,081 (39%) | 3,884 (19%) | | Safari.app `Safari` (stub) | 12,038 | 3,915 | 1 | 1 | 451 | 69 | A typical import item from Calculator: ```json {"name":"_$s9Calculate0A10ExpressionC04RichB0C6isZeroSbvgTj","address":null,"weak":null,"reexport":null,"source":"Calculate"} ``` On these targets, `address`, `weak` and `reexport` are `null` on every import. That is 19–34% of the whole result. The `"source"` key and value repeat on every import (for example, 133,563 chars of Keynote's result). Everything else (header, load commands, dependencies, segments, provenance, limitations) is much smaller: 12k (Safari) to 59k (Keynote) chars. ### Options (for maintainers to choose) 1. **Facet selection.** Add an input such as `symbols: "all" | "counts" | {offset, limit}`, or a facet list. The default stays the complete result, so existing callers are unaffected. Agents that want an overview ask for counts and page through symbols. This follows the "supported selection, summaries or views" guidance in `docs/tool-design.md`. It would also fit `inspect_analysis_view`, which already projects views of completed Evidence. 2. **Facet-level unknowns instead of per-item `null`s.** When a source never observes a field for a whole facet (dyld_info imports never carry an address here), state that once in the facet (for example `unobserved_fields: ["address", "weak", "reexport"]`) and omit it per item. This keeps "unknown" explicit without repeating it per symbol. It cuts 19–34% on its own but still leaves Keynote above 140k tokens. 3. **Group imports by `source`.** Return `{source, names[]}` rows. That saves roughly another 15–18% on import-heavy targets, at the cost of a different shape from `exports`. Option 1 is the only one that brings large apps under typical client budgets. Options 2 and 3 reduce the complete-result case. ### Not measured - Real clients' truncation behavior. I only measured REA's output. - Linux/ELF and other native inspection tools, which may have the same pattern. - `binary_session` is a separate, smaller case (about 45k chars here, mostly `tool_availability` and `analysis_provider_candidates`). I haven't proposed anything for it. ### AI agent note An AI agent (Claude Code) collected these measurements and wrote this issue during a scheduled maintenance loop. It made no code changes for this issue.

bsakaguchi

2026-10-15 18:28:46 +00:00

closed this issue
added the
area: domain

enhancement
labels

bsakaguchi commented

2026-10-15 18:28:47 +00:00

Author

Owner

@morluto commented on GitHub (Oct 13, 2026):

Verified the compatible opt-in summary delivery requested in this issue is already on main via #2634 (commit bf34d0d2f). inspect_macho supports caller-selected summary, preserves complete retained Evidence, and inspect_analysis_view pages exact rows with coverage. Current tests/boundary/mcp/nativeMachoSummaryMcp.test.ts passed (2 tests) with the built runtime and real SDK transport; the source view suite also passed (4 tests). Complete-by-default behavior is preserved as requested. No new token/time/CPU/memory performance claim is made. Closing the implemented scope.

@morluto commented on GitHub (Oct 13, 2026): Verified the compatible opt-in summary delivery requested in this issue is already on main via #2634 (commit bf34d0d2f). `inspect_macho` supports caller-selected summary, preserves complete retained Evidence, and `inspect_analysis_view` pages exact rows with coverage. Current `tests/boundary/mcp/nativeMachoSummaryMcp.test.ts` passed (2 tests) with the built runtime and real SDK transport; the source view suite also passed (4 tests). Complete-by-default behavior is preserved as requested. No new token/time/CPU/memory performance claim is made. Closing the implemented scope.

bsakaguchi referenced this issue

2026-10-16 21:16:30 +00:00

[PR #825] [MERGED] fix(browser): preserve form destinations and satisfies mutation uncertainty #2266

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:42 +00:00