morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[PR #4473] feat(dex): inspect Android DEX source-line and local-variable debug metadata #6263

New issue

Open

opened 2026-10-31 01:13:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:13:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4473
Author: @FerroLx
Created: 10/16/2026
Status: 🔄 Open

Base: main ← Head: feat/dex-debug-metadata-standalone

📝 Commits (10+)

0cd3596 feat(dex): inspect Android DEX source-line and local-variable debug metadata
be8baa3 fix(dex): bound class_data parsing and summary working sets
b2d9022 fix(dex): stream class_defs and methods instead of materialising them
c025f8e fix(dex): skip zero-length local scopes; size the work budget per member
f72d238 docs(dex): state the budget, zero-length and cancellation guarantees precisely
14bd4fc fix(dex): address the Greptile review on #4473
44b214a fix(dex): address the Codex review on #4473 and bound the class lookup
1ff7225 fix(dex): stop string scans once the budget is spent; retain the magic-peek handle
8b1c284 fix(dex): fail closed on unverifiable class, code-item and debug facts; load multidex like ART
66db884 fix(dex): keep archive member selection array-bound and describe the multidex sequence

📊 Changes

82 files changed (+11457 additions, -110 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+6 -0)
📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+15 -0)
📝 README.md (+39 -38)
📝 docs/architecture.mermaid (+3 -0)
➕ docs/dex-debug-metadata.md (+215 -0)
📝 docs/index.md (+14 -14)
➕ docs/specs/dex-debug-metadata/plan.md (+26 -0)
➕ docs/specs/dex-debug-metadata/spec.md (+119 -0)
➕ docs/specs/dex-debug-metadata/tasks.md (+22 -0)
➕ docs/specs/dex-debug-metadata/verification.md (+82 -0)
📝 experiments/analysis-efficiency/source-to-bundle.mjs (+1 -2)
📝 package.json (+1 -0)
📝 scripts/lib/catalog-builders.mjs (+5 -0)
📝 scripts/lib/catalog-core.mjs (+2 -0)
📝 scripts/lib/test-projects.mjs (+1 -1)
➕ scripts/verify-dex-debug-metadata.mjs (+435 -0)
📝 src/application/WebSourceLocationPorts.ts (+4 -2)
➕ src/application/dex/DexDebugMetadataPort.ts (+25 -0)
➕ src/application/dex/DexDebugMetadataService.test.ts (+53 -0)
➕ src/application/dex/DexDebugMetadataService.ts (+101 -0)

...and 62 more files

📄 Description

Independent of #3586 (JVM debug metadata). This branch is cut from main and carries only the DEX tool, so either PR can merge first. The few helpers both readers need are shared modules introduced here: the modified-UTF8 decoder, archive-member selection and reading, and the magic peek. Whichever PR merges second will merge main and adopt them in place of its own copies. Both PRs also carry the same shared ZIP-reader changes: storedPath is the raw central-directory name, a member is a directory only when its stored name ends in /, and the abort reason is kept as the cancellation cause. In ZipArtifactReader.ts they differ by one line, normalizeArchiveMemberPath here against the equivalent ./ strip in #3586; the second to merge keeps normalizeArchiveMemberPath.

Summary

Closes #4538.

Adds inspect_dex_debug_metadata (CLI rea inspect-dex-debug-metadata), the Android DEX sibling of inspect_jvm_debug_metadata (#3586). It statically reports the compiler-emitted debug metadata in a selected local .dex, or the classes*.dex members of an APK/ZIP. That lets an analyst judge debugging readiness before attaching a debugger or reading a decompilation: do source files, line positions, parameter names and local-variable names, types and scopes survive, or were they stripped by R8/ProGuard?

Problem and expected behavior

When triaging an APK, the first practical question is whether there is debug metadata to work with: a debug build, a release build with line numbers kept, or a fully stripped R8 release. Today that takes dexdump from the Android SDK and manual counting, and an agent can't get it as Evidence.

Change and scope

The tool works at two granularities. A DEX holds thousands of classes, so neither alone is enough.

Summary (default): one entry per DEX member. For an APK, these are the members ART's multidex loader loads: classes.dex, classes2.dex, … while each is present under its exact stored central-directory name. The raw name is used, never an Info-ZIP Unicode Path name. At most 64 members are read; past that, REA refuses and asks for entry. It gives method tallies, line positions, local declarations (with explicit_local_declarations separating locals the stream declares from the implicit this/parameter locals), distinct source-file names, and a derived readiness of present, partial, stripped or unknown. unknown means unreadable sections with no present evidence; it is never reported as absent.
Per-class detail (class_descriptor = Lpkg/Name;): each method's name and JVM-form descriptor, plus its locals (name, type, signature, register, scope, out_of_range). Locals include the implicit this and the parameters with their parameter names, resolved exactly as dexdump resolves them.

How the code is laid out:

Domain (src/domain/dex/, dependency-free, Result plus tagged errors): DEX framing, the debug_info_item state machine, a string/type resolver, and summary/detail assembly.
Shared helpers: src/domain/modifiedUtf8.ts (MUTF-8, used by DEX string_data_item and JVM class files alike), src/artifacts/archiveMember.ts and peekArtifactMagic. Each has one owner, ready for the JVM reader to adopt.

Bounds (each limit and its reason is in src/dex/DexFileRelease.ts):

One call-wide decode budget of 64 Mi debug opcodes covers every method and member. Identical debug streams are decoded once.
The summary only counts, with no per-position or per-local allocation.
The class detail projects its serialized size from memoised per-string bounds while it decodes, and refuses before building past the 8 MiB output budget. The exact check is kept as a backstop.
Members are read, analysed and released one at a time.
Class-data parsing is charged to that budget too. Memo maps and the distinct-source count have fixed ceilings, and a local's register must lie inside the method's frame. Decoding is cooperative. It yields in bounded slices, including during class_data validation, so cancellation lands within one slice: the worst crafted member stalls the event loop for about 47 ms, and cancellation settles in under 1 ms. A class_data entry's cumulative index must stay inside its id table, which is ART's CheckIndex.
Every text path is byte-bounded:
- named members and spellings: at most 128 names or 4 KiB, with the rest counted;
- display names over 160 UTF-16 units: shortened by start and end with a marker (reported entry values are never shortened);
- archive-reader notes: at most 1,024 characters each and 8 KiB in total;
- class_descriptor echoes: shortened the same way.
The 8 MiB output bound counts the result and its repeated limitations.

Member selection with entry:

entry matches the stored name, with a leading ./ optional. The exactly-stored spelling wins; otherwise ambiguity is refused and the stored spellings are listed.
Members REA cannot read are refused as unsupported_target "unavailable, not absent": ZIP directories, encrypted members, and members stored with a symbolic-link mode, which ART loads but REA does not read.
Skipped-member notes mark names that entry would refuse, and advise entry only while a selectable member remains.
Reader notes, such as a Unicode Path name differing from the stored name, or directory attributes, are carried into the result.

Not included: odex/vdex/oat/cdex (refused), nested non-root DEX auto-selection (pass entry), mapping.txt deobfuscation, and runtime/JDWP debugging.

Contract and boundary impact

Semantic owner and earliest changed stage: new src/domain/dex/ modules, plus the shared helpers above (one owner each).
Contribution scope fit: static, read-only inspection of a caller-selected local file. No Android code executes, no device or adb is involved, nothing is mutated, so no scope decision is required. This draft invites one anyway.
CLI and MCP/tool-catalog contract: adds one tool and command with Evidence parity, which the real lane asserts. The catalog is regenerated.
Evidence contract: confidence is observed, the subject is path, format and SHA-256, and the service re-validates identity. Archive Evidence names the selected member.
Generated metadata: produced by the normal scripts; skill sources are edited under .agents/skills/.

Expected effects:

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Real toolchain lane (npm run verify:dex): passes 14 cases. Using the Android SDK's d8 and dexdump plus a JDK, it:

compiles an owned fixture with -g and -g:none and dexes each
cross-checks the decoded facts against dexdump, including the this and parameter local tuples
covers APK summarize-all and entry selection
runs CLI and stdio MCP in Evidence parity, with the advertised schema validated

Independent review: it caught a blocking correctness bug. Implicit parameter locals weren't seeded, which dropped every parameter and this from debug builds. After the fix, a real debug APK's locals went from 57,505 to 155,804, and its line positions from 198,768 to 204,456, exactly dexdump's count.

Adversarial audits: independent passes ran after every fix round, and every finding is fixed in this PR. Later rounds fixed:

un-paced class_data validation;
an over-cap refusal blamed on Android;
member selection by Unicode Path name instead of the stored name (ART uses the raw name);
dropped reader notes;
a count bound that refused ART-valid class_data;
a public CLI/MCP regression when selecting a ./-stored member;
symlink-mode members silently dropped;
unbounded text (a 7.7 MB refusal, now 4.2 KB);
dead-end entry advice;
a short non-DEX file (a 111-byte text decoy in a blind-triage exercise) refused as a "truncated DEX": the magic is now identified on whatever bytes exist before any length check, compact dex is named only from the full 4-byte cdex, and the truncation message states the bytes needed and present.

The final audit at head 0f6c7194f passed 1,758 public CLI + stdio MCP cases with no adapter failure, schema violation or CLI/MCP divergence, and no wrong-member answer: every reported entry's SHA-256 equals the raw member bytes. The checks compare REA with an ART-style selection model; its 32 non-agreements are all expected. 24 are documented refusals of symlink-mode members. 8 are an APK with an invalid UTF-8 flagged name, which ART also refuses and the model doesn't handle. The numbers below are measured on each auditor's own inputs, under a 256 MB heap where noted.

Attack	Before	After
Class detail, 76 KB (2,000 locals sharing a 64 KiB name)	1,249 MB peak; fatal OOM under a 256 MB heap	refused in 9 ms at 66 MB, including under 256 MB
1,000 locals × 256 KiB name (quadratic resolver)	18.6 s	9 ms
Summary of a 2 MB file (2 billion positions via shared streams)	55 s, 878 MB	27 ms, 70 MB, with identical counts
16 MB DEX declaring 8M proto parameters	1.8 GB (2.35 GB in detail mode); fatal OOM under 256 MB	36 ms at 97 MB under 256 MB (proto refused as malformed, over Dalvik's 255 argument registers)
40 MB DEX with `parameters_size` = 40M	1.2 GB; fatal OOM under 256 MB	62 ms at 143 MB
500 classes sharing an inflated class_data (field count over filler)	85 s, un-cancellable	3 ms (impossible count refused up front; parsing charged)
1.5M methods with distinct debug streams (49.5 MB)	fatal OOM under 256 MB (memo `Map`)	completes in 1.5 s; memo capped
2M locals in distinct registers (11 MB)	332 MB	81 MB (registers bounded by `registers_size`)
4M methods in one class (15 MiB)	fatal OOM under 256 MB	1.25 s at 72 MB (classes and methods are streamed, never materialised)
2M methods sharing a 255-parameter proto	13.9 s	0.64 s (O(1) memo key)
2.09M class_defs (64 MiB)	353 MB	115 MB

Checked against dexdump on a real R8 release APK. This surfaced two semantic bugs, both fixed here:

Positions past a method's code were counted. R8 shares debug streams across methods, which inflated line positions about 3×: 4,497,768 against dexdump's 1,468,433. Only positions with address < insns_size are counted now, giving 1,473,289. The remaining 0.2–0.7% on R8 members is dexdump stopping at the start of a method's last instruction; this is documented. Debug builds match exactly.
Zero-length scopes, which kotlinc leaves when it re-declares a generic parameter at address 0, were counted as locals. That put local_declarations 12% above dexdump on a real Kotlin debug APK. They're skipped now, and locals match dexdump exactly on all 15 real DEX members checked.
A single work budget shared across members falsely refused a legitimate 16 × 10 MB multidex app. Each member now gets a budget proportional to its own bytes; real members use at most 22% of it.
readiness counted the implicit this and parameter locals, so that release APK read as present. It now reads stripped. local_declarations keeps exact dexdump parity (115,191 = 115,191), and the new explicit_local_declarations drives readiness.

The real lane now includes a d8 --release build and asserts exact equality with dexdump on line positions and locals for both debug and release builds.

Review round (head 15e098095): all 7 automated review findings were fixed, each with a test that fails on c936fdca5, followed by two independent audit rounds:

an unreadable proto parameter list makes the method malformed (readiness unknown), never present;
every string reference in a debug stream (parameter names, local name/type/signature, DBG_SET_FILE) must name a readable string, as ART's verifier requires; validity is memoised per string_data offset;
the fixed 2,000,000-opcode guard is gone; each opcode is charged to the work budget, and the walk is resumable, pausing mid-stream when the pacer's slice is due (a 30M-opcode stream's longest stall fell from about 0.35 s to 26 ms; an abort at 100 ms lands within 1 ms);
every class_def visit costs a work unit, so a long table of empty classes yields;
a later member's refusal, cancellation or failed read keeps the completed members' summaries as a dex-debug-metadata-members partial observation, retained on the primary failure so a confirmed cleanup keeps it;
entry advice counts only selectable members;
acceptance tests assert facts and owning formatters instead of whole sentences.

The audits also aligned DBG_END_LOCAL/DBG_RESTART_LOCAL with ART's DecodeDebugLocalInfo and drop locals still live when a stream breaks partway, as dexdump does. 880 methods from 14 compiler-built DEX files match dexdump with 0 malformed; pausing and one uninterrupted run agree on all 218,074 methods checked.

Second review round (head a51111649): 2 more automated findings were fixed. The first, a pacing change, has a test that fails on 15e098095. Three independent audit rounds followed; the last was clean.

Pause on work done, not on an opcode count: long string checks now pause. The string resolver scans and validates modified UTF-8 in 16 KiB steps and asks the pacer after each step. Paused scans continue on the next call.
Each string is scanned once: validity and the JSON bound come from one memoised scan, charged once. An invalid string is never decoded.
Method names and descriptors in class detail are paced the same way.
No decode past the budget: a decode the budget cannot cover is refused before it runs. The detail re-checks the budget and cancellation after its last decodes.
Measured:
- Greptile's probe (256 DBG_SET_FILE names of 64 KiB) went from 33.5M units before the first yield, against a 262,144-unit slice, to a longest stall of about 1 ms.
- Detail stalls on 60 MB crafted members fell from about 238 ms to 3 ms, and an abort lands within 1 ms.
- Pausing changes neither results nor charge: 0 mismatches over 48,790 equivalence runs, 273,565 walked methods and 1,200 mutants.
- Against 15e098095, provider output is identical on 43,116 summary and class-detail results.
- Work charged is equal to or lower than 15e098095 everywhere.
Remaining: a valid string that the budget covers is still decoded unpaced when it goes into a class-detail report. The 8 MiB output limit bounds this to about 350 ms; an abort during it is observed when the decode finishes.
Test change: the CLI acceptance test now imports INSPECT_DEX_DEBUG_METADATA_CONTRACT instead of the full catalog.

Shared ZIP reader change: a CI fix in this round makes ZipArtifactReader add a limitation note to every member whose valid Info-ZIP Unicode Path name differs from its stored name. Every inventoryArtifact consumer sees this note; member selection still uses the stored name.

Remaining proof gaps:

In CI without the Android SDK the real lane prints skipped.
Line positions under R8 may be renumbered synthetic lines that need a mapping.txt; that is reported, not asserted.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

At head a51111649 (the second review round; every gate run fresh with TURBO_FORCE=true):

npm run verify:dex -- --require-tools (build-tools 36.1, JDK 25): passed, 14 cases against dexdump.
npm run test:local on the DEX tests plus modifiedUtf8: 152 passed in 11 files.
npm run test:focused on the DEX acceptance and lifecycle tests: 27 passed.
npm run check:fast, docs:check, format:check, mcp-catalog:check, verify:module-boundaries: passed.

At head 15e098095 (the review round and two audit rounds on top of c936fdca5):

npm run verify:dex -- --require-tools (build-tools 36.1, JDK 25): passed, 14 cases against dexdump.
npm run test:local on src/dex, src/domain/dex and src/application/dex: 123 passed.
npm run test:focused on the DEX acceptance and lifecycle tests: 27 passed.
npm run check:fast and docs:check: passed.

At head c936fdca5 (merge of main at bc143125e):

Unit tests: DEX and shared 113, artifacts 219, all passed.
npm run test:focused: 249 tests passed across 10 files (DEX acceptance and lifecycle, ZIP acceptance, read-failure precedence, stable-artifact consumers).
npm run check:fast, docs:check, format:check, mcp-catalog:check, verify:module-boundaries: passed.
DexFileProvider now reads through main's admitted StableRegularFileDescriptor.
Merge note: in the merge commit, the repository's pre-commit hook (oxlint --fix) merged duplicate imports in 14 upstream files this PR does not otherwise touch (+38/−32). Behaviour is unchanged.

Earlier heads:

npm run check:fast: passed.
npm run test:local with src/dex, src/domain/dex, src/application/dex and src/artifacts tests: 168 passed at 94b9c6447. Head 149f342ad only adds a regression test pinning class_data at end of file as unreadable (dexDebugMetadata.test.ts: 27 passed).
npm run test:focused -- tests/acceptance/dexDebugMetadataCli.test.ts tests/process-global/dexProviderLifecycle.test.ts: 26 passed. The magic-order audit drove 384 malformed-input cases through both the CLI and stdio MCP with 0 mismatches. The acceptance tests drive the public CLI and stdio MCP.
30 ZIP-consumer test files (inventory, extraction, Flutter, jadx, SAZ, ASAR, Harmony and others): 251 passed at the previous head, after the shared-reader changes.
npm run format:check, npm run verify:module-boundaries, npx knip: passed.
git merge-tree against current main: merges cleanly.
npm run verify:dex at head 94b9c6447: passed, 14 cases against dexdump, including exact equality for debug and d8 --release builds.
npm run docs:check and npm run mcp-catalog:check: passed.
Real APKs through the CLI: a debug build reports present; a 40 MB, 4-member R8 release reports stripped with lines kept, in 1.5 s at 430 MB peak.

Representative output (that R8 release APK; first member, sample trimmed):

{
  "container": "apk",
  "members": 4,
  "members[0]": {
    "entry": "classes.dex",
    "bytes": 9983472,
    "summary": {
      "dex_version": "039",
      "class_count": 10255,
      "methods_with_code": 56414,
      "methods_with_debug_info": 55172,
      "methods_with_parameter_names": 0,
      "line_positions": 1473289,
      "local_declarations": 115191,
      "explicit_local_declarations": 0,
      "set_file_events": 0,
      "source_file_present": 10177,
      "distinct_source_files": { "count": 2895, "sample": ["CoroutineDebugging.kt", "RowColumnImpl.kt", "Arrangement.kt"] },
      "readiness": "stripped"
    }
  }
}

Compatibility, safety, and release

Breaking changes: none for the new tool. The shared ZIP reader now treats a member marked as a directory only by attribute bits (no trailing /) as a readable file with a limitation. Two such shapes now fail closed with explicit errors: foo next to foo/bar is a prefix conflict, and . or an empty name is not a normalized path.
Known differences from ART (documented):
- REA does not run ART's verifier, so cross-list index repeats, access-flag and list placement, and code-presence rules are not checked.
- ART refuses NUL or invalid UTF-8 in any name. REA refuses invalid UTF-8 only in UTF-8-flagged names, decodes unflagged names as CP437, and tolerates NUL.
- ART refuses duplicate names outright; REA reports them as ambiguous.
Real provider coverage: Android SDK d8/dexdump as the oracle on Linux.
Package impact: no new dependencies.
Security/containment: read-only, no execution. Hostile-input bounds are listed above and measured.

Author readiness

Human owner: @FerroLx

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

https://claude.ai/code/session_0117t1LTnmt5nHnp1WyyJVFC

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4473 **Author:** [@FerroLx](https://github.com/FerroLx) **Created:** 10/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/dex-debug-metadata-standalone` --- ### 📝 Commits (10+) - [`0cd3596`](https://github.com/morluto/rea/commit/0cd3596fd5ecfe479586a8a6f369c3c6fe4f0834) feat(dex): inspect Android DEX source-line and local-variable debug metadata - [`be8baa3`](https://github.com/morluto/rea/commit/be8baa395c0a32153c9dbf2f14db45991baf39f8) fix(dex): bound class_data parsing and summary working sets - [`b2d9022`](https://github.com/morluto/rea/commit/b2d9022d6847735941b03d0ba5d9768f996a4cdc) fix(dex): stream class_defs and methods instead of materialising them - [`c025f8e`](https://github.com/morluto/rea/commit/c025f8e662e46daff8a05e4f03d52d2f03c48a8d) fix(dex): skip zero-length local scopes; size the work budget per member - [`f72d238`](https://github.com/morluto/rea/commit/f72d23809dd1d675ca12a8a1d287eddcc34405fb) docs(dex): state the budget, zero-length and cancellation guarantees precisely - [`14bd4fc`](https://github.com/morluto/rea/commit/14bd4fc145fb8fc9c4b8aacc1427b80ce9a9d731) fix(dex): address the Greptile review on #4473 - [`44b214a`](https://github.com/morluto/rea/commit/44b214a29465ffa0f60341c76ff7422a9630b3f1) fix(dex): address the Codex review on #4473 and bound the class lookup - [`1ff7225`](https://github.com/morluto/rea/commit/1ff72258acf2a04be9a582bfd428cb0bb689834c) fix(dex): stop string scans once the budget is spent; retain the magic-peek handle - [`8b1c284`](https://github.com/morluto/rea/commit/8b1c28416b0fbc80be173a7673b0c72fca5f511d) fix(dex): fail closed on unverifiable class, code-item and debug facts; load multidex like ART - [`66db884`](https://github.com/morluto/rea/commit/66db884495d4ec9ccd2bd75518358019cb10ee9b) fix(dex): keep archive member selection array-bound and describe the multidex sequence ### 📊 Changes **82 files changed** (+11457 additions, -110 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+6 -0) 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+15 -0) 📝 `README.md` (+39 -38) 📝 `docs/architecture.mermaid` (+3 -0) ➕ `docs/dex-debug-metadata.md` (+215 -0) 📝 `docs/index.md` (+14 -14) ➕ `docs/specs/dex-debug-metadata/plan.md` (+26 -0) ➕ `docs/specs/dex-debug-metadata/spec.md` (+119 -0) ➕ `docs/specs/dex-debug-metadata/tasks.md` (+22 -0) ➕ `docs/specs/dex-debug-metadata/verification.md` (+82 -0) 📝 `experiments/analysis-efficiency/source-to-bundle.mjs` (+1 -2) 📝 `package.json` (+1 -0) 📝 `scripts/lib/catalog-builders.mjs` (+5 -0) 📝 `scripts/lib/catalog-core.mjs` (+2 -0) 📝 `scripts/lib/test-projects.mjs` (+1 -1) ➕ `scripts/verify-dex-debug-metadata.mjs` (+435 -0) 📝 `src/application/WebSourceLocationPorts.ts` (+4 -2) ➕ `src/application/dex/DexDebugMetadataPort.ts` (+25 -0) ➕ `src/application/dex/DexDebugMetadataService.test.ts` (+53 -0) ➕ `src/application/dex/DexDebugMetadataService.ts` (+101 -0) _...and 62 more files_ </details> ### 📄 Description > **Independent of #3586** (JVM debug metadata). This branch is cut from `main` and carries only the DEX tool, so either PR can merge first. The few helpers both readers need are shared modules introduced here: the modified-UTF8 decoder, archive-member selection and reading, and the magic peek. Whichever PR merges second will merge `main` and adopt them in place of its own copies. Both PRs also carry the same shared ZIP-reader changes: `storedPath` is the raw central-directory name, a member is a directory only when its stored name ends in `/`, and the abort reason is kept as the cancellation cause. In `ZipArtifactReader.ts` they differ by one line, `normalizeArchiveMemberPath` here against the equivalent `./` strip in #3586; the second to merge keeps `normalizeArchiveMemberPath`. ## Summary Closes #4538. Adds `inspect_dex_debug_metadata` (CLI `rea inspect-dex-debug-metadata`), the Android DEX sibling of `inspect_jvm_debug_metadata` (#3586). It statically reports the compiler-emitted debug metadata in a selected local `.dex`, or the `classes*.dex` members of an APK/ZIP. That lets an analyst judge **debugging readiness before attaching a debugger or reading a decompilation**: do source files, line positions, parameter names and local-variable names, types and scopes survive, or were they stripped by R8/ProGuard? ## Problem and expected behavior When triaging an APK, the first practical question is whether there is debug metadata to work with: a debug build, a release build with line numbers kept, or a fully stripped R8 release. Today that takes `dexdump` from the Android SDK and manual counting, and an agent can't get it as Evidence. ## Change and scope The tool works at two granularities. A DEX holds thousands of classes, so neither alone is enough. - **Summary (default):** one entry per DEX member. For an APK, these are the members ART's multidex loader loads: `classes.dex`, `classes2.dex`, … while each is present under its exact stored central-directory name. The raw name is used, never an Info-ZIP Unicode Path name. At most 64 members are read; past that, REA refuses and asks for `entry`. It gives method tallies, line positions, local declarations (with `explicit_local_declarations` separating locals the stream declares from the implicit `this`/parameter locals), distinct source-file names, and a derived `readiness` of `present`, `partial`, `stripped` or `unknown`. `unknown` means unreadable sections with no present evidence; it is never reported as absent. - **Per-class detail (`class_descriptor` = `Lpkg/Name;`):** each method's `name` and JVM-form `descriptor`, plus its `locals` (name, type, signature, register, scope, `out_of_range`). Locals include the implicit `this` and the parameters with their parameter names, resolved exactly as `dexdump` resolves them. How the code is laid out: - **Domain** (`src/domain/dex/`, dependency-free, `Result` plus tagged errors): DEX framing, the `debug_info_item` state machine, a string/type resolver, and summary/detail assembly. - **Shared helpers:** `src/domain/modifiedUtf8.ts` (MUTF-8, used by DEX `string_data_item` and JVM class files alike), `src/artifacts/archiveMember.ts` and `peekArtifactMagic`. Each has one owner, ready for the JVM reader to adopt. **Bounds** (each limit and its reason is in `src/dex/DexFileRelease.ts`): - One call-wide decode budget of 64 Mi debug opcodes covers every method and member. Identical debug streams are decoded once. - The summary only counts, with no per-position or per-local allocation. - The class detail projects its serialized size from memoised per-string bounds while it decodes, and refuses before building past the 8 MiB output budget. The exact check is kept as a backstop. - Members are read, analysed and released one at a time. - Class-data parsing is charged to that budget too. Memo maps and the distinct-source count have fixed ceilings, and a local's register must lie inside the method's frame. Decoding is cooperative. It yields in bounded slices, including during class_data validation, so cancellation lands within one slice: the worst crafted member stalls the event loop for about 47 ms, and cancellation settles in under 1 ms. A class_data entry's cumulative index must stay inside its id table, which is ART's `CheckIndex`. - Every text path is byte-bounded: - named members and spellings: at most 128 names or 4 KiB, with the rest counted; - display names over 160 UTF-16 units: shortened by start and end with a marker (reported `entry` values are never shortened); - archive-reader notes: at most 1,024 characters each and 8 KiB in total; - `class_descriptor` echoes: shortened the same way. The 8 MiB output bound counts the result and its repeated `limitations`. **Member selection with `entry`:** - `entry` matches the stored name, with a leading `./` optional. The exactly-stored spelling wins; otherwise ambiguity is refused and the stored spellings are listed. - Members REA cannot read are refused as `unsupported_target` "unavailable, not absent": ZIP directories, encrypted members, and members stored with a symbolic-link mode, which ART loads but REA does not read. - Skipped-member notes mark names that `entry` would refuse, and advise `entry` only while a selectable member remains. - Reader notes, such as a Unicode Path name differing from the stored name, or directory attributes, are carried into the result. **Not included:** `odex`/`vdex`/`oat`/`cdex` (refused), nested non-root DEX auto-selection (pass `entry`), `mapping.txt` deobfuscation, and runtime/JDWP debugging. ## Contract and boundary impact - **Semantic owner and earliest changed stage:** new `src/domain/dex/` modules, plus the shared helpers above (one owner each). - **Contribution scope fit:** static, read-only inspection of a caller-selected local file. No Android code executes, no device or `adb` is involved, nothing is mutated, so no scope decision is required. This draft invites one anyway. - **CLI and MCP/tool-catalog contract:** adds one tool and command with Evidence parity, which the real lane asserts. The catalog is regenerated. - **Evidence contract:** confidence is `observed`, the subject is path, format and SHA-256, and the service re-validates identity. Archive Evidence names the selected member. - **Generated metadata:** produced by the normal scripts; skill sources are edited under `.agents/skills/`. Expected effects: - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage **Real toolchain lane** (`npm run verify:dex`): passes 14 cases. Using the Android SDK's `d8` and `dexdump` plus a JDK, it: - compiles an owned fixture with `-g` and `-g:none` and dexes each - cross-checks the decoded facts against `dexdump`, including the `this` and parameter local tuples - covers APK summarize-all and entry selection - runs CLI and stdio MCP in Evidence parity, with the advertised schema validated **Independent review:** it caught a blocking correctness bug. Implicit parameter locals weren't seeded, which dropped every parameter and `this` from debug builds. After the fix, a real debug APK's locals went from 57,505 to 155,804, and its line positions from 198,768 to **204,456, exactly `dexdump`'s count**. **Adversarial audits:** independent passes ran after every fix round, and every finding is fixed in this PR. Later rounds fixed: - un-paced class_data validation; - an over-cap refusal blamed on Android; - member selection by Unicode Path name instead of the stored name (ART uses the raw name); - dropped reader notes; - a count bound that refused ART-valid class_data; - a public CLI/MCP regression when selecting a `./`-stored member; - symlink-mode members silently dropped; - unbounded text (a 7.7 MB refusal, now 4.2 KB); - dead-end `entry` advice; - a short non-DEX file (a 111-byte text decoy in a blind-triage exercise) refused as a "truncated DEX": the magic is now identified on whatever bytes exist before any length check, compact dex is named only from the full 4-byte `cdex`, and the truncation message states the bytes needed and present. The final audit at head `0f6c7194f` passed 1,758 public CLI + stdio MCP cases with no adapter failure, schema violation or CLI/MCP divergence, and no wrong-member answer: every reported entry's SHA-256 equals the raw member bytes. The checks compare REA with an ART-style selection model; its 32 non-agreements are all expected. 24 are documented refusals of symlink-mode members. 8 are an APK with an invalid UTF-8 flagged name, which ART also refuses and the model doesn't handle. The numbers below are measured on each auditor's own inputs, under a 256 MB heap where noted. | Attack | Before | After | |---|---|---| | Class detail, 76 KB (2,000 locals sharing a 64 KiB name) | 1,249 MB peak; fatal OOM under a 256 MB heap | refused in 9 ms at 66 MB, including under 256 MB | | 1,000 locals × 256 KiB name (quadratic resolver) | 18.6 s | 9 ms | | Summary of a 2 MB file (2 billion positions via shared streams) | 55 s, 878 MB | 27 ms, 70 MB, with identical counts | | 16 MB DEX declaring 8M proto parameters | 1.8 GB (2.35 GB in detail mode); fatal OOM under 256 MB | 36 ms at 97 MB under 256 MB (proto refused as malformed, over Dalvik's 255 argument registers) | | 40 MB DEX with `parameters_size` = 40M | 1.2 GB; fatal OOM under 256 MB | 62 ms at 143 MB | | 500 classes sharing an inflated class_data (field count over filler) | 85 s, un-cancellable | 3 ms (impossible count refused up front; parsing charged) | | 1.5M methods with distinct debug streams (49.5 MB) | fatal OOM under 256 MB (memo `Map`) | completes in 1.5 s; memo capped | | 2M locals in distinct registers (11 MB) | 332 MB | 81 MB (registers bounded by `registers_size`) | | 4M methods in one class (15 MiB) | fatal OOM under 256 MB | 1.25 s at 72 MB (classes and methods are streamed, never materialised) | | 2M methods sharing a 255-parameter proto | 13.9 s | 0.64 s (O(1) memo key) | | 2.09M class_defs (64 MiB) | 353 MB | 115 MB | **Checked against `dexdump` on a real R8 release APK.** This surfaced two semantic bugs, both fixed here: - Positions past a method's code were counted. R8 shares debug streams across methods, which inflated line positions about 3×: 4,497,768 against `dexdump`'s 1,468,433. Only positions with `address < insns_size` are counted now, giving 1,473,289. The remaining 0.2–0.7% on R8 members is `dexdump` stopping at the start of a method's last instruction; this is documented. Debug builds match exactly. - Zero-length scopes, which kotlinc leaves when it re-declares a generic parameter at address 0, were counted as locals. That put `local_declarations` 12% above `dexdump` on a real Kotlin debug APK. They're skipped now, and locals match `dexdump` **exactly on all 15 real DEX members** checked. - A single work budget shared across members falsely refused a legitimate 16 × 10 MB multidex app. Each member now gets a budget proportional to its own bytes; real members use at most 22% of it. - `readiness` counted the implicit `this` and parameter locals, so that release APK read as `present`. It now reads `stripped`. `local_declarations` keeps exact `dexdump` parity (115,191 = 115,191), and the new `explicit_local_declarations` drives readiness. The real lane now includes a `d8 --release` build and asserts **exact** equality with `dexdump` on line positions and locals for both debug and release builds. **Review round (head `15e098095`):** all 7 automated review findings were fixed, each with a test that fails on `c936fdca5`, followed by two independent audit rounds: - an unreadable proto parameter list makes the method `malformed` (readiness `unknown`), never `present`; - every string reference in a debug stream (parameter names, local name/type/signature, `DBG_SET_FILE`) must name a readable string, as ART's verifier requires; validity is memoised per `string_data` offset; - the fixed 2,000,000-opcode guard is gone; each opcode is charged to the work budget, and the walk is resumable, pausing mid-stream when the pacer's slice is due (a 30M-opcode stream's longest stall fell from about 0.35 s to 26 ms; an abort at 100 ms lands within 1 ms); - every class_def visit costs a work unit, so a long table of empty classes yields; - a later member's refusal, cancellation or failed read keeps the completed members' summaries as a `dex-debug-metadata-members` partial observation, retained on the primary failure so a confirmed cleanup keeps it; - `entry` advice counts only selectable members; - acceptance tests assert facts and owning formatters instead of whole sentences. The audits also aligned `DBG_END_LOCAL`/`DBG_RESTART_LOCAL` with ART's `DecodeDebugLocalInfo` and drop locals still live when a stream breaks partway, as `dexdump` does. 880 methods from 14 compiler-built DEX files match `dexdump` with 0 malformed; pausing and one uninterrupted run agree on all 218,074 methods checked. **Second review round (head `a51111649`):** 2 more automated findings were fixed. The first, a pacing change, has a test that fails on `15e098095`. Three independent audit rounds followed; the last was clean. - **Pause on work done, not on an opcode count:** long string checks now pause. The string resolver scans and validates modified UTF-8 in 16 KiB steps and asks the pacer after each step. Paused scans continue on the next call. - **Each string is scanned once:** validity and the JSON bound come from one memoised scan, charged once. An invalid string is never decoded. - **Method names and descriptors in class detail** are paced the same way. - **No decode past the budget:** a decode the budget cannot cover is refused before it runs. The detail re-checks the budget and cancellation after its last decodes. - **Measured:** - Greptile's probe (256 `DBG_SET_FILE` names of 64 KiB) went from 33.5M units before the first yield, against a 262,144-unit slice, to a longest stall of about 1 ms. - Detail stalls on 60 MB crafted members fell from about 238 ms to 3 ms, and an abort lands within 1 ms. - Pausing changes neither results nor charge: 0 mismatches over 48,790 equivalence runs, 273,565 walked methods and 1,200 mutants. - Against `15e098095`, provider output is identical on 43,116 summary and class-detail results. - Work charged is equal to or lower than `15e098095` everywhere. - **Remaining:** a valid string that the budget covers is still decoded unpaced when it goes into a class-detail report. The 8 MiB output limit bounds this to about 350 ms; an abort during it is observed when the decode finishes. - **Test change:** the CLI acceptance test now imports `INSPECT_DEX_DEBUG_METADATA_CONTRACT` instead of the full catalog. **Shared ZIP reader change:** a CI fix in this round makes `ZipArtifactReader` add a limitation note to every member whose valid Info-ZIP Unicode Path name differs from its stored name. Every `inventoryArtifact` consumer sees this note; member selection still uses the stored name. **Remaining proof gaps:** - In CI without the Android SDK the real lane prints `skipped`. - Line positions under R8 may be renumbered synthetic lines that need a `mapping.txt`; that is reported, not asserted. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed At head `a51111649` (the second review round; every gate run fresh with `TURBO_FORCE=true`): - `npm run verify:dex -- --require-tools` (build-tools 36.1, JDK 25): passed, 14 cases against `dexdump`. - `npm run test:local` on the DEX tests plus `modifiedUtf8`: 152 passed in 11 files. - `npm run test:focused` on the DEX acceptance and lifecycle tests: 27 passed. - `npm run check:fast`, `docs:check`, `format:check`, `mcp-catalog:check`, `verify:module-boundaries`: passed. At head `15e098095` (the review round and two audit rounds on top of `c936fdca5`): - `npm run verify:dex -- --require-tools` (build-tools 36.1, JDK 25): passed, 14 cases against `dexdump`. - `npm run test:local` on `src/dex`, `src/domain/dex` and `src/application/dex`: 123 passed. - `npm run test:focused` on the DEX acceptance and lifecycle tests: 27 passed. - `npm run check:fast` and `docs:check`: passed. At head `c936fdca5` (merge of `main` at `bc143125e`): - Unit tests: DEX and shared 113, artifacts 219, all passed. - `npm run test:focused`: 249 tests passed across 10 files (DEX acceptance and lifecycle, ZIP acceptance, read-failure precedence, stable-artifact consumers). - `npm run check:fast`, `docs:check`, `format:check`, `mcp-catalog:check`, `verify:module-boundaries`: passed. - `DexFileProvider` now reads through `main`'s admitted `StableRegularFileDescriptor`. - **Merge note:** in the merge commit, the repository's pre-commit hook (`oxlint --fix`) merged duplicate imports in 14 upstream files this PR does not otherwise touch (+38/−32). Behaviour is unchanged. Earlier heads: - `npm run check:fast`: passed. - `npm run test:local` with src/dex, src/domain/dex, src/application/dex and src/artifacts tests: 168 passed at `94b9c6447`. Head `149f342ad` only adds a regression test pinning class_data at end of file as unreadable (`dexDebugMetadata.test.ts`: 27 passed). - `npm run test:focused -- tests/acceptance/dexDebugMetadataCli.test.ts tests/process-global/dexProviderLifecycle.test.ts`: 26 passed. The magic-order audit drove 384 malformed-input cases through both the CLI and stdio MCP with 0 mismatches. The acceptance tests drive the public CLI and stdio MCP. - 30 ZIP-consumer test files (inventory, extraction, Flutter, jadx, SAZ, ASAR, Harmony and others): 251 passed at the previous head, after the shared-reader changes. - `npm run format:check`, `npm run verify:module-boundaries`, `npx knip`: passed. - `git merge-tree` against current `main`: merges cleanly. - `npm run verify:dex` at head `94b9c6447`: passed, 14 cases against `dexdump`, including exact equality for debug and `d8 --release` builds. - `npm run docs:check` and `npm run mcp-catalog:check`: passed. - Real APKs through the CLI: a debug build reports `present`; a 40 MB, 4-member R8 release reports `stripped` with lines kept, in 1.5 s at 430 MB peak. **Representative output** (that R8 release APK; first member, sample trimmed): ```json { "container": "apk", "members": 4, "members[0]": { "entry": "classes.dex", "bytes": 9983472, "summary": { "dex_version": "039", "class_count": 10255, "methods_with_code": 56414, "methods_with_debug_info": 55172, "methods_with_parameter_names": 0, "line_positions": 1473289, "local_declarations": 115191, "explicit_local_declarations": 0, "set_file_events": 0, "source_file_present": 10177, "distinct_source_files": { "count": 2895, "sample": ["CoroutineDebugging.kt", "RowColumnImpl.kt", "Arrangement.kt"] }, "readiness": "stripped" } } } ``` ## Compatibility, safety, and release - **Breaking changes:** none for the new tool. The shared ZIP reader now treats a member marked as a directory only by attribute bits (no trailing `/`) as a readable file with a limitation. Two such shapes now fail closed with explicit errors: `foo` next to `foo/bar` is a prefix conflict, and `.` or an empty name is not a normalized path. - **Known differences from ART (documented):** - REA does not run ART's verifier, so cross-list index repeats, access-flag and list placement, and code-presence rules are not checked. - ART refuses NUL or invalid UTF-8 in any name. REA refuses invalid UTF-8 only in UTF-8-flagged names, decodes unflagged names as CP437, and tolerates NUL. - ART refuses duplicate names outright; REA reports them as ambiguous. - **Real provider coverage:** Android SDK `d8`/`dexdump` as the oracle on Linux. - **Package impact:** no new dependencies. - **Security/containment:** read-only, no execution. Hostile-input bounds are listed above and measured. ## Author readiness  Human owner: @FerroLx - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0117t1LTnmt5nHnp1WyyJVFC --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:13:07 +00:00

bsakaguchi referenced this issue

2026-11-01 21:46:24 +00:00

[PR #6263] [CLOSED] docs(browser): explain destination-dependent case isolation in script export #7237