morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #2049] [MERGED] feat(ghidra): decode Swift String literals in function dossiers #4989

New issue

Closed

opened 2026-10-31 00:57:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:57:44 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2049
Author: @thefolahan
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @morluto

Base: main ← Head: feat/swift-string-literals

📝 Commits (8)

2f89a1b feat(ghidra): decode Swift String literals in function dossiers
702dc9f fix(ghidra): bound Swift literal reads and compare decoded strings
facd589 fix(ghidra): budget Swift literal decoding and tighten use sites
d5b2edc fix(ghidra): charge escaped JSON and follow copied selects
560257b fix(ghidra): bound literal records and trace selects across branches
59be535 fix(ghidra): stop every flow at the literal budget and route omissions
d3ba724 fix(ghidra): preserve Swift literal provenance and partial coverage
f8f18e0 fix(ghidra): keep unresolved Swift tail-call sites unknown

📊 Changes

12 files changed (+1035 additions, -5 deletions)

View changed files

📝 bridge/ghidra/ReaGhidraBridge.java (+380 -0)
📝 scripts/verify-real-ghidra-swift.mjs (+68 -0)
📝 src/domain/analysisView/analysisView.test.ts (+19 -1)
📝 src/domain/analysisView/analysisView.ts (+2 -0)
📝 src/domain/analysisView/nativeFunctionView.ts (+4 -1)
📝 src/domain/functionComparison.test.ts (+230 -0)
📝 src/domain/functionComparisonDimensions.ts (+39 -3)
📝 src/domain/functionComparisonNormalization.ts (+79 -0)
📝 src/domain/hopperValues.ts (+76 -0)
📝 src/ghidra/GhidraFunctionValues.test.ts (+98 -0)
📝 src/ghidra/GhidraFunctionValues.ts (+17 -0)
➕ tests/conformance/swift/string-literals.swift (+23 -0)

📄 Description

Summary

Fixes #2030. On Swift targets analyzed with Ghidra, analyze_function now reports Swift String literals that Ghidra keeps only as register constants. They appear in a new optional swift_string_literals dossier field and a matching inspect_analysis_view facet. referenced_strings is unchanged.

Problem and expected behavior

Swift passes a String as two words. A small literal (up to 15 UTF-8 bytes) keeps its bytes inside the words, so the text never exists as data. A large immortal literal keeps a count and a pointer to its bytes biased by 32, so Ghidra creates no reference to the real address. Before this change, the dossiers for VigilKit.LidMode.Failure.errorDescription and VigilKit.Format.rate in the issue both returned referenced_strings: [] and nothing else about their text.

Change and scope

The Ghidra bridge runs SymbolicPropogator over the function with a ContextEvaluatorAdapter that never creates references, and decodes constant register pairs where Swift consumes a String:

consecutive argument registers among x0 to x7 at calls and at tail calls (a jump toward a target outside the function, or a computed jump with no resolved target)
x0 and x1 at returns (an instruction with a RETURN p-code operation)
both registers of one stp

Decoding at those use points, rather than wherever two registers happen to hold matching values, is what keeps the output clean. Before I restricted it this way, the probe reported partial values from mov/movk sequences ("Lid clos") and pairs mixed from different switch cases ("exclusioles").

Validation per representation:

Small: discriminator 0xE (ASCII) or 0xA (UTF-8), the count in bits 56 to 59, zero bytes past the count, and valid UTF-8. Control characters are kept, because Swift strings may contain them and JSON escapes them. Count zero is the canonical empty string.
Large: _countAndFlags flags 0xD (ASCII) or 0x1 (UTF-8) with zero reserved bits, and an object top byte of 0x80. The bytes and their NUL terminator must lie inside one initialized block that is not executable. Every literal record draws on one heap budget per dossier, measured before any bytes are allocated: a fixed cost per record plus the worst case escaped size of its text. The first record that does not fit is listed in swift_string_literal_omissions with its byte length and reason, and decoding stops there.

A pair is skipped when both registers come from conditional selects. Ghidra forks each csel without correlating the condition, so two selects on the same flags can pair words from different cases. A pair with only one selected register is kept, because every combination is then a real literal (for example "%.0f%%" and "%.1f%%", which share an object word).

Gating: the field is produced only for AArch64 programs that contain __swift5_* or swift5_* blocks. A Swift program on another architecture gets a limitation saying that no literals were decoded. Programs that are not Swift are unchanged.

Not included:

string search over decoded small literals (a separate surface, happy to follow up)
x86_64 register pairs (same layout, not verified)
bridged NSString literals
literals stored one word at a time
literals chosen by two correlated selects

Contract and boundary impact

Semantic owner and earliest changed stage: Ghidra bridge (ReaGhidraBridge.java) produces the field; functionDossierSchema owns the contract.
CLI and MCP/tool-catalog contract: additive. Optional swift_string_literals and swift_string_literal_omissions on analyze_function output, with matching inspect_analysis_view facets. compare_functions includes decoded literals in strings_names, and reports unknown when the referenced strings and names match but literal coverage differs or is partial. An absent record pages as { available: false, reason: "not recorded" }, like unresolved_calls.
Provider, bridge, target-format, or platform compatibility: Ghidra only, AArch64 Swift programs only. Hopper and IDA dossiers are unchanged.
Evidence, artifact, provenance, or reconstruction contract: decoded values are kept apart from observed references and carry their source instruction, registers and raw words. A limitation line states what is not reported.
Process execution, authorization, cleanup, or containment impact: none. The propagation is read only and adds no references to the program.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: none committed; npm run docs:check passes.

Evidence and regression coverage

Tests added or updated:
- GhidraFunctionValues.test.ts: a small literal without storage beside a large literal with its address, and a literal address that is not canonical is rejected.
- analysisView.test.ts: the new facet pages recorded literals and reports an absent record as not recorded.
- verify:ghidra:swift: compiles tests/conformance/swift/string-literals.swift with swiftc -O and asserts, through MCP, the exact value and representation of small, large, small UTF-8, large UTF-8 and returned literals.
Base reproduction or other evidence: on the base revision the dossiers have no swift_string_literals; the issue shows the referenced_strings: [] output. Across all 2561 functions of the issue's Vigil arm64 slice, the decoder reports 263 distinct literals in about 2 seconds. I checked each one against the app's source. All of them are real literals in one of these forms:

written verbatim in the source
interpolations of constants that the compiler folded ("Library/LaunchAgents/\(guardLabel).plist")
imported IOKit keys (Current Capacity)
text inlined from the Swift runtime

None of them is a partial or mixed value.

User-visible CLI/MCP output (if applicable): rea function Vigil-arm64 0x10004da08 --provider ghidra (LidMode.Failure.errorDescription):

  referenced_strings: []
  swift_string_literals[3]:
    - source_address: 0x10004daa0
      value: "The user name "
      representation: small
      address: null
      registers[2]: x0,x1
      words[2]: 0x7265737520656854,0xee0020656d616e20
    - source_address: 0x10004dad4
      value: " cannot be written into a sudoers rule safely."
      representation: large
      address: 0x100070270
      registers[2]: x0,x1
      words[2]: 0xd00000000000002e,0x8000000100070250
    - source_address: 0x10004db08
      value: The power setting could not be changed.
      representation: large
      address: 0x100070240
      registers[2]: x0,x1
      words[2]: 0xd000000000000027,0x8000000100070220

VigilKit.Format.rate (0x10004d260) now lists " B/s", %.1f KB/s and %.1f MB/s. The KB and MB formats share their final movk instructions across two branches, and both are reported at the tail call to String(format:).

Remaining proof gaps: x86_64 Swift and ELF Swift on AArch64 were not run; the field is not produced for x86_64.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run verify:ghidra:swift (macOS 27.0.1 on Apple silicon, Ghidra 12.1.4 with the native decompiler, JDK 21, Swift 6.4): passed. The existing inventory checks are unchanged, and swift_string_literals decoded 1, 1, 2 and 1 literals for the four fixture functions.
npm run test:changed -- --base upstream/main: 69 files, 917 passed, 3 skipped.
npm run check:fast: passed.
npm run docs:check: passed.
rea function on the Vigil arm64 slice for both functions in the issue: output shown above.

Compatibility, safety, and release

Breaking changes or migration steps: none; the field and facet are additive and optional.
Real Hopper/Ghidra, browser, or OS coverage: real Ghidra 12.1.4 on macOS arm64. Hopper and IDA are untouched.
Package or release metadata impact: none.
Security, privacy, process, or containment review: reads program memory only through Ghidra's memory API, inside initialized blocks; no program mutation.

Review checklist

This PR addresses a concrete problem or an agreed enhancement.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2049 **Author:** [@thefolahan](https://github.com/thefolahan) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `feat/swift-string-literals` --- ### 📝 Commits (8) - [`2f89a1b`](https://github.com/morluto/rea/commit/2f89a1b5c0e45748f1a7f4ec6bbc480e83348037) feat(ghidra): decode Swift String literals in function dossiers - [`702dc9f`](https://github.com/morluto/rea/commit/702dc9f9329559a86a52fbf379a8d799f8856783) fix(ghidra): bound Swift literal reads and compare decoded strings - [`facd589`](https://github.com/morluto/rea/commit/facd58973a57a051ef8405accf1be8ea25ba9d85) fix(ghidra): budget Swift literal decoding and tighten use sites - [`d5b2edc`](https://github.com/morluto/rea/commit/d5b2edc217008dc18ee636565cb3b148375b239e) fix(ghidra): charge escaped JSON and follow copied selects - [`560257b`](https://github.com/morluto/rea/commit/560257b00e92188a7fabded623fefd8655bc761a) fix(ghidra): bound literal records and trace selects across branches - [`59be535`](https://github.com/morluto/rea/commit/59be5357df07fcf4a2dbab08ecbd100179a40785) fix(ghidra): stop every flow at the literal budget and route omissions - [`d3ba724`](https://github.com/morluto/rea/commit/d3ba7243d73b76a66d5c44c64caf076d5026c810) fix(ghidra): preserve Swift literal provenance and partial coverage - [`f8f18e0`](https://github.com/morluto/rea/commit/f8f18e0bc745f670e72bef43f5b39c943e8073af) fix(ghidra): keep unresolved Swift tail-call sites unknown ### 📊 Changes **12 files changed** (+1035 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `bridge/ghidra/ReaGhidraBridge.java` (+380 -0) 📝 `scripts/verify-real-ghidra-swift.mjs` (+68 -0) 📝 `src/domain/analysisView/analysisView.test.ts` (+19 -1) 📝 `src/domain/analysisView/analysisView.ts` (+2 -0) 📝 `src/domain/analysisView/nativeFunctionView.ts` (+4 -1) 📝 `src/domain/functionComparison.test.ts` (+230 -0) 📝 `src/domain/functionComparisonDimensions.ts` (+39 -3) 📝 `src/domain/functionComparisonNormalization.ts` (+79 -0) 📝 `src/domain/hopperValues.ts` (+76 -0) 📝 `src/ghidra/GhidraFunctionValues.test.ts` (+98 -0) 📝 `src/ghidra/GhidraFunctionValues.ts` (+17 -0) ➕ `tests/conformance/swift/string-literals.swift` (+23 -0) </details> ### 📄 Description ## Summary Fixes #2030. On Swift targets analyzed with Ghidra, `analyze_function` now reports Swift `String` literals that Ghidra keeps only as register constants. They appear in a new optional `swift_string_literals` dossier field and a matching `inspect_analysis_view` facet. `referenced_strings` is unchanged. ## Problem and expected behavior Swift passes a `String` as two words. A small literal (up to 15 UTF-8 bytes) keeps its bytes inside the words, so the text never exists as data. A large immortal literal keeps a count and a pointer to its bytes biased by 32, so Ghidra creates no reference to the real address. Before this change, the dossiers for `VigilKit.LidMode.Failure.errorDescription` and `VigilKit.Format.rate` in the issue both returned `referenced_strings: []` and nothing else about their text. ## Change and scope The Ghidra bridge runs `SymbolicPropogator` over the function with a `ContextEvaluatorAdapter` that never creates references, and decodes constant register pairs where Swift consumes a `String`: * consecutive argument registers among x0 to x7 at calls and at tail calls (a jump toward a target outside the function, or a computed jump with no resolved target) * x0 and x1 at returns (an instruction with a RETURN p-code operation) * both registers of one `stp` Decoding at those use points, rather than wherever two registers happen to hold matching values, is what keeps the output clean. Before I restricted it this way, the probe reported partial values from `mov`/`movk` sequences (`"Lid clos"`) and pairs mixed from different switch cases (`"exclusioles"`). Validation per representation: * **Small:** discriminator `0xE` (ASCII) or `0xA` (UTF-8), the count in bits 56 to 59, zero bytes past the count, and valid UTF-8. Control characters are kept, because Swift strings may contain them and JSON escapes them. Count zero is the canonical empty string. * **Large:** `_countAndFlags` flags `0xD` (ASCII) or `0x1` (UTF-8) with zero reserved bits, and an object top byte of `0x80`. The bytes and their NUL terminator must lie inside one initialized block that is not executable. Every literal record draws on one heap budget per dossier, measured before any bytes are allocated: a fixed cost per record plus the worst case escaped size of its text. The first record that does not fit is listed in `swift_string_literal_omissions` with its byte length and reason, and decoding stops there. A pair is skipped when both registers come from conditional selects. Ghidra forks each `csel` without correlating the condition, so two selects on the same flags can pair words from different cases. A pair with only one selected register is kept, because every combination is then a real literal (for example `"%.0f%%"` and `"%.1f%%"`, which share an object word). Gating: the field is produced only for AArch64 programs that contain `__swift5_*` or `swift5_*` blocks. A Swift program on another architecture gets a limitation saying that no literals were decoded. Programs that are not Swift are unchanged. Not included: * string `search` over decoded small literals (a separate surface, happy to follow up) * x86_64 register pairs (same layout, not verified) * bridged `NSString` literals * literals stored one word at a time * literals chosen by two correlated selects ## Contract and boundary impact - Semantic owner and earliest changed stage: Ghidra bridge (`ReaGhidraBridge.java`) produces the field; `functionDossierSchema` owns the contract. - CLI and MCP/tool-catalog contract: additive. Optional `swift_string_literals` and `swift_string_literal_omissions` on `analyze_function` output, with matching `inspect_analysis_view` facets. `compare_functions` includes decoded literals in `strings_names`, and reports unknown when the referenced strings and names match but literal coverage differs or is partial. An absent record pages as `{ available: false, reason: "not recorded" }`, like `unresolved_calls`. - Provider, bridge, target-format, or platform compatibility: Ghidra only, AArch64 Swift programs only. Hopper and IDA dossiers are unchanged. - Evidence, artifact, provenance, or reconstruction contract: decoded values are kept apart from observed references and carry their source instruction, registers and raw words. A limitation line states what is not reported. - Process execution, authorization, cleanup, or containment impact: none. The propagation is read only and adds no references to the program. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none committed; `npm run docs:check` passes. ## Evidence and regression coverage - Tests added or updated: * `GhidraFunctionValues.test.ts`: a small literal without storage beside a large literal with its address, and a literal address that is not canonical is rejected. * `analysisView.test.ts`: the new facet pages recorded literals and reports an absent record as not recorded. * `verify:ghidra:swift`: compiles `tests/conformance/swift/string-literals.swift` with `swiftc -O` and asserts, through MCP, the exact value and representation of small, large, small UTF-8, large UTF-8 and returned literals. - Base reproduction or other evidence: on the base revision the dossiers have no `swift_string_literals`; the issue shows the `referenced_strings: []` output. Across all 2561 functions of the issue's Vigil arm64 slice, the decoder reports 263 distinct literals in about 2 seconds. I checked each one against the app's source. All of them are real literals in one of these forms: * written verbatim in the source * interpolations of constants that the compiler folded (`"Library/LaunchAgents/\(guardLabel).plist"`) * imported IOKit keys (`Current Capacity`) * text inlined from the Swift runtime None of them is a partial or mixed value. - User-visible CLI/MCP output (if applicable): `rea function Vigil-arm64 0x10004da08 --provider ghidra` (LidMode.Failure.errorDescription): ```yaml referenced_strings: [] swift_string_literals[3]: - source_address: 0x10004daa0 value: "The user name " representation: small address: null registers[2]: x0,x1 words[2]: 0x7265737520656854,0xee0020656d616e20 - source_address: 0x10004dad4 value: " cannot be written into a sudoers rule safely." representation: large address: 0x100070270 registers[2]: x0,x1 words[2]: 0xd00000000000002e,0x8000000100070250 - source_address: 0x10004db08 value: The power setting could not be changed. representation: large address: 0x100070240 registers[2]: x0,x1 words[2]: 0xd000000000000027,0x8000000100070220 ``` `VigilKit.Format.rate` (0x10004d260) now lists `" B/s"`, `%.1f KB/s` and `%.1f MB/s`. The KB and MB formats share their final `movk` instructions across two branches, and both are reported at the tail call to `String(format:)`. - Remaining proof gaps: x86_64 Swift and ELF Swift on AArch64 were not run; the field is not produced for x86_64. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run verify:ghidra:swift` (macOS 27.0.1 on Apple silicon, Ghidra 12.1.4 with the native decompiler, JDK 21, Swift 6.4): passed. The existing inventory checks are unchanged, and `swift_string_literals` decoded 1, 1, 2 and 1 literals for the four fixture functions. - `npm run test:changed -- --base upstream/main`: 69 files, 917 passed, 3 skipped. - `npm run check:fast`: passed. - `npm run docs:check`: passed. - `rea function` on the Vigil arm64 slice for both functions in the issue: output shown above. ## Compatibility, safety, and release - Breaking changes or migration steps: none; the field and facet are additive and optional. - Real Hopper/Ghidra, browser, or OS coverage: real Ghidra 12.1.4 on macOS arm64. Hopper and IDA are untouched. - Package or release metadata impact: none. - Security, privacy, process, or containment review: reads program memory only through Ghidra's memory API, inside initialized blocks; no program mutation. ## Review checklist - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:57:44 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:42:41 +00:00

[PR #5019] [MERGED] test(update): isolate CLI subprocess deadlines #6737

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4989

No description provided.

Rows
Columns