morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4308] [Feature] Read unified log archives and correlate entries with os_log call sites #1713

New issue

Open

opened 2026-10-16 21:09:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:09:20 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 16, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4308

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

Accept an existing unified log archive (.logarchive, from log collect or a sysdiagnose's system_logs.logarchive) as read-only Evidence. Filter it by process, subsystem, category, sender image and time. Report each entry's sender image, format string and message. When an opened Mach-O's LC_UUID matches an entry's sender image, map the entry to its os_log call site and to the format string in the binary. Report redacted arguments (<private>) as unknown, not as empty or literal values. Live streaming and collection are excluded; they would need a separate maintainer scope decision.

Unified log entries are often the only runtime trace an analyst has from a crash, a sysdiagnose or a support case, and each entry names the binary image and format string that produced it. REA can neither read the archive nor connect an entry to the code that logged it. Measured with REA 6.5.0 (npx -y rea-agents@6.5.0, tag rea-agents-6.5.0, commit 3b79af28e80785afce9f5998d8f03289402a1847) on macOS 27.0.1 (26A434) arm64 with Xcode 27.0 (27A266a) and Node.js v26.10.0:

A self-built fixture logs two messages through os_log_create("com.example.rea-fixture", "probe"), and /usr/bin/log collect --last 2m --output ./fx.logarchive writes an archive. On the measurement host log collect ran without sudo.
/usr/bin/log show --archive ./fx.logarchive --style json --predicate 'subsystem == "com.example.rea-fixture"' returns two entries. Each carries formatString (rea fixture public=%{public}s private=%s count=%d), eventMessage, subsystem, category, messageType, processImageUUID, senderImageUUID, senderImagePath and senderProgramCounter (1384).
senderImageUUID equals the fixture's LC_UUID. senderProgramCounter 1384 is 0x568; in the fixture otool -tv shows 0x100000568 as the instruction after bl __os_log_impl in main, and atos -arch arm64 -o ./rea-log-fixture -l 0x100000000 0x100000568 returns main + 184. The format string is stored at 0x100000712 in __TEXT,__oslogstring (otool -v -s __TEXT __oslogstring). The join key exists in both artifacts.
rea inspect-artifact ./fx.logarchive --json fails with target_unavailable: "open_binary accepts files and macOS app bundles. This target is a directory; JavaScript/Electron application directories can be analyzed directly with analyze_javascript_application or rea analyze <directory>." (analysisErrorPresentation.ts). A search of src, bridge and docs in the release and in main at d52a488 finds no handling of logarchive, os_log or __oslogstring.

Use cases

No response

Desired outcome and success criteria

Accept one caller-selected .logarchive directory as captured data. Require or default a bounded selection: a time range, an entry cap, and optional filters on process, PID, subsystem, category, message type, and sender image UUID or path. Report truncation explicitly and never return a silently cut list.
For each entry, report the timestamp with its recorded zone, process path and UUID, PID, thread and activity IDs, subsystem, category, level, formatString, the composed message, sender image path and UUID, and sender offset. Report fields the reader does not supply as unknown, not empty.
When an opened Mach-O slice's LC_UUID equals an entry's sender image UUID, add the static address (image base plus sender offset), the containing function REA already knows, and the location of the identical format string in the binary's os_log string section. A UUID or architecture mismatch never correlates, and the result says why.
Report each argument's declared privacy from the format string's modifiers (%{public}, %{private}, or no modifier, which means the documented default). Where the message carries a <private> placeholder, report that argument as redacted with an unknown value. Never report the literal text <private>, or an empty string, as the value.
The archive's binary format is undocumented, and Apple directs readers to its tools, so the reader may be macOS-only. On other hosts the operation returns a typed unsupported_host result. The archive is never modified.
Evidence carries archive identity (path and digest of its metadata files), the filters used, coverage (entries matched versus returned) and limitations.

Acceptance criteria

Build the fixture with printf '#include <os/log.h>\nint main(void){os_log_t h=os_log_create("com.example.rea-fixture","probe");os_log(h,"rea fixture public=%%{public}s private=%%s count=%%d","visible-token","hidden-token",7);os_log_error(h,"rea fixture error path=%%{public}s","/tmp/rea-fixture");return 0;}\n' > logfix.c && clang -o rea-log-fixture logfix.c && ./rea-log-fixture && /usr/bin/log collect --last 2m --output ./fx.logarchive.
Filtering the archive by subsystem com.example.rea-fixture returns exactly the two entries, and their formatString, message, level and sender image UUID equal those from log show --archive ./fx.logarchive --style json with the same predicate.
With rea-log-fixture opened, both entries correlate to main, at the addresses atos gives for image base 0x100000000 plus each sender offset, and to the two strings otool -v -s __TEXT __oslogstring lists. After a rebuild that changes the UUID, the old archive's entries do not correlate.
The private=%s argument is reported as declared default-private. Given an archive whose log show --archive output shows <private> in place of that argument, the value is reported as redacted and unknown, never as the text <private> or an empty string. On the measurement host log show printed the argument in clear text (private=hidden-token), so this check needs a host, or a supplied archive, where Apple's default redaction applies.
A time window or cap that excludes some matching entries reports truncation with counts. A plain directory that is not an archive returns a typed input error. On Linux the operation returns unsupported_host.
CLI and MCP results match, and the archive's file digests are unchanged after the call.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Affected area

A new Apple artifact operation for log archives, with its CLI command, MCP schema and documentation, and the join to inspect_macho slice identity plus Mach-O string-section reading for an opened target.

Non-goals

log stream, log collect, device or simulator collection, or triggering a sysdiagnose. Collecting new logs observes or acts on the host and needs an explicit maintainer scope decision under CONTRIBUTING.md.
Changing logging configuration (log config), enabling debug or private-data logging, or recovering redacted values.
Parsing the archive's internal binary files without Apple's reader.
Signposts, activities as trees, and Instruments .trace import. These are candidate follow-ups.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

No response

Alternatives considered

log show --archive ... --style json gives the raw entries, but the JSON style's keys are not described in Apple's documentation corpus, the output is not Evidence, and correlating with a binary needs manual atos and otool work for every entry.

Possible direction

No response

References

Viewing Log Messages: "The unified log system stores log messages in a binary compressed format ... this means you can't read and parse the log files directly; you need to use tools".
OSLogStore: instances "may be backed by a logarchive or your Mac's local store", and init(url:) "Creates a log store based on a log archive." This documented API is a possible backend for the reader, not a user-facing alternative. OSLogEntryFromProcess documents sender as "The name of the binary image that made the entry", and OSLogEntryWithPayload documents formatString as "The payload's format string".
Generating Log Messages from Your Code: "By default, the system doesn't redact integer, floating-point and Boolean values, but it does redact the contents of dynamic strings and complex dynamic objects."
TN3163: Understanding the synchronization of NSPersistentCloudKitContainer shows that sysdiagnose system logs "are archived in the file named system_logs.logarchive" and reads them with log show ... ./system_logs.logarchive.
Redaction on the measurement host: the clear-text argument appeared in both --style json and --style compact output. The cause was not determined, because log config --status requires root.
Scope: reading captured data is format inspection (CONTRIBUTING.md). The proposal launches no target, and the only process it might start is Apple's archive reader, as REA already runs otool and codesign.
GitHub searches for "logarchive", "unified log", "os_log" and "log show" found no related issue or pull request.

Originally created by @coygeek on GitHub (Oct 16, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4308 ### Before submitting - [X] I searched the open and closed issues and pull requests for a related proposal. - [X] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [X] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation Accept an existing unified log archive (`.logarchive`, from `log collect` or a sysdiagnose's `system_logs.logarchive`) as read-only Evidence. Filter it by process, subsystem, category, sender image and time. Report each entry's sender image, format string and message. When an opened Mach-O's `LC_UUID` matches an entry's sender image, map the entry to its `os_log` call site and to the format string in the binary. Report redacted arguments (`<private>`) as unknown, not as empty or literal values. Live streaming and collection are excluded; they would need a separate maintainer scope decision. Unified log entries are often the only runtime trace an analyst has from a crash, a sysdiagnose or a support case, and each entry names the binary image and format string that produced it. REA can neither read the archive nor connect an entry to the code that logged it. Measured with REA 6.5.0 (`npx -y rea-agents@6.5.0`, tag `rea-agents-6.5.0`, commit `3b79af28e80785afce9f5998d8f03289402a1847`) on macOS 27.0.1 (26A434) arm64 with Xcode 27.0 (27A266a) and Node.js v26.10.0: - A self-built fixture logs two messages through `os_log_create("com.example.rea-fixture", "probe")`, and `/usr/bin/log collect --last 2m --output ./fx.logarchive` writes an archive. On the measurement host `log collect` ran without `sudo`. - `/usr/bin/log show --archive ./fx.logarchive --style json --predicate 'subsystem == "com.example.rea-fixture"'` returns two entries. Each carries `formatString` (`rea fixture public=%{public}s private=%s count=%d`), `eventMessage`, `subsystem`, `category`, `messageType`, `processImageUUID`, `senderImageUUID`, `senderImagePath` and `senderProgramCounter` (`1384`). - `senderImageUUID` equals the fixture's `LC_UUID`. `senderProgramCounter` 1384 is `0x568`; in the fixture `otool -tv` shows `0x100000568` as the instruction after `bl __os_log_impl` in `main`, and `atos -arch arm64 -o ./rea-log-fixture -l 0x100000000 0x100000568` returns `main + 184`. The format string is stored at `0x100000712` in `__TEXT,__oslogstring` (`otool -v -s __TEXT __oslogstring`). The join key exists in both artifacts. - `rea inspect-artifact ./fx.logarchive --json` fails with `target_unavailable`: "open_binary accepts files and macOS app bundles. This target is a directory; JavaScript/Electron application directories can be analyzed directly with analyze_javascript_application or `rea analyze <directory>`." ([analysisErrorPresentation.ts](https://github.com/morluto/rea/blob/3b79af28e80785afce9f5998d8f03289402a1847/src/domain/analysisErrorPresentation.ts#L266-L268)). A search of `src`, `bridge` and `docs` in the release and in main at `d52a488` finds no handling of `logarchive`, `os_log` or `__oslogstring`. ### Use cases _No response_ ### Desired outcome and success criteria - Accept one caller-selected `.logarchive` directory as captured data. Require or default a bounded selection: a time range, an entry cap, and optional filters on process, PID, subsystem, category, message type, and sender image UUID or path. Report truncation explicitly and never return a silently cut list. - For each entry, report the timestamp with its recorded zone, process path and UUID, PID, thread and activity IDs, subsystem, category, level, `formatString`, the composed message, sender image path and UUID, and sender offset. Report fields the reader does not supply as unknown, not empty. - When an opened Mach-O slice's `LC_UUID` equals an entry's sender image UUID, add the static address (image base plus sender offset), the containing function REA already knows, and the location of the identical format string in the binary's `os_log` string section. A UUID or architecture mismatch never correlates, and the result says why. - Report each argument's declared privacy from the format string's modifiers (`%{public}`, `%{private}`, or no modifier, which means the documented default). Where the message carries a `<private>` placeholder, report that argument as redacted with an unknown value. Never report the literal text `<private>`, or an empty string, as the value. - The archive's binary format is undocumented, and Apple directs readers to its tools, so the reader may be macOS-only. On other hosts the operation returns a typed `unsupported_host` result. The archive is never modified. - Evidence carries archive identity (path and digest of its metadata files), the filters used, coverage (entries matched versus returned) and limitations. **Acceptance criteria** - Build the fixture with `printf '#include <os/log.h>\nint main(void){os_log_t h=os_log_create("com.example.rea-fixture","probe");os_log(h,"rea fixture public=%%{public}s private=%%s count=%%d","visible-token","hidden-token",7);os_log_error(h,"rea fixture error path=%%{public}s","/tmp/rea-fixture");return 0;}\n' > logfix.c && clang -o rea-log-fixture logfix.c && ./rea-log-fixture && /usr/bin/log collect --last 2m --output ./fx.logarchive`. - Filtering the archive by subsystem `com.example.rea-fixture` returns exactly the two entries, and their `formatString`, message, level and sender image UUID equal those from `log show --archive ./fx.logarchive --style json` with the same predicate. - With `rea-log-fixture` opened, both entries correlate to `main`, at the addresses `atos` gives for image base `0x100000000` plus each sender offset, and to the two strings `otool -v -s __TEXT __oslogstring` lists. After a rebuild that changes the UUID, the old archive's entries do not correlate. - The `private=%s` argument is reported as declared default-private. Given an archive whose `log show --archive` output shows `<private>` in place of that argument, the value is reported as redacted and unknown, never as the text `<private>` or an empty string. On the measurement host `log show` printed the argument in clear text (`private=hidden-token`), so this check needs a host, or a supplied archive, where Apple's default redaction applies. - A time window or cap that excludes some matching entries reports truncation with counts. A plain directory that is not an archive returns a typed input error. On Linux the operation returns `unsupported_host`. - CLI and MCP results match, and the archive's file digests are unchanged after the call. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) **Affected area** A new Apple artifact operation for log archives, with its CLI command, MCP schema and documentation, and the join to `inspect_macho` slice identity plus Mach-O string-section reading for an opened target. **Non-goals** - `log stream`, `log collect`, device or simulator collection, or triggering a sysdiagnose. Collecting new logs observes or acts on the host and needs an explicit maintainer scope decision under [CONTRIBUTING.md](https://github.com/morluto/rea/blob/3b79af28e80785afce9f5998d8f03289402a1847/CONTRIBUTING.md#L117-L124). - Changing logging configuration (`log config`), enabling debug or private-data logging, or recovering redacted values. - Parsing the archive's internal binary files without Apple's reader. - Signposts, activities as trees, and Instruments `.trace` import. These are candidate follow-ups. ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) _No response_ ### Alternatives considered - `log show --archive ... --style json` gives the raw entries, but the JSON style's keys are not described in Apple's documentation corpus, the output is not Evidence, and correlating with a binary needs manual `atos` and `otool` work for every entry. ### Possible direction _No response_ ### References - [Viewing Log Messages](https://developer.apple.com/documentation/os/viewing-log-messages): "The unified log system stores log messages in a binary compressed format ... this means you can't read and parse the log files directly; you need to use tools". - [OSLogStore](https://developer.apple.com/documentation/oslog/oslogstore): instances "may be backed by a `logarchive` or your Mac's local store", and `init(url:)` "Creates a log store based on a log archive." This documented API is a possible backend for the reader, not a user-facing alternative. [OSLogEntryFromProcess](https://developer.apple.com/documentation/oslog/oslogentryfromprocess) documents `sender` as "The name of the binary image that made the entry", and [OSLogEntryWithPayload](https://developer.apple.com/documentation/oslog/oslogentrywithpayload) documents `formatString` as "The payload's format string". - [Generating Log Messages from Your Code](https://developer.apple.com/documentation/os/generating-log-messages-from-your-code): "By default, the system doesn't redact integer, floating-point and Boolean values, but it does redact the contents of dynamic strings and complex dynamic objects." - [TN3163: Understanding the synchronization of NSPersistentCloudKitContainer](https://developer.apple.com/documentation/technotes/tn3163-understanding-the-synchronization-of-nspersistentcloudkitcontainer) shows that sysdiagnose system logs "are archived in the file named `system_logs.logarchive`" and reads them with `log show ... ./system_logs.logarchive`. - Redaction on the measurement host: the clear-text argument appeared in both `--style json` and `--style compact` output. The cause was not determined, because `log config --status` requires root. - Scope: reading captured data is format inspection ([CONTRIBUTING.md](https://github.com/morluto/rea/blob/3b79af28e80785afce9f5998d8f03289402a1847/CONTRIBUTING.md#L111-L115)). The proposal launches no target, and the only process it might start is Apple's archive reader, as REA already runs `otool` and `codesign`. - GitHub searches for "logarchive", "unified log", "os_log" and "log show" found no related issue or pull request.

bsakaguchi referenced this issue

2026-10-31 00:54:54 +00:00

[PR #1713] [MERGED] fix(native): retain failed command cleanup owners #4772

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1713

No description provided.

Rows
Columns