morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3967] feat(procmon): read saved Process Monitor XML and CSV exports as Evidence #5972

New issue

Open

opened 2026-10-31 01:10:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:10:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3967
Author: @aftabkh4n
Created: 10/15/2026
Status: 🔄 Open

Base: main ← Head: feat/procmon-export-evidence

📝 Commits (10+)

c9c14b4 feat(procmon): read saved Process Monitor XML and CSV exports as Evidence
42b3837 fix(procmon): bound parsing memory and address review findings
c5245e4 fix(procmon): drop a cut final CSV row and size events exactly
e014743 fix(procmon): address Codex review of the parsing fixes
6e44894 fix(procmon): address review of e014743
e3aaded fix(procmon): derive missing columns from the final coverage
46cba65 fix(procmon): track nested XML field names and refine budget errors
8d3d950 fix(procmon): bound the declared-encoding error and count nested fields as seen
2948181 fix(procmon): bound XML parser diagnostics in partial coverage
f171915 fix(procmon): copy retained text out of the decoded export

📊 Changes

54 files changed (+3044 additions, -1 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+6 -0)
📝 .gitattributes (+3 -0)
📝 README.md (+1 -0)
📝 docs/.vitepress/config.ts (+1 -0)
📝 docs/cli.md (+10 -0)
➕ docs/procmon-exports.md (+94 -0)
📝 docs/readme/README_ar.md (+1 -0)
📝 docs/readme/README_de.md (+1 -0)
📝 docs/readme/README_el.md (+1 -0)
📝 docs/readme/README_es.md (+1 -0)
📝 docs/readme/README_fa.md (+1 -0)
📝 docs/readme/README_fr.md (+1 -0)
📝 docs/readme/README_id.md (+1 -0)
📝 docs/readme/README_it.md (+1 -0)
📝 docs/readme/README_ja.md (+1 -0)
📝 docs/readme/README_ko.md (+1 -0)
📝 docs/readme/README_pl.md (+1 -0)
📝 docs/readme/README_pt-BR.md (+1 -0)
📝 docs/readme/README_ru.md (+1 -0)
📝 docs/readme/README_th.md (+1 -0)

...and 34 more files

📄 Description

Summary

Fixes #3766.

Analysts often receive Process Monitor captures that someone else exported. REA had no way to turn those exports into Evidence. This adds a read-only inspect_procmon_export tool (CLI: inspect-procmon-export) that reads one caller-selected XML or CSV export and returns its events as inline Evidence through one shared service.

Problem and expected behavior

Given a local Procmon XML or CSV export, an agent or analyst can page through its events and get, for each one, its 1-based ordinal and the Time of Day, Process Name, PID, Operation, Path, Result and Detail text exactly as exported. Other columns (TID, Sequence, …) stay under additional by source name. A column missing from the export is listed in missing_columns and is null in every event, not "".

A truncated or malformed export keeps the events read before the problem, with coverage: partial, the reason (truncated/malformed), a message and the stopping line, instead of failing silently or entirely.

Change and scope

src/domain/procmon/procmonExport.ts: input and report schemas and limits. The report has a superRefine that checks the returned window against offset/limit, like packetCaptureSchema.
src/procmon/: the in-process reader.
- CSV: RFC 4180 (quotes, doubled quotes, embedded line breaks, CRLF/LF), with the header row giving the columns.
- XML: streaming saxes parse of procmon > eventlist > event. DTDs are rejected. Fields with child elements such as stack are named in omitted_nested_fields and not projected. The process list is not read.
- Encodings: UTF-8 with or without BOM, or BOM-marked UTF-16. XML encoding declarations are checked with the existing decodeXmlPlistText.
ProcmonExportService validates input and output and builds Evidence. It is wired the same way as inspect_rust_binary: contract, effects (evidence), capability inventory, CLI, MCP registration, catalog builders and module boundaries.
Docs:
- docs/procmon-exports.md
- a CLI section, sidebar entry and README row (plus the translated rows docs:check requires)
- the skill entry
- docs/specs/3766-procmon-export/

Not included: live ETW collection (#2689), PML parsing, interpreting what events mean, and localized Procmon column headers (see gaps below).

Contract and boundary impact

Semantic owner and earliest changed stage: new src/procmon/ reader owns export parsing; domain schema in src/domain/procmon/.
Contribution scope fit and linked maintainer decision: reads a recorded file only (no target execution, injection, mutation or protection recovery), so no scope decision is required per CONTRIBUTING; implements the issue's acceptance criteria.
CLI and MCP/tool-catalog contract: adds one tool, inspect_procmon_export, and one command, inspect-procmon-export (with --event_offset, --event_limit, --all_events and --export_format). Effects: mutatesSession only, the same as other static Evidence tools.
Provider, bridge, target-format, or platform compatibility: bundled in-process reader (procmon-export v1); no external tool; any platform.
Evidence, artifact, provenance, or reconstruction contract: subject format file with path and SHA-256. The report carries the artifact path, SHA-256 and byte length, plus the source format, encoding, BOM and column order.
Target selection, authority, upstream behavior, and cleanup details: the single absolute path is read with readStableArtifact. No temp files or child processes.
Generated metadata, package, or installation impact: catalog builders include the new contract and provider identity; no new dependencies (saxes is already a dependency).

Expected effects:

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated:
- tests/acceptance/procmonExportCli.test.ts (6 tests, public CLI):
  - CSV with BOM, CRLF, quoted commas and doubled quotes
  - XML with an absent field (null) and a nested stack field
  - paging with offset/limit and --all_events
  - missing columns as null, with an extra column kept under additional
  - truncated CSV (unclosed quote), malformed CSV (wrong field count) and truncated XML, each returning earlier events with partial coverage
  - non-Procmon CSV and XML rejected as invalid_request
- tests/boundary/mcp/procmonExportMcp.test.ts (3 tests):
  - advertised annotations and MCP paging over the XML fixture
  - BOM-marked UTF-16LE CSV
  - rejection of relative paths and DTDs
- Hand-built fixtures in tests/fixtures/procmon/. A .gitattributes rule keeps the CSV fixture's CRLF rows.
Base reproduction or other evidence: new feature; no prior behavior.
User-visible CLI/MCP output (rea inspect-procmon-export tests/fixtures/procmon/events.xml --event_limit 1 --json, trimmed to normalized_result):

Output

{
  "source": {
    "format": "xml",
    "encoding": "utf-8",
    "byte_order_mark": false,
    "columns": ["ProcessIndex", "Time_of_Day", "Process_Name", "PID", "Operation", "Path", "Result", "Detail"]
  },
  "missing_columns": [],
  "omitted_nested_fields": ["stack"],
  "events": [
    {
      "ordinal": 1,
      "time_of_day": "9:41:07.1234567 AM",
      "process_name": "sample.exe",
      "pid": "4120",
      "operation": "RegOpenKey",
      "path": "HKLM\\Software\\Sample",
      "result": "SUCCESS",
      "detail": "Desired Access: Read",
      "additional": [{ "column": "ProcessIndex", "value": "1" }]
    }
  ],
  "output_window": { "examined_events": 3, "returned_events": 1, "has_more": true },
  "coverage": { "status": "complete" }
}

Remaining proof gaps:
- The fixtures follow the Procmon CSV and XML layouts but were written by hand, not exported from a running Process Monitor. Real exports have not been checked.
- Procmon versions that write localized column headers would only map if the header names match the English names. Such exports are rejected when no English column is present; otherwise the unmatched columns land under additional.
- The 8 MiB output-budget failure path is implemented but has no dedicated test.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run test:focused -- tests/acceptance/procmonExportCli.test.ts tests/boundary/mcp/procmonExportMcp.test.ts: 9 passed
npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts src/contracts/toolEffects.test.ts: 20 passed (run before the reader was split into modules; the contract and schemas did not change in that split)
npm run test:changed -- --base upstream/main: 61 files, 1113 passed, 4 skipped
npm run check (typecheck, lint:built, format:check, knip, metadata:check, architecture guards): passed; the only new lint warning is max-params on registerProcmonTools, which every register*Tools sibling shares
npm run docs:check: passed
`Review fixes in d3f0ffea: Procmon reader, CLI and MCP tests 58 passed; typecheck, lint, knip, metadata and docs checks passed.

Compatibility, safety, and release

Breaking changes or migration steps: none; additive tool and command.
Real Hopper/Ghidra, browser, or OS coverage: not applicable. Linux only (Node 22.22); not run on Windows or macOS.
Package or release metadata impact: catalog entries for the new tool and provider identity; no dependency changes.
Security, privacy, process, or containment review:
- Reads one file of at most 128 MiB.
- XML DTDs are rejected, so there is no entity expansion.
- At most 1024 columns.
- Each event is admitted against the 8 MiB output budget before it is retained.
- Event text is target content and is returned as data.

Author readiness

Human owner: @aftabkh4n

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

https://claude.ai/code/session_019S6gW1R7hWkvBKbBYnSYgK

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3967 **Author:** [@aftabkh4n](https://github.com/aftabkh4n) **Created:** 10/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/procmon-export-evidence` --- ### 📝 Commits (10+) - [`c9c14b4`](https://github.com/morluto/rea/commit/c9c14b4a3c10acf2e13acf071f15527b519ceb76) feat(procmon): read saved Process Monitor XML and CSV exports as Evidence - [`42b3837`](https://github.com/morluto/rea/commit/42b3837a59340fb67f87759ef23942986d5e3d90) fix(procmon): bound parsing memory and address review findings - [`c5245e4`](https://github.com/morluto/rea/commit/c5245e466988440d0154a05be92252cc8455f4f4) fix(procmon): drop a cut final CSV row and size events exactly - [`e014743`](https://github.com/morluto/rea/commit/e0147435464928ae7f8081e9d2cb99c3a74e325c) fix(procmon): address Codex review of the parsing fixes - [`6e44894`](https://github.com/morluto/rea/commit/6e448948c2224b2c9acdc70ec14848a06dd9beb1) fix(procmon): address review of e014743 - [`e3aaded`](https://github.com/morluto/rea/commit/e3aaded506d33a7e4dfbe100fefb888958645348) fix(procmon): derive missing columns from the final coverage - [`46cba65`](https://github.com/morluto/rea/commit/46cba6529f9ea9d12d3f70aa8270ad383640e215) fix(procmon): track nested XML field names and refine budget errors - [`8d3d950`](https://github.com/morluto/rea/commit/8d3d950287ae98b4dfc4355b2ec7ca3eb51748ab) fix(procmon): bound the declared-encoding error and count nested fields as seen - [`2948181`](https://github.com/morluto/rea/commit/294818157d3c8aef6445321e5b44d92e0031daf4) fix(procmon): bound XML parser diagnostics in partial coverage - [`f171915`](https://github.com/morluto/rea/commit/f1719150d725f1b40a2cc9a91cf47968f1234e1b) fix(procmon): copy retained text out of the decoded export ### 📊 Changes **54 files changed** (+3044 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+6 -0) 📝 `.gitattributes` (+3 -0) 📝 `README.md` (+1 -0) 📝 `docs/.vitepress/config.ts` (+1 -0) 📝 `docs/cli.md` (+10 -0) ➕ `docs/procmon-exports.md` (+94 -0) 📝 `docs/readme/README_ar.md` (+1 -0) 📝 `docs/readme/README_de.md` (+1 -0) 📝 `docs/readme/README_el.md` (+1 -0) 📝 `docs/readme/README_es.md` (+1 -0) 📝 `docs/readme/README_fa.md` (+1 -0) 📝 `docs/readme/README_fr.md` (+1 -0) 📝 `docs/readme/README_id.md` (+1 -0) 📝 `docs/readme/README_it.md` (+1 -0) 📝 `docs/readme/README_ja.md` (+1 -0) 📝 `docs/readme/README_ko.md` (+1 -0) 📝 `docs/readme/README_pl.md` (+1 -0) 📝 `docs/readme/README_pt-BR.md` (+1 -0) 📝 `docs/readme/README_ru.md` (+1 -0) 📝 `docs/readme/README_th.md` (+1 -0) _...and 34 more files_ </details> ### 📄 Description ## Summary Fixes #3766. Analysts often receive Process Monitor captures that someone else exported. REA had no way to turn those exports into Evidence. This adds a read-only `inspect_procmon_export` tool (CLI: `inspect-procmon-export`) that reads one caller-selected XML or CSV export and returns its events as inline Evidence through one shared service. ## Problem and expected behavior Given a local Procmon XML or CSV export, an agent or analyst can page through its events and get, for each one, its 1-based ordinal and the Time of Day, Process Name, PID, Operation, Path, Result and Detail text exactly as exported. Other columns (`TID`, `Sequence`, …) stay under `additional` by source name. A column missing from the export is listed in `missing_columns` and is `null` in every event, not `""`. A truncated or malformed export keeps the events read before the problem, with `coverage: partial`, the reason (`truncated`/`malformed`), a message and the stopping line, instead of failing silently or entirely. ## Change and scope - `src/domain/procmon/procmonExport.ts`: input and report schemas and limits. The report has a superRefine that checks the returned window against offset/limit, like `packetCaptureSchema`. - `src/procmon/`: the in-process reader. - CSV: RFC 4180 (quotes, doubled quotes, embedded line breaks, CRLF/LF), with the header row giving the columns. - XML: streaming `saxes` parse of `procmon > eventlist > event`. DTDs are rejected. Fields with child elements such as `stack` are named in `omitted_nested_fields` and not projected. The process list is not read. - Encodings: UTF-8 with or without BOM, or BOM-marked UTF-16. XML encoding declarations are checked with the existing `decodeXmlPlistText`. - `ProcmonExportService` validates input and output and builds Evidence. It is wired the same way as `inspect_rust_binary`: contract, effects (`evidence`), capability inventory, CLI, MCP registration, catalog builders and module boundaries. - Docs: - `docs/procmon-exports.md` - a CLI section, sidebar entry and README row (plus the translated rows `docs:check` requires) - the skill entry - `docs/specs/3766-procmon-export/` Not included: live ETW collection (#2689), PML parsing, interpreting what events mean, and localized Procmon column headers (see gaps below). ## Contract and boundary impact - Semantic owner and earliest changed stage: new `src/procmon/` reader owns export parsing; domain schema in `src/domain/procmon/`. - Contribution scope fit and linked maintainer decision: reads a recorded file only (no target execution, injection, mutation or protection recovery), so no scope decision is required per CONTRIBUTING; implements the issue's acceptance criteria. - CLI and MCP/tool-catalog contract: adds one tool, `inspect_procmon_export`, and one command, `inspect-procmon-export` (with `--event_offset`, `--event_limit`, `--all_events` and `--export_format`). Effects: `mutatesSession` only, the same as other static Evidence tools. - Provider, bridge, target-format, or platform compatibility: bundled in-process reader (`procmon-export` v1); no external tool; any platform. - Evidence, artifact, provenance, or reconstruction contract: subject format `file` with path and SHA-256. The report carries the artifact path, SHA-256 and byte length, plus the source format, encoding, BOM and column order. - Target selection, authority, upstream behavior, and cleanup details: the single absolute path is read with `readStableArtifact`. No temp files or child processes. - Generated metadata, package, or installation impact: catalog builders include the new contract and provider identity; no new dependencies (`saxes` is already a dependency). Expected effects: - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `tests/acceptance/procmonExportCli.test.ts` (6 tests, public CLI): - CSV with BOM, CRLF, quoted commas and doubled quotes - XML with an absent field (null) and a nested `stack` field - paging with offset/limit and `--all_events` - missing columns as null, with an extra column kept under `additional` - truncated CSV (unclosed quote), malformed CSV (wrong field count) and truncated XML, each returning earlier events with partial coverage - non-Procmon CSV and XML rejected as `invalid_request` - `tests/boundary/mcp/procmonExportMcp.test.ts` (3 tests): - advertised annotations and MCP paging over the XML fixture - BOM-marked UTF-16LE CSV - rejection of relative paths and DTDs - Hand-built fixtures in `tests/fixtures/procmon/`. A `.gitattributes` rule keeps the CSV fixture's CRLF rows. - Base reproduction or other evidence: new feature; no prior behavior. - User-visible CLI/MCP output (`rea inspect-procmon-export tests/fixtures/procmon/events.xml --event_limit 1 --json`, trimmed to `normalized_result`): <details><summary>Output</summary> ```json { "source": { "format": "xml", "encoding": "utf-8", "byte_order_mark": false, "columns": ["ProcessIndex", "Time_of_Day", "Process_Name", "PID", "Operation", "Path", "Result", "Detail"] }, "missing_columns": [], "omitted_nested_fields": ["stack"], "events": [ { "ordinal": 1, "time_of_day": "9:41:07.1234567 AM", "process_name": "sample.exe", "pid": "4120", "operation": "RegOpenKey", "path": "HKLM\\Software\\Sample", "result": "SUCCESS", "detail": "Desired Access: Read", "additional": [{ "column": "ProcessIndex", "value": "1" }] } ], "output_window": { "examined_events": 3, "returned_events": 1, "has_more": true }, "coverage": { "status": "complete" } } ``` </details> - Remaining proof gaps: - The fixtures follow the Procmon CSV and XML layouts but were written by hand, not exported from a running Process Monitor. Real exports have not been checked. - Procmon versions that write localized column headers would only map if the header names match the English names. Such exports are rejected when no English column is present; otherwise the unmatched columns land under `additional`. - The 8 MiB output-budget failure path is implemented but has no dedicated test. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- tests/acceptance/procmonExportCli.test.ts tests/boundary/mcp/procmonExportMcp.test.ts`: 9 passed - `npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts src/contracts/toolEffects.test.ts`: 20 passed (run before the reader was split into modules; the contract and schemas did not change in that split) - `npm run test:changed -- --base upstream/main`: 61 files, 1113 passed, 4 skipped - `npm run check` (typecheck, lint:built, format:check, knip, metadata:check, architecture guards): passed; the only new lint warning is `max-params` on `registerProcmonTools`, which every `register*Tools` sibling shares - `npm run docs:check`: passed - `Review fixes in d3f0ffea: Procmon reader, CLI and MCP tests 58 passed; typecheck, lint, knip, metadata and docs checks passed. ## Compatibility, safety, and release - Breaking changes or migration steps: none; additive tool and command. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. Linux only (Node 22.22); not run on Windows or macOS. - Package or release metadata impact: catalog entries for the new tool and provider identity; no dependency changes. - Security, privacy, process, or containment review: - Reads one file of at most 128 MiB. - XML DTDs are rejected, so there is no entity expansion. - At most 1024 columns. - Each event is admitted against the 8 MiB output budget before it is retained. - Event text is target content and is returned as data. ## Author readiness Human owner: @aftabkh4n - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019S6gW1R7hWkvBKbBYnSYgK --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:10:51 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5972

No description provided.

Rows
Columns