[PR #2216] [MERGED] test(native): verify real PE and DOS entry views #5090

Closed
opened 2026-10-31 01:03:47 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2216
Author: @N0zoM1z0
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/12/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/native-view-verification-20261012


📝 Commits (7)

  • 6d7f688 test(native): add real PE and DOS entry view experiments
  • b74b9b7 docs(native): record real PE and DOS delivery costs
  • ba5cf12 fix(experiments): separate native facts from Evidence references
  • 0e92a5d fix(experiments): reject reused native view identities
  • 4e149fb fix(experiments): retain DOS load-image source identity
  • aada535 fix(experiments): verify DOS load-image target identity
  • 0667f7c fix(experiments): bind native claims to observed source evidence

📊 Changes

10 files changed (+920 additions, -124 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+4 -0)
📝 experiments/analysis-efficiency/compare.mjs (+15 -1)
📝 experiments/analysis-efficiency/corpus.json (+30 -0)
📝 experiments/analysis-efficiency/measure.mjs (+32 -116)
➕ experiments/analysis-efficiency/native-entry-views.md (+178 -0)
➕ experiments/analysis-efficiency/native-entry.mjs (+272 -0)
➕ experiments/analysis-efficiency/questions/native-dos.txt (+1 -0)
➕ experiments/analysis-efficiency/questions/native-pe.txt (+1 -0)
📝 experiments/analysis-efficiency/review-model.py (+51 -7)
➕ tests/boundary/cli/nativeEfficiencyComparison.test.ts (+336 -0)

📄 Description

Summary

Native paged delivery needs real-artifact evidence before expanding the WASM summary approach. Record public ShareX native PE and FreeDOS DOS MZ entry workflows, including a small PE function where four retained facets cost more bytes than its complete dossier. Continues #2016; the broader issue stays open.

Problem and expected behavior

A smaller individual reply does not establish a cheaper complete investigation. Verify identical source facts and identities through actual Ghidra/MCP and real Codex CLI, retain failed calls and unknowns, and measure the complete question before recommending another API change.

Change and scope

Add two digest-pinned shipped inputs and question templates, extract the existing jq native experiment into a shared native helper, and add an explicit entry-views verification mode. In that new mode, keep selected facts separate from original Evidence reference maps at the producing boundary; preserve both runs' provenance in comparison reports. Preserve the existing jq entry/main call sequence. Extend the grader to compare exact values for equivalent native answer representations and reject changed units/ranges/identity/load metadata.

Curated findings: native entry-view checkpoint. Detailed JSON, binaries, model transcripts and credentials remain outside Git.

Contract and boundary impact

  • Semantic owner and earliest changed stage: experiment driver and its answer grader; no production semantics change.
  • CLI and MCP/tool-catalog contract: none. Existing tool names, accepted inputs, complete defaults and schemas are unchanged.
  • Provider, bridge, target-format, or platform compatibility: no provider code changes. Actual Ghidra 12.1.4 on Linux inspects shipped Windows PE and original compressed DOS MZ.
  • Evidence, artifact, provenance, or reconstruction contract: no production changes. Facets are checked against original producer facts, target digest, parent ID and distinct view IDs. Unknown/static/packed-target limitations remain explicit.
  • Process execution, authorization, cleanup, or containment impact: existing serial one-CPU/4 GiB hard memory scopes; default auto-analysis, ephemeral session close, unchanged target bytes, copied-profile cleanup. No target execution.
  • Generated metadata, package, or installation impact: none; no generated outputs or release metadata committed.

Evidence and regression coverage

  • Tests added or updated: reproducible experimental native workflows and exact-value grading; sixteen focused CLI producer/comparator regression cases. Different run-specific references compare successfully, while item/coverage/limitations/unknown differences remain failures; reused parent/facet/summary IDs are rejected, and the DOS load-image source is retained for its metadata and derived entry seed. A stale verified load image is rejected before procedure lookup and the session is closed. Completed DOS load-image observations and their reported source IDs are required for a load-image grade; missing/failed/unreferenced sources or wrong observed target/entry are rejected. These negative cases are synthetic grader fixtures, not real-provider/resource measurements. No production behavior change.

  • Executed evidence: both public SDK workflows; three actual Codex CLI trials (gpt-5.6-terra, high; PE twice, DOS once). Selected facts/references/lifecycle checks match original observations.

  • Existing driver regression: replay actual recorded jq entry and entry/main replies, preserving selected facts/profile and 4/12-call ordering. Replay actual PE/DOS replies against the committed extracted helper with exact answer/9/10-call agreement. These are replays, not fresh provider runs.

  • User-visible CLI/MCP output: unchanged; experimental notes include real response sizes and complete task costs.

  • Remaining proof gaps: Linux static inspection is not Windows runtime/OS coverage; compressed DOS entry analysis is not editor logic or COM coverage; final cgroup closeout counters are unverified. No overall native speedup or resource-qualified performance win is claimed.

  • Observed, derived, and inferred claims remain distinguishable.

  • Artifact identity, source provenance, and failed attempts remain preserved.

  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • Public measure.mjs --native-question entry-views --schemas selected — PE/DOS exact selected-fact and Evidence checks pass; native summary refusal recorded separately.
  • agent-trial.py + review-model.py --domain native — three real model answers match selected facts and cleanup; strict overall grades remain unverified because final resource counters are unavailable. Raw first grader failures and equivalent-representation regrades are preserved.
  • Five grader negative probes — reject wrong text unit, returned length, inclusive range end, containing-procedure display and load segment, without re-running models.
  • npm run test:focused -- tests/boundary/cli/nativeEfficiencyComparison.test.ts — sixteen cases pass; final focused check 2.9 s, sampled family peak about 215 MiB. The old producer reproduces the false fact differences; the duplicate-ID recording port is also rejected; corrected recorded PE/DOS and jq replays preserve facts/references/order. All three original real transcripts were regraded with the observed-source checks and preserve selected facts/lifecycle outcomes; resource final-counter uncertainty remains unchanged. No providers or models rerun for these review fixes.
  • node --check, Python py_compile, formatting and git diff --check — pass.
  • npm run check:fast — both tasks pass; cold gate 112.6 s, sampled peak about 3 GiB with memory.high events, explicitly excluded from performance claims. Initial pre-push gate then passed from cache. After adding the focused regression, the required pre-push type/lint gate passed cold in 113.1 s and the identity follow-up in 110.2 s DOS-source follow-up in 110.1 s, digest follow-up in 115.1 s and source-binding follow-up in 109.5 s; its memory.high events remain excluded from performance claims.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real provider scope: actual Ghidra 12.1.4 Linux; two specific shipped artifacts, no Hopper/IDA or Windows runtime claim.
  • Package/release metadata impact: none; no release or deployment.
  • Containment review: original digests verified before/after; session close and profile removal confirmed. SDK/model last-observed event counters are zero but final counters are unknown, never substituted with zeros.
  • Known REA issue: advertised native summary is rejected. Both SDK and PE agents reproduced it; logged in the requested /tmp handoff for team repair. This PR does not fix it.

Review checklist

  • This PR addresses a concrete agreed investigation.
  • One focused outcome and Conventional Commit title.
  • Related issue linked and remains open.
  • Relevant observable behavior and meaningful grading failures checked.
  • Owning experiment docs/inputs updated; generated metadata not affected.
  • Representative public workflow observations recorded; product output unchanged.
  • Final diff checked for credentials, unrelated cleanup and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2216 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/12/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/native-view-verification-20261012` --- ### 📝 Commits (7) - [`6d7f688`](https://github.com/morluto/rea/commit/6d7f6885c41b19cb7c6d8b0792ebef7de56b670d) test(native): add real PE and DOS entry view experiments - [`b74b9b7`](https://github.com/morluto/rea/commit/b74b9b79ab6d00f9efe77eba6a8ef6263373866b) docs(native): record real PE and DOS delivery costs - [`ba5cf12`](https://github.com/morluto/rea/commit/ba5cf12bfa964fdc40158c224c1ddba2e20a6175) fix(experiments): separate native facts from Evidence references - [`0e92a5d`](https://github.com/morluto/rea/commit/0e92a5da2804f85b461396031075da6b09abf8d7) fix(experiments): reject reused native view identities - [`4e149fb`](https://github.com/morluto/rea/commit/4e149fbee15b533435f0727480206bf68d9ccab7) fix(experiments): retain DOS load-image source identity - [`aada535`](https://github.com/morluto/rea/commit/aada535deabfca3ad42f276295cd3b16b36ae5fe) fix(experiments): verify DOS load-image target identity - [`0667f7c`](https://github.com/morluto/rea/commit/0667f7cb7c85f134311c7470ff4ee99ef9cc45f4) fix(experiments): bind native claims to observed source evidence ### 📊 Changes **10 files changed** (+920 additions, -124 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+4 -0) 📝 `experiments/analysis-efficiency/compare.mjs` (+15 -1) 📝 `experiments/analysis-efficiency/corpus.json` (+30 -0) 📝 `experiments/analysis-efficiency/measure.mjs` (+32 -116) ➕ `experiments/analysis-efficiency/native-entry-views.md` (+178 -0) ➕ `experiments/analysis-efficiency/native-entry.mjs` (+272 -0) ➕ `experiments/analysis-efficiency/questions/native-dos.txt` (+1 -0) ➕ `experiments/analysis-efficiency/questions/native-pe.txt` (+1 -0) 📝 `experiments/analysis-efficiency/review-model.py` (+51 -7) ➕ `tests/boundary/cli/nativeEfficiencyComparison.test.ts` (+336 -0) </details> ### 📄 Description ## Summary Native paged delivery needs real-artifact evidence before expanding the WASM summary approach. Record public ShareX native PE and FreeDOS DOS MZ entry workflows, including a small PE function where four retained facets cost **more** bytes than its complete dossier. Continues #2016; the broader issue stays open. ## Problem and expected behavior A smaller individual reply does not establish a cheaper complete investigation. Verify identical source facts and identities through actual Ghidra/MCP and real Codex CLI, retain failed calls and unknowns, and measure the complete question before recommending another API change. ## Change and scope Add two digest-pinned shipped inputs and question templates, extract the existing jq native experiment into a shared native helper, and add an explicit `entry-views` verification mode. In that new mode, keep selected facts separate from original Evidence reference maps at the producing boundary; preserve both runs' provenance in comparison reports. Preserve the existing jq entry/main call sequence. Extend the grader to compare exact values for equivalent native answer representations and reject changed units/ranges/identity/load metadata. Curated findings: [native entry-view checkpoint](https://github.com/morluto/rea/blob/0667f7cb7/experiments/analysis-efficiency/native-entry-views.md). Detailed JSON, binaries, model transcripts and credentials remain outside Git. ## Contract and boundary impact - Semantic owner and earliest changed stage: experiment driver and its answer grader; no production semantics change. - CLI and MCP/tool-catalog contract: none. Existing tool names, accepted inputs, complete defaults and schemas are unchanged. - Provider, bridge, target-format, or platform compatibility: no provider code changes. Actual Ghidra 12.1.4 on Linux inspects shipped Windows PE and original compressed DOS MZ. - Evidence, artifact, provenance, or reconstruction contract: no production changes. Facets are checked against original producer facts, target digest, parent ID and distinct view IDs. Unknown/static/packed-target limitations remain explicit. - Process execution, authorization, cleanup, or containment impact: existing serial one-CPU/4 GiB hard memory scopes; default auto-analysis, ephemeral session close, unchanged target bytes, copied-profile cleanup. No target execution. - Generated metadata, package, or installation impact: none; no generated outputs or release metadata committed. ## Evidence and regression coverage - Tests added or updated: reproducible experimental native workflows and exact-value grading; sixteen focused CLI producer/comparator regression cases. Different run-specific references compare successfully, while item/coverage/limitations/unknown differences remain failures; reused parent/facet/summary IDs are rejected, and the DOS load-image source is retained for its metadata and derived entry seed. A stale verified load image is rejected before procedure lookup and the session is closed. Completed DOS load-image observations and their reported source IDs are required for a load-image grade; missing/failed/unreferenced sources or wrong observed target/entry are rejected. These negative cases are synthetic grader fixtures, not real-provider/resource measurements. No production behavior change. - Executed evidence: both public SDK workflows; three actual Codex CLI trials (`gpt-5.6-terra`, high; PE twice, DOS once). Selected facts/references/lifecycle checks match original observations. - Existing driver regression: replay actual recorded jq entry and entry/main replies, preserving selected facts/profile and 4/12-call ordering. Replay actual PE/DOS replies against the committed extracted helper with exact answer/9/10-call agreement. These are replays, not fresh provider runs. - User-visible CLI/MCP output: unchanged; experimental notes include real response sizes and complete task costs. - Remaining proof gaps: Linux static inspection is not Windows runtime/OS coverage; compressed DOS entry analysis is not editor logic or COM coverage; final cgroup closeout counters are unverified. No overall native speedup or resource-qualified performance win is claimed. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - Public `measure.mjs --native-question entry-views --schemas selected` — PE/DOS exact selected-fact and Evidence checks pass; native summary refusal recorded separately. - `agent-trial.py` + `review-model.py --domain native` — three real model answers match selected facts and cleanup; strict overall grades remain unverified because final resource counters are unavailable. Raw first grader failures and equivalent-representation regrades are preserved. - Five grader negative probes — reject wrong text unit, returned length, inclusive range end, containing-procedure display and load segment, without re-running models. - `npm run test:focused -- tests/boundary/cli/nativeEfficiencyComparison.test.ts` — sixteen cases pass; final focused check 2.9 s, sampled family peak about 215 MiB. The old producer reproduces the false fact differences; the duplicate-ID recording port is also rejected; corrected recorded PE/DOS and jq replays preserve facts/references/order. All three original real transcripts were regraded with the observed-source checks and preserve selected facts/lifecycle outcomes; resource final-counter uncertainty remains unchanged. No providers or models rerun for these review fixes. - `node --check`, Python `py_compile`, formatting and `git diff --check` — pass. - `npm run check:fast` — both tasks pass; cold gate 112.6 s, sampled peak about 3 GiB with `memory.high` events, explicitly excluded from performance claims. Initial pre-push gate then passed from cache. After adding the focused regression, the required pre-push type/lint gate passed cold in 113.1 s and the identity follow-up in 110.2 s DOS-source follow-up in 110.1 s, digest follow-up in 115.1 s and source-binding follow-up in 109.5 s; its `memory.high` events remain excluded from performance claims. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real provider scope: actual Ghidra 12.1.4 Linux; two specific shipped artifacts, no Hopper/IDA or Windows runtime claim. - Package/release metadata impact: none; no release or deployment. - Containment review: original digests verified before/after; session close and profile removal confirmed. SDK/model last-observed event counters are zero but final counters are unknown, never substituted with zeros. - Known REA issue: advertised native summary is rejected. Both SDK and PE agents reproduced it; logged in the requested `/tmp` handoff for team repair. This PR does not fix it. ## Review checklist - [x] This PR addresses a concrete agreed investigation. - [x] One focused outcome and Conventional Commit title. - [x] Related issue linked and remains open. - [x] Relevant observable behavior and meaningful grading failures checked. - [x] Owning experiment docs/inputs updated; generated metadata not affected. - [x] Representative public workflow observations recorded; product output unchanged. - [x] Final diff checked for credentials, unrelated cleanup and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:03:47 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5090
No description provided.