[GH-ISSUE #5418] [Bug] Process capture blames normalization for unchanged rendered-frame omissions #3284

New issue

Closed

opened 2026-10-31 00:38:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:38:46 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5418

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

capture-process reports that a normalization stage exceeded its budget when unchanged terminal text exhausts the independent cumulative rendered-byte budget, even with all normalization transformations disabled.

An independently reproduced CLI capture of 600 ASCII x characters with limits.output_bytes: 1024 retained all 600 raw bytes and the sole raw frame, with normalized_terminal.omitted_frames: 0, but omitted its rendered frame and returned the normalization-specific limitation quoted below. The CLI and target exited normally. Raising only the budget to 2048 retained the rendered frame at 1200 bytes and removed the limitation.

Reproduction

In the built source checkout identified below, with its installed dependencies and native PTY prerequisites, run this exact independently verified reproduction from the repository root. It invokes the supported scenario-file CLI in an isolated environment and prints the relevant result fields.

python3 <<'PYTHON'
import json
import os
from pathlib import Path
import shutil
import subprocess
import tempfile

repo = Path.cwd()
work = Path(tempfile.mkdtemp(prefix="rea-render-budget-"))
env = {"PATH": os.environ["PATH"], "LANG": "C.UTF-8"}
for key, directory in [
    ("HOME", "home"),
    ("XDG_CONFIG_HOME", "config"),
    ("XDG_DATA_HOME", "data"),
    ("XDG_CACHE_HOME", "cache"),
    ("XDG_STATE_HOME", "state"),
    ("TMPDIR", "temp"),
    ("REA_DARWIN_MODULE_CACHE_PATH", "module-cache"),
]:
    path = work / directory
    path.mkdir()
    env[key] = str(path)
config = work / "config" / "ida.json"
config.write_text("{}")
env["REA_IDA_MCP_CONFIG"] = str(config)
node = shutil.which("node")
scenario = {
    "executable": node,
    "arguments": ["-e", "process.stdout.write('x'.repeat(600))"],
    "working_directory": str(work),
    "normalization": {
        "paths": False,
        "pids": False,
        "ports": False,
        "patterns": [],
    },
    "limits": {"output_bytes": 1024},
}
scenario_path = work / "scenario.json"
scenario_path.write_text(json.dumps(scenario))
result = subprocess.run(
    [node, str(repo / "scripts/rea.mjs"), "capture-process",
     str(scenario_path), "--json"],
    cwd=work, env=env, capture_output=True, text=True, timeout=60,
)
(work / "capture.json").write_text(result.stdout)
(work / "stderr.log").write_text(result.stderr)
result.check_returncode()
capture = json.loads(result.stdout)["normalized_result"]
print(json.dumps({
    "raw_terminal": capture["truncation_details"]["raw_terminal"],
    "normalized_terminal": capture["truncation_details"]["normalized_terminal"],
    "rendered_terminal": capture["truncation_details"]["rendered_terminal"],
    "rendered_limitations": [item for item in capture["limitations"]
                             if item.startswith("Rendered frames")],
}, indent=2))
PYTHON

The independent verifier also ran the same unchanged payload with default normalization at both budgets and explicitly disabled transformations at both budgets. Both 1024-byte cases showed the false cause; both 2048-byte controls retained the rendered frame. These are local CLI observations, not MCP or external-provider tests.

Expected behavior

The omission reason should identify cumulative rendered representation bytes exceeding output_bytes when no normalization transformation expands the text. The frame should still be omitted under its independent budget, and incomplete rendered coverage should remain explicit.

docs/process-capture.md:409-416 defines the rendered budget as cumulative serialized-state plus visible-line bytes, independent of raw PTY bytes. src/process/capture/ProcessCaptureLifecycle.ts:435-443 already distinguishes ordinary cumulative-byte exhaustion from normalization-specific omissions. Unknown complete frame sizes may remain unknown; partial row sizes need not be counted as complete frame sizes.

Impact and scope

P3: the observed defect is misleading cause wording for a real rendered omission. Raw coverage stays complete in the reduced case, and rendered coverage correctly reports the omitted frame. No incorrect comparison decision, target exit, or report digest was demonstrated. Raising the reduced input's budget from 1024 to 2048 is a verified workaround. Frequency outside these inputs and other platforms was not measured.

REA version

6.7.0; source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8.

Runtime and platform

Node.js v26.12.0; macOS 27.0.1 arm64; native PTY available.

Target and artifact context

Harmless local Node command printing 600 ASCII x characters without a newline; default terminal dimensions; returned process-capture Evidence, particularly truncation_details and limitations. No external analysis provider was used.

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

The disabled-transformation 1024-byte CLI capture returned this rendered_terminal coverage:

{
  "budget_bytes": 1024,
  "observed_bytes": 0,
  "retained_bytes": 0,
  "observed_frames": 1,
  "retained_frames": 0,
  "unknown_size_frames": 1
}

Its limitation was:

Rendered frames were omitted because a normalization stage exceeded the remaining output_bytes budget; final rendered byte totals are unknown for those frames. See truncation_details.rendered_terminal.

At 2048 bytes, the same unchanged payload returned one retained rendered frame with observed_bytes and retained_bytes both 1200 and no rendered omission message. This supports ordinary representation size as the trigger.

A separately executed genuine-expansion control printed one x, disabled the built-in normalization rules, and used a literal pattern replacing each space with 2000 Y characters. At the 1024-byte budget it retained its one-byte raw frame, omitted the rendered frame, and produced the normalization-specific message because padded rows actually expanded.

Typed outcome or recovery expected

Keep complete raw retention and zero normalized raw omissions for the reduced input. Omit its rendered frame under the 1024-byte budget with an accurate cumulative rendered-budget cause. Preserve honest unknown-size accounting and the distinction from genuine normalization replacement overruns. The 2048-byte control must still retain its 1200-byte rendered frame.

Analysis or suspected cause

Source-derived explanation at the reported baseline: the final unchanged-text budget check in src/process/capture/ProcessNormalization.ts:271-277 returns over_budget. In this reduced case visible rows fit, then serialized-state normalization exceeds the remaining rendered budget. src/process/capture/TerminalRenderer.ts:162-169 records an unknown-size omission, and src/process/capture/ProcessCaptureLifecycle.ts:441-443 selects the normalization-specific cause. This reproduction does not establish the separate visible-line comparison branch as the cause.

Additional context

Prepared with AI assistance and independently reproduced through the public CLI, including replay of the exact script above. Submission freshness check: upstream main 1417bf943bf49e49b414e8a508bf28a31304be83 has no changes from the tested baseline in the cited contract, coverage, lifecycle, normalizer, or renderer files; the full runtime reproduction was not rerun for submission. 6.7.0 remains the latest published release checked.

Related #1250 / #1284 introduced per-dimension truncation diagnostics; #3168 concerned unbounded expanded normalized chunk retention. This report concerns the current false omission cause for unchanged rendered content. Last known good version: NOT_ENOUGH_INFO; no regression boundary was established.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5418 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior `capture-process` reports that a normalization stage exceeded its budget when unchanged terminal text exhausts the independent cumulative rendered-byte budget, even with all normalization transformations disabled. An independently reproduced CLI capture of 600 ASCII `x` characters with `limits.output_bytes: 1024` retained all 600 raw bytes and the sole raw frame, with `normalized_terminal.omitted_frames: 0`, but omitted its rendered frame and returned the normalization-specific limitation quoted below. The CLI and target exited normally. Raising only the budget to 2048 retained the rendered frame at 1200 bytes and removed the limitation. ### Reproduction In the built source checkout identified below, with its installed dependencies and native PTY prerequisites, run this exact independently verified reproduction from the repository root. It invokes the supported scenario-file CLI in an isolated environment and prints the relevant result fields. ```sh python3 <<'PYTHON' import json import os from pathlib import Path import shutil import subprocess import tempfile repo = Path.cwd() work = Path(tempfile.mkdtemp(prefix="rea-render-budget-")) env = {"PATH": os.environ["PATH"], "LANG": "C.UTF-8"} for key, directory in [ ("HOME", "home"), ("XDG_CONFIG_HOME", "config"), ("XDG_DATA_HOME", "data"), ("XDG_CACHE_HOME", "cache"), ("XDG_STATE_HOME", "state"), ("TMPDIR", "temp"), ("REA_DARWIN_MODULE_CACHE_PATH", "module-cache"), ]: path = work / directory path.mkdir() env[key] = str(path) config = work / "config" / "ida.json" config.write_text("{}") env["REA_IDA_MCP_CONFIG"] = str(config) node = shutil.which("node") scenario = { "executable": node, "arguments": ["-e", "process.stdout.write('x'.repeat(600))"], "working_directory": str(work), "normalization": { "paths": False, "pids": False, "ports": False, "patterns": [], }, "limits": {"output_bytes": 1024}, } scenario_path = work / "scenario.json" scenario_path.write_text(json.dumps(scenario)) result = subprocess.run( [node, str(repo / "scripts/rea.mjs"), "capture-process", str(scenario_path), "--json"], cwd=work, env=env, capture_output=True, text=True, timeout=60, ) (work / "capture.json").write_text(result.stdout) (work / "stderr.log").write_text(result.stderr) result.check_returncode() capture = json.loads(result.stdout)["normalized_result"] print(json.dumps({ "raw_terminal": capture["truncation_details"]["raw_terminal"], "normalized_terminal": capture["truncation_details"]["normalized_terminal"], "rendered_terminal": capture["truncation_details"]["rendered_terminal"], "rendered_limitations": [item for item in capture["limitations"] if item.startswith("Rendered frames")], }, indent=2)) PYTHON ``` The independent verifier also ran the same unchanged payload with default normalization at both budgets and explicitly disabled transformations at both budgets. Both 1024-byte cases showed the false cause; both 2048-byte controls retained the rendered frame. These are local CLI observations, not MCP or external-provider tests. ### Expected behavior The omission reason should identify cumulative rendered representation bytes exceeding `output_bytes` when no normalization transformation expands the text. The frame should still be omitted under its independent budget, and incomplete rendered coverage should remain explicit. `docs/process-capture.md:409-416` defines the rendered budget as cumulative serialized-state plus visible-line bytes, independent of raw PTY bytes. `src/process/capture/ProcessCaptureLifecycle.ts:435-443` already distinguishes ordinary cumulative-byte exhaustion from normalization-specific omissions. Unknown complete frame sizes may remain unknown; partial row sizes need not be counted as complete frame sizes. ### Impact and scope P3: the observed defect is misleading cause wording for a real rendered omission. Raw coverage stays complete in the reduced case, and rendered coverage correctly reports the omitted frame. No incorrect comparison decision, target exit, or report digest was demonstrated. Raising the reduced input's budget from 1024 to 2048 is a verified workaround. Frequency outside these inputs and other platforms was not measured. ### REA version 6.7.0; source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`. ### Runtime and platform Node.js v26.12.0; macOS 27.0.1 arm64; native PTY available. ### Target and artifact context Harmless local Node command printing 600 ASCII `x` characters without a newline; default terminal dimensions; returned process-capture Evidence, particularly `truncation_details` and `limitations`. No external analysis provider was used. ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence The disabled-transformation 1024-byte CLI capture returned this `rendered_terminal` coverage: ```json { "budget_bytes": 1024, "observed_bytes": 0, "retained_bytes": 0, "observed_frames": 1, "retained_frames": 0, "unknown_size_frames": 1 } ``` Its limitation was: ```text Rendered frames were omitted because a normalization stage exceeded the remaining output_bytes budget; final rendered byte totals are unknown for those frames. See truncation_details.rendered_terminal. ``` At 2048 bytes, the same unchanged payload returned one retained rendered frame with `observed_bytes` and `retained_bytes` both 1200 and no rendered omission message. This supports ordinary representation size as the trigger. A separately executed genuine-expansion control printed one `x`, disabled the built-in normalization rules, and used a literal pattern replacing each space with 2000 `Y` characters. At the 1024-byte budget it retained its one-byte raw frame, omitted the rendered frame, and produced the normalization-specific message because padded rows actually expanded. ### Typed outcome or recovery expected Keep complete raw retention and zero normalized raw omissions for the reduced input. Omit its rendered frame under the 1024-byte budget with an accurate cumulative rendered-budget cause. Preserve honest unknown-size accounting and the distinction from genuine normalization replacement overruns. The 2048-byte control must still retain its 1200-byte rendered frame. ### Analysis or suspected cause Source-derived explanation at the reported baseline: the final unchanged-text budget check in `src/process/capture/ProcessNormalization.ts:271-277` returns `over_budget`. In this reduced case visible rows fit, then serialized-state normalization exceeds the remaining rendered budget. `src/process/capture/TerminalRenderer.ts:162-169` records an unknown-size omission, and `src/process/capture/ProcessCaptureLifecycle.ts:441-443` selects the normalization-specific cause. This reproduction does not establish the separate visible-line comparison branch as the cause. ### Additional context Prepared with AI assistance and independently reproduced through the public CLI, including replay of the exact script above. Submission freshness check: upstream main `1417bf943bf49e49b414e8a508bf28a31304be83` has no changes from the tested baseline in the cited contract, coverage, lifecycle, normalizer, or renderer files; the full runtime reproduction was not rerun for submission. 6.7.0 remains the latest published release checked. Related #1250 / #1284 introduced per-dimension truncation diagnostics; #3168 concerned unbounded expanded normalized chunk retention. This report concerns the current false omission cause for unchanged rendered content. Last known good version: NOT_ENOUGH_INFO; no regression boundary was established.

bsakaguchi closed this issue

2026-10-31 00:38:47 +00:00

bsakaguchi referenced this issue

2026-10-31 01:08:44 +00:00

[PR #3284] [CLOSED] fix(native): exclude synthetic entry references from incoming references #5697