morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3330] [Bug] Cutter discards mixed Unicode output that fits the frame limit #1199

New issue

Closed

opened 2026-10-15 18:31:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:31:56 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3330

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

cutter-command discards a completed command's entire output when a 5 MiB ASCII string has its last character replaced with é. The authenticated response would occupy only 5,243,164 bytes, below the 16 MiB frame limit, but the CLI returns output: null and says the result exceeded that limit. The unchanged ASCII control succeeds. This loses usable command output from both the normalized result and raw Evidence.

All four public CLI calls in the reproduction exited 0. The mixed case reports execution_state: "complete", output_truncated: true, and raw_result: null. Small Unicode output succeeds, so the observed failure depends on size and content rather than rejecting Unicode generally. The bridge stopped and removed its descriptor after the probe.

Reproduction

Start from a clean source checkout and prepare the locked dependencies and runtime from the repository root:

git clone https://github.com/morluto/rea.git rea-cutter-repro
cd rea-cutter-repro
git checkout 8dde6370d0809ad04cda3342d4e6f290f48aa063
npm ci
npm run build:cached

The observed reproduction used the prepared runtime and installed dependencies; the installation and build commands above describe preparation for a clean checkout.

Run the following from the repository root. Python 3 supplies a minimal Cutter/Qt host stand-in. The production Python bridge, dispatcher, authenticated transport, and compiled CLI remain in use. Every input is generated inline. No Cutter installation or target execution is required.

export REA_PROBE_DIR="$(mktemp -d)"
PYTHONDONTWRITEBYTECODE=1 python3 - <<'PY'
import importlib.util
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
import types

probe = Path(os.environ['REA_PROBE_DIR'])
(probe / 'home').mkdir()
os.environ['HOME'] = str(probe / 'home')
os.environ['REA_CUTTER_BRIDGE_DIR'] = str(probe / 'bridge')

class Signal:
    def __init__(self, *args):
        pass
    def connect(self, callback):
        self.callback = callback
    def emit(self, *args):
        self.callback(*args)

class Core:
    def currentlyOpenFile(self):
        return 'sample.bin'
    def getVersionInformation(self):
        return 'simulated-Cutter'

outputs = {
    'ascii': 'a' * (5 * 1024 * 1024),
    'mixed': 'a' * (5 * 1024 * 1024 - 1) + 'é',
    'small': 'a' * 100 + 'é',
    'oversized': 'a' * (17 * 1024 * 1024),
}
cutter = types.ModuleType('cutter')
cutter.CutterPlugin = object
cutter.core = lambda: Core()
cutter.cmd = lambda command: outputs[command]
sys.modules['cutter'] = cutter
qtcore = types.ModuleType('PySide6.QtCore')
qtcore.QObject = object
qtcore.Signal = Signal
sys.modules['PySide6'] = types.ModuleType('PySide6')
sys.modules['PySide6.QtCore'] = qtcore
spec = importlib.util.spec_from_file_location(
    'bridge', Path('bridge/cutter/rea_bridge.py'))
bridge = importlib.util.module_from_spec(spec)
spec.loader.exec_module(bridge)
server = bridge._BridgeServer(bridge._RequestDispatcher())
server.start()
try:
    deadline = time.monotonic() + 5
    while not server.descriptor.exists() and time.monotonic() < deadline:
        time.sleep(0.005)
    assert server.descriptor.exists(), 'Bridge did not publish its descriptor'
    for command, output in outputs.items():
        response = {
            'ok': True, 'output': output, 'current_file': 'sample.bin',
            'document_generation': 0, 'cutter_version': 'simulated-Cutter',
            'identity_status': 'partial', 'execution_state': 'complete',
        }
        text = json.dumps(response, ensure_ascii=False, separators=(',', ':'))
        frame = json.dumps({'response': text, 'proof': '0' * 64},
                           ensure_ascii=False, separators=(',', ':')).encode() + b'\n'
        completed = subprocess.run(
            ['node', 'scripts/rea.mjs', 'cutter-command', server.session_id,
             '0', command, '--format', 'json'],
            env=os.environ, capture_output=True, text=True, timeout=20)
        assert completed.returncode == 0, completed.stderr
        evidence = json.loads(completed.stdout)
        result = evidence['normalized_result']
        print(json.dumps({
            'command': command,
            'authenticated_frame_bytes': len(frame),
            'frame_limit': bridge.MAX_FRAME_BYTES,
            'output_characters': len(result['output'] or ''),
            'output_truncated': result['output_truncated'],
            'execution_state': result['execution_state'],
            'message': result['message'],
            'raw_output_is_null': evidence['raw_result'] is None,
        }, ensure_ascii=False))
finally:
    server.stop()
    server.join(3)
    assert not server.is_alive(), 'Bridge did not stop'
    assert not server.descriptor.exists(), 'Bridge descriptor remains'
    shutil.rmtree(probe)
PY

Compare the frame sizes and returned output lengths for the ASCII, mixed Unicode, small Unicode, and oversized cases. The frame measurement includes JSON escaping, the 64-character proof, and the terminating newline used by the authenticated response format. The proof's value does not affect its encoded length. The probe stops the bridge, checks that its descriptor was removed, and removes its generated directory.

Expected behavior

Preserve the ascii, mixed, and small results because their encoded frames fit within 16 MiB. The mixed case should return all 5,242,880 characters, including the last é, with execution_state: "complete" and output_truncated: false. A response actually exceeding the frame limit should retain the existing explicit truncation behavior.

AGENTS.md:5 requires preservation of Evidence provenance and session semantics through CLI and MCP. The Cutter bridge section at docs/reverse-engineering-tools.md:60 describes the local authenticated transport, and docs/reverse-engineering-tools.md:65 identifies its 16 MiB encoded frame limit. docs/tool-design.md:51 requires requested facts inline, docs/tool-design.md:54 requires access to complete source Evidence, and docs/tool-design.md:67 grounds limits in actual format, protocol, authority, or measured resource constraints.

Impact and scope

P2 impact is supported. A valid completed command's medium-sized result is discarded despite fitting the framed transport budget. Only bounded size/content combinations are affected; ASCII and small Unicode controls remain usable. Both normalized output and raw Evidence lose the command output.

Workaround status is unverified. Smaller results pass, but no general way to split and faithfully reassemble arbitrary Cutter command output was verified. Raising an MCP response budget would not address the producer's earlier omission.

REA version

6.4.0 (main at 8dde6370d0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1; arm64; Python 3. The reproduction used the source checkout's prebuilt runtime and Python bridge with a simulated Cutter/Qt host. A Python patch version was not recorded.

Target and artifact context

Simulated Cutter command outputs named ascii, mixed, small, and oversized contain respectively 5 MiB of ASCII, the same character count with a final é, 100 ASCII characters plus é, and 17 MiB of ASCII. The host reports sample.bin, simulated-Cutter, document generation 0, and partial identity. No actual binary content is read or executed.

The production bridge and compiled CLI were executed with simulated Cutter output. This does not establish behavior of a particular installed Cutter/Rizin release.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The observed CLI results were:

command    authenticated frame bytes   returned characters   output_truncated
ascii      5243163                     5242880               false
mixed      5243164                     0                     true
small      385                         101                   false
oversized  17826075                    0                     true

mixed: execution_state="complete", output=null, raw_result=null
mixed message: "The command result exceeded the bridge frame limit; completion state is preserved and output was omitted."

Typed outcome or recovery expected

Repeat the public CLI calls above. Both 5 MiB controls must return their exact original strings in normalized output and raw Evidence with no truncation. The small Unicode case must continue to succeed. The 17 MiB control must remain explicitly truncated, and the authenticated frame limit must remain enforced. Verify that bridge shutdown still removes the descriptor.

Analysis or suspected cause

Source-derived reasoning: bridge/cutter/rea_bridge.py:777 defines the response-limit precheck. Its non-ASCII branch at bridge/cutter/rea_bridge.py:790 estimates each non-ASCII string as four bytes per character, including that string's ASCII characters. A single final é therefore changes the estimate for the entire 5 MiB string.

bridge/cutter/rea_bridge.py:666 defines authenticated response encoding and rejects that estimate before measuring the actual encoded response. The encoded-size check remains in bridge/cutter/rea_bridge.py:678. bridge/cutter/rea_bridge.py:643 substitutes the output-omitting response when encoding returns no frame. This source-derived explanation is consistent with the observed successful ASCII and small-Unicode controls and the failing mixed case.

Additional context

Related open issue #3182 concerns bare --json leaving command-specific JSON disabled; this report concerns loss of completed text output during Python response-size admission.

Related open issue #3183 concerns bridge-directory resolution hiding live sessions; this report reaches the running bridge and completes its commands.

Related completed issue #3168 concerns expanding process-capture normalization exceeding its output budget; this report concerns in-budget Cutter output being omitted.

Related completed issue #1053 concerns huge full-ASAR MCP result serialization; this report loses a roughly 5 MiB Cutter result before MCP delivery and is reproduced through CLI JSON.

Related completed issue #3221 concerns oversized MCP validation diagnostics bypassing admission and retention; this report concerns a valid completed command whose actual response fits the producer's frame limit.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3330 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `cutter-command` discards a completed command's entire output when a 5 MiB ASCII string has its last character replaced with `é`. The authenticated response would occupy only 5,243,164 bytes, below the 16 MiB frame limit, but the CLI returns `output: null` and says the result exceeded that limit. The unchanged ASCII control succeeds. This loses usable command output from both the normalized result and raw Evidence. All four public CLI calls in the reproduction exited 0. The mixed case reports `execution_state: "complete"`, `output_truncated: true`, and `raw_result: null`. Small Unicode output succeeds, so the observed failure depends on size and content rather than rejecting Unicode generally. The bridge stopped and removed its descriptor after the probe. ### Reproduction 1. Start from a clean source checkout and prepare the locked dependencies and runtime from the repository root: ```sh git clone https://github.com/morluto/rea.git rea-cutter-repro cd rea-cutter-repro git checkout 8dde6370d0809ad04cda3342d4e6f290f48aa063 npm ci npm run build:cached ``` The observed reproduction used the prepared runtime and installed dependencies; the installation and build commands above describe preparation for a clean checkout. 2. Run the following from the repository root. Python 3 supplies a minimal Cutter/Qt host stand-in. The production Python bridge, dispatcher, authenticated transport, and compiled CLI remain in use. Every input is generated inline. No Cutter installation or target execution is required. ```sh export REA_PROBE_DIR="$(mktemp -d)" PYTHONDONTWRITEBYTECODE=1 python3 - <<'PY' import importlib.util import json import os from pathlib import Path import shutil import subprocess import sys import time import types probe = Path(os.environ['REA_PROBE_DIR']) (probe / 'home').mkdir() os.environ['HOME'] = str(probe / 'home') os.environ['REA_CUTTER_BRIDGE_DIR'] = str(probe / 'bridge') class Signal: def __init__(self, *args): pass def connect(self, callback): self.callback = callback def emit(self, *args): self.callback(*args) class Core: def currentlyOpenFile(self): return 'sample.bin' def getVersionInformation(self): return 'simulated-Cutter' outputs = { 'ascii': 'a' * (5 * 1024 * 1024), 'mixed': 'a' * (5 * 1024 * 1024 - 1) + 'é', 'small': 'a' * 100 + 'é', 'oversized': 'a' * (17 * 1024 * 1024), } cutter = types.ModuleType('cutter') cutter.CutterPlugin = object cutter.core = lambda: Core() cutter.cmd = lambda command: outputs[command] sys.modules['cutter'] = cutter qtcore = types.ModuleType('PySide6.QtCore') qtcore.QObject = object qtcore.Signal = Signal sys.modules['PySide6'] = types.ModuleType('PySide6') sys.modules['PySide6.QtCore'] = qtcore spec = importlib.util.spec_from_file_location( 'bridge', Path('bridge/cutter/rea_bridge.py')) bridge = importlib.util.module_from_spec(spec) spec.loader.exec_module(bridge) server = bridge._BridgeServer(bridge._RequestDispatcher()) server.start() try: deadline = time.monotonic() + 5 while not server.descriptor.exists() and time.monotonic() < deadline: time.sleep(0.005) assert server.descriptor.exists(), 'Bridge did not publish its descriptor' for command, output in outputs.items(): response = { 'ok': True, 'output': output, 'current_file': 'sample.bin', 'document_generation': 0, 'cutter_version': 'simulated-Cutter', 'identity_status': 'partial', 'execution_state': 'complete', } text = json.dumps(response, ensure_ascii=False, separators=(',', ':')) frame = json.dumps({'response': text, 'proof': '0' * 64}, ensure_ascii=False, separators=(',', ':')).encode() + b'\n' completed = subprocess.run( ['node', 'scripts/rea.mjs', 'cutter-command', server.session_id, '0', command, '--format', 'json'], env=os.environ, capture_output=True, text=True, timeout=20) assert completed.returncode == 0, completed.stderr evidence = json.loads(completed.stdout) result = evidence['normalized_result'] print(json.dumps({ 'command': command, 'authenticated_frame_bytes': len(frame), 'frame_limit': bridge.MAX_FRAME_BYTES, 'output_characters': len(result['output'] or ''), 'output_truncated': result['output_truncated'], 'execution_state': result['execution_state'], 'message': result['message'], 'raw_output_is_null': evidence['raw_result'] is None, }, ensure_ascii=False)) finally: server.stop() server.join(3) assert not server.is_alive(), 'Bridge did not stop' assert not server.descriptor.exists(), 'Bridge descriptor remains' shutil.rmtree(probe) PY ``` 3. Compare the frame sizes and returned output lengths for the ASCII, mixed Unicode, small Unicode, and oversized cases. The frame measurement includes JSON escaping, the 64-character proof, and the terminating newline used by the authenticated response format. The proof's value does not affect its encoded length. The probe stops the bridge, checks that its descriptor was removed, and removes its generated directory. ### Expected behavior Preserve the `ascii`, `mixed`, and `small` results because their encoded frames fit within 16 MiB. The `mixed` case should return all 5,242,880 characters, including the last `é`, with `execution_state: "complete"` and `output_truncated: false`. A response actually exceeding the frame limit should retain the existing explicit truncation behavior. `AGENTS.md:5` requires preservation of Evidence provenance and session semantics through CLI and MCP. The Cutter bridge section at `docs/reverse-engineering-tools.md:60` describes the local authenticated transport, and `docs/reverse-engineering-tools.md:65` identifies its 16 MiB encoded frame limit. `docs/tool-design.md:51` requires requested facts inline, `docs/tool-design.md:54` requires access to complete source Evidence, and `docs/tool-design.md:67` grounds limits in actual format, protocol, authority, or measured resource constraints. ### Impact and scope P2 impact is supported. A valid completed command's medium-sized result is discarded despite fitting the framed transport budget. Only bounded size/content combinations are affected; ASCII and small Unicode controls remain usable. Both normalized output and raw Evidence lose the command output. Workaround status is unverified. Smaller results pass, but no general way to split and faithfully reassemble arbitrary Cutter command output was verified. Raising an MCP response budget would not address the producer's earlier omission. ### REA version 6.4.0 (main at 8dde6370d0809ad04cda3342d4e6f290f48aa063) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1; arm64; Python 3. The reproduction used the source checkout's prebuilt runtime and Python bridge with a simulated Cutter/Qt host. A Python patch version was not recorded. ### Target and artifact context Simulated Cutter command outputs named `ascii`, `mixed`, `small`, and `oversized` contain respectively 5 MiB of ASCII, the same character count with a final `é`, 100 ASCII characters plus `é`, and 17 MiB of ASCII. The host reports `sample.bin`, `simulated-Cutter`, document generation 0, and partial identity. No actual binary content is read or executed. The production bridge and compiled CLI were executed with simulated Cutter output. This does not establish behavior of a particular installed Cutter/Rizin release. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The observed CLI results were: ```text command authenticated frame bytes returned characters output_truncated ascii 5243163 5242880 false mixed 5243164 0 true small 385 101 false oversized 17826075 0 true mixed: execution_state="complete", output=null, raw_result=null mixed message: "The command result exceeded the bridge frame limit; completion state is preserved and output was omitted." ``` ### Typed outcome or recovery expected Repeat the public CLI calls above. Both 5 MiB controls must return their exact original strings in normalized output and raw Evidence with no truncation. The small Unicode case must continue to succeed. The 17 MiB control must remain explicitly truncated, and the authenticated frame limit must remain enforced. Verify that bridge shutdown still removes the descriptor. ### Analysis or suspected cause Source-derived reasoning: `bridge/cutter/rea_bridge.py:777` defines the response-limit precheck. Its non-ASCII branch at `bridge/cutter/rea_bridge.py:790` estimates each non-ASCII string as four bytes per character, including that string's ASCII characters. A single final `é` therefore changes the estimate for the entire 5 MiB string. `bridge/cutter/rea_bridge.py:666` defines authenticated response encoding and rejects that estimate before measuring the actual encoded response. The encoded-size check remains in `bridge/cutter/rea_bridge.py:678`. `bridge/cutter/rea_bridge.py:643` substitutes the output-omitting response when encoding returns no frame. This source-derived explanation is consistent with the observed successful ASCII and small-Unicode controls and the failing mixed case. ### Additional context Related open issue [#3182](https://github.com/morluto/rea/issues/3182) concerns bare `--json` leaving command-specific JSON disabled; this report concerns loss of completed text output during Python response-size admission. Related open issue [#3183](https://github.com/morluto/rea/issues/3183) concerns bridge-directory resolution hiding live sessions; this report reaches the running bridge and completes its commands. Related completed issue [#3168](https://github.com/morluto/rea/issues/3168) concerns expanding process-capture normalization exceeding its output budget; this report concerns in-budget Cutter output being omitted. Related completed issue [#1053](https://github.com/morluto/rea/issues/1053) concerns huge full-ASAR MCP result serialization; this report loses a roughly 5 MiB Cutter result before MCP delivery and is reproduced through CLI JSON. Related completed issue [#3221](https://github.com/morluto/rea/issues/3221) concerns oversized MCP validation diagnostics bypassing admission and retention; this report concerns a valid completed command whose actual response fits the producer's frame limit.

bsakaguchi closed this issue

2026-10-17 21:01:41 +00:00

bsakaguchi referenced this issue

2026-10-17 21:09:22 +00:00

[PR #1199] [MERGED] fix(config): name the rejected setting and its constraint in configuration errors #2712

bsakaguchi referenced this issue

2026-10-17 21:09:23 +00:00

[PR #1200] [MERGED] fix(ghidra): reject relative installation and Java paths in doctor and setup #2713

bsakaguchi referenced this issue

2026-10-31 00:50:51 +00:00

[GH-ISSUE #7644] [Bug] Hopper guide mentions loader overrides without documenting their setting #4568