morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5091] [Bug] Rizin recent output retains three times its UTF-8 byte budget for CJK frames #3129

New issue

Closed

opened 2026-10-31 00:36:57 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:36:57 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5091

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

A mocked-provider MCP probe supplied one valid 60,000-byte UTF-8 NUL frame containing 20,000 copies of 雪. rizin_debug_session_status returned a 49,152-byte retained prefix plus the 24-byte [recent frame truncated] label. The implementation's recent-frame prefix budget is 16,384 bytes.

An ASCII control retained the expected 16,384-byte prefix. A separate valid Unicode fixture was cut between an emoji's surrogate halves, leaving an ill-formed history prefix. All three status responses reported recent_output_truncated: true. Full rizin_debug_command output remained exact and untruncated.

This executed the production manager, registered MCP handlers, schemas and linked in-memory transport with an injected process. No Rizin engine or target program ran.

Reproduction

In a checkout of source revision 3537e58c1ab29309f06eabf50bc52adba9817a0c with locked dependencies installed, save this as tests/boundary/mcp/rizinRecentHistoryProbe.test.ts:

import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { McpServer } from "@modelcontextprotocol/server";
import { expect, it } from "vitest";
import { registerRizinDebugTools } from "../../../src/server/registerRizinDebugTools.js";
import { createToolResultDelivery } from "../../../src/server/toolResult.js";
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { silentLogger } from "../../../src/logger.js";
import { createInjectedProcess } from "../providers/RizinDebugSessionCoverage.fixtures.js";
import { createRizinTestManager, rizinTestInferiorTty } from "../providers/RizinTestInferiorTty.js";

it.each([
  { label: "ascii", payload: "a".repeat(20000), retainedBytes: 16408, wellFormed: true },
  { label: "cjk", payload: "雪".repeat(20000), retainedBytes: 49176, wellFormed: true },
  { label: "surrogate", payload: "a".repeat(16383) + "😀" + "x".repeat(100), retainedBytes: 16410, wellFormed: false },
])("observes $label history through MCP", async ({label, payload, retainedBytes, wellFormed}) => {
  const injected = createInjectedProcess({onWrite: (text, control) => {
    const marker = /!echo (.+)\n/u.exec(text)?.[1];
    if (marker === undefined) return;
    setImmediate(() => control.output(`${payload}\0${marker}\n\0`));
  }});
  const manager = createRizinTestManager({}, {spawnProcess: injected.spawnProcess, frameTimeoutMs: 1000});
  const server = new McpServer({name: "history-probe", version: "1"});
  registerRizinDebugTools(server, manager, silentLogger, createToolResultDelivery(undefined));
  const client = new Client({name: "history-probe", version: "1"});
  const [ct, st] = InMemoryTransport.createLinkedPair();
  try {
    await server.connect(st);
    await client.connect(ct);
    const startReply = await client.callTool({name: "start_rizin_debug_session", arguments: {
      path: "fixture.bin", inferior_tty: await rizinTestInferiorTty(),
    }});
    expect(startReply.isError).not.toBe(true);
    const started = toolContract("start_rizin_debug_session").outputSchema.parse(startReply.structuredContent);
    const commandReply = await client.callTool({name: "rizin_debug_command", arguments: {
      session_id: started.session_id, command: "dr",
    }});
    expect(commandReply.isError).not.toBe(true);
    const evidence = toolContract("rizin_debug_command").outputSchema.parse(commandReply.structuredContent);
    expect(evidence.normalized_result.output).toBe(payload);
    expect(evidence.normalized_result.output_truncated).toBe(false);
    const statusReply = await client.callTool({name: "rizin_debug_session_status", arguments: {
      session_id: started.session_id,
    }});
    expect(statusReply.isError).not.toBe(true);
    const status = toolContract("rizin_debug_session_status").outputSchema.parse(statusReply.structuredContent);
    const retained = status.recent_output.find(x => x.endsWith("[recent frame truncated]"));
    expect(retained).toBeDefined();
    const prefix = retained!.slice(0, -"[recent frame truncated]".length);
    const observed = {label, input_bytes: Buffer.byteLength(payload), retained_bytes: Buffer.byteLength(retained!),
      prefix_bytes: Buffer.byteLength(prefix), prefix_well_formed: prefix.isWellFormed(),
      recent_output_truncated: status.recent_output_truncated, output_invalid_utf8: status.output_invalid_utf8,
      command_output_bytes: Buffer.byteLength(evidence.normalized_result.output)};
    console.log(JSON.stringify(observed));
    expect(observed.retained_bytes).toBe(retainedBytes);
    expect(observed.prefix_well_formed).toBe(wellFormed);
    expect(status.recent_output_truncated).toBe(true);
    expect(status.output_invalid_utf8).toBe(false);
  } finally {
    injected.control.exit();
    await client.close();
    await server.close();
    await manager.closeAll();
  }
});

Run with an isolated subprocess home and temporary directory:

WORK="$(mktemp -d)"
mkdir -p "$WORK/home" "$WORK/tmp"
HOME="$WORK/home" TMPDIR="$WORK/tmp" node node_modules/vitest/vitest.mjs run \
  --project mcp-boundary tests/boundary/mcp/rizinRecentHistoryProbe.test.ts \
  --maxWorkers=1 --no-file-parallelism

Clear inherited REA_*, XDG_*, NODE_OPTIONS and NODE_PATH overrides when running this probe. The verification run did so and placed XDG directories under its isolated home. Each case ran once; all three characterization assertions passed. The test helpers create and close a host fixture TTY and inject a Linux-mode provider process with no inferior.

Expected behavior

Retain at most 16,384 UTF-8 bytes of original frame text, ending on a complete Unicode code point, followed by the existing truncation label. Keep the truncation flag and preserve the full command-result path.

The status contract promises bounded recent output. The specific byte unit follows the MAX_RECENT_FRAME_BYTES constant and its byte-count guard. The fixed label is outside this prefix budget, as the ASCII control demonstrates; no separately documented exact public response-size guarantee was found.

Impact and scope

Large multibyte frames can retain and return up to three times the intended prefix bytes per history entry. The observed CJK case retained 32,768 excess prefix bytes; the emoji case also damaged the final retained character. The same source caps history at 64 entries, but aggregate retention was not measured.

The defect is bounded and affects session history. Full command Evidence was preserved in these cases. No crash, target effect, memory-use measurement or token-cost measurement is claimed. Real-provider frequency and a general caller workaround are NOT_ENOUGH_INFO.

REA version

Source revision 3537e58c1ab29309f06eabf50bc52adba9817a0c; package metadata 6.7.0.

Runtime and platform

Node v26.12.0; macOS 27.0.1 build 26A434; arm64; locked Vitest 5.0.3. The repository pins Node 24.18.0; that runtime was not exercised. The injected provider models Linux protocol behavior on this host.

Target and artifact context

Generated ASCII, CJK and emoji text only. The nominal fixture.bin path is recorded by the injected provider workflow; no binary is opened or executed. No installed Rizin version applies to this mocked-boundary result.

Interface

MCP

Earliest failure stage

Capture or observation

Relevant output or evidence

Measured through the registered MCP tools:

case       input bytes  retained prefix bytes  bytes including label  prefix well formed
ascii      20000        16384                  16408                  true
cjk        60000        49152                  49176                  true
surrogate  16487        16386                  16410                  false

Each full command result matched its input. The history's output_invalid_utf8 remained false because the received frame bytes were valid; that field describes input decoding, so no separate flag-contract defect is claimed.

Frame decoding computes the Buffer byte length. History retention compares that count to the byte budget, then uses a UTF-16 slice. The MCP status handler returns the stored strings.

Typed outcome or recovery expected

Keep recent_output_truncated: true for over-budget frames. Bound the retained original-text prefix in UTF-8 bytes and avoid splitting code points; keep the existing complete command output for these under-command-budget inputs.

Analysis or suspected cause

Source-derived cause, consistent with the measurements: String.slice counts UTF-16 code units while the truncation guard counts bytes. The 16,384-code-unit CJK prefix therefore occupies 49,152 UTF-8 bytes, and an emoji crossing that code-unit boundary is split.

Additional context

The relevant manager, handler and contract files have no diff between rea-agents-6.7.0 and the tested main revision. This establishes stable source coverage only; the stable runtime was not rerun.

The supplied complete open-and-closed issue snapshot had no matching recent-history byte-budget report. Related #4015 concerns interrupted command stdout; #4016 concerns completion-marker whitespace. Neither concerns recent-history prefix retention.

NOT_ENOUGH_INFO: real Rizin behavior/frequency, Linux descriptor verification, compiled CLI, public stdio transport and pinned Node runtime. The probe establishes the exercised adapter/MCP boundary only.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5091 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Process capture or runtime observation ## Observed behavior A mocked-provider MCP probe supplied one valid 60,000-byte UTF-8 NUL frame containing 20,000 copies of `雪`. `rizin_debug_session_status` returned a 49,152-byte retained prefix plus the 24-byte `[recent frame truncated]` label. The implementation's recent-frame prefix budget is 16,384 bytes. An ASCII control retained the expected 16,384-byte prefix. A separate valid Unicode fixture was cut between an emoji's surrogate halves, leaving an ill-formed history prefix. All three status responses reported `recent_output_truncated: true`. Full `rizin_debug_command` output remained exact and untruncated. This executed the production manager, registered MCP handlers, schemas and linked in-memory transport with an injected process. No Rizin engine or target program ran. ## Reproduction In a checkout of source revision `3537e58c1ab29309f06eabf50bc52adba9817a0c` with locked dependencies installed, save this as `tests/boundary/mcp/rizinRecentHistoryProbe.test.ts`: ```ts import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { McpServer } from "@modelcontextprotocol/server"; import { expect, it } from "vitest"; import { registerRizinDebugTools } from "../../../src/server/registerRizinDebugTools.js"; import { createToolResultDelivery } from "../../../src/server/toolResult.js"; import { toolContract } from "../../../src/contracts/toolContracts.js"; import { silentLogger } from "../../../src/logger.js"; import { createInjectedProcess } from "../providers/RizinDebugSessionCoverage.fixtures.js"; import { createRizinTestManager, rizinTestInferiorTty } from "../providers/RizinTestInferiorTty.js"; it.each([ { label: "ascii", payload: "a".repeat(20000), retainedBytes: 16408, wellFormed: true }, { label: "cjk", payload: "雪".repeat(20000), retainedBytes: 49176, wellFormed: true }, { label: "surrogate", payload: "a".repeat(16383) + "😀" + "x".repeat(100), retainedBytes: 16410, wellFormed: false }, ])("observes $label history through MCP", async ({label, payload, retainedBytes, wellFormed}) => { const injected = createInjectedProcess({onWrite: (text, control) => { const marker = /!echo (.+)\n/u.exec(text)?.[1]; if (marker === undefined) return; setImmediate(() => control.output(`${payload}\0${marker}\n\0`)); }}); const manager = createRizinTestManager({}, {spawnProcess: injected.spawnProcess, frameTimeoutMs: 1000}); const server = new McpServer({name: "history-probe", version: "1"}); registerRizinDebugTools(server, manager, silentLogger, createToolResultDelivery(undefined)); const client = new Client({name: "history-probe", version: "1"}); const [ct, st] = InMemoryTransport.createLinkedPair(); try { await server.connect(st); await client.connect(ct); const startReply = await client.callTool({name: "start_rizin_debug_session", arguments: { path: "fixture.bin", inferior_tty: await rizinTestInferiorTty(), }}); expect(startReply.isError).not.toBe(true); const started = toolContract("start_rizin_debug_session").outputSchema.parse(startReply.structuredContent); const commandReply = await client.callTool({name: "rizin_debug_command", arguments: { session_id: started.session_id, command: "dr", }}); expect(commandReply.isError).not.toBe(true); const evidence = toolContract("rizin_debug_command").outputSchema.parse(commandReply.structuredContent); expect(evidence.normalized_result.output).toBe(payload); expect(evidence.normalized_result.output_truncated).toBe(false); const statusReply = await client.callTool({name: "rizin_debug_session_status", arguments: { session_id: started.session_id, }}); expect(statusReply.isError).not.toBe(true); const status = toolContract("rizin_debug_session_status").outputSchema.parse(statusReply.structuredContent); const retained = status.recent_output.find(x => x.endsWith("[recent frame truncated]")); expect(retained).toBeDefined(); const prefix = retained!.slice(0, -"[recent frame truncated]".length); const observed = {label, input_bytes: Buffer.byteLength(payload), retained_bytes: Buffer.byteLength(retained!), prefix_bytes: Buffer.byteLength(prefix), prefix_well_formed: prefix.isWellFormed(), recent_output_truncated: status.recent_output_truncated, output_invalid_utf8: status.output_invalid_utf8, command_output_bytes: Buffer.byteLength(evidence.normalized_result.output)}; console.log(JSON.stringify(observed)); expect(observed.retained_bytes).toBe(retainedBytes); expect(observed.prefix_well_formed).toBe(wellFormed); expect(status.recent_output_truncated).toBe(true); expect(status.output_invalid_utf8).toBe(false); } finally { injected.control.exit(); await client.close(); await server.close(); await manager.closeAll(); } }); ``` Run with an isolated subprocess home and temporary directory: ```sh WORK="$(mktemp -d)" mkdir -p "$WORK/home" "$WORK/tmp" HOME="$WORK/home" TMPDIR="$WORK/tmp" node node_modules/vitest/vitest.mjs run \ --project mcp-boundary tests/boundary/mcp/rizinRecentHistoryProbe.test.ts \ --maxWorkers=1 --no-file-parallelism ``` Clear inherited `REA_*`, `XDG_*`, `NODE_OPTIONS` and `NODE_PATH` overrides when running this probe. The verification run did so and placed XDG directories under its isolated home. Each case ran once; all three characterization assertions passed. The test helpers create and close a host fixture TTY and inject a Linux-mode provider process with no inferior. ## Expected behavior Retain at most 16,384 UTF-8 bytes of original frame text, ending on a complete Unicode code point, followed by the existing truncation label. Keep the truncation flag and preserve the full command-result path. The [status contract](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/contracts/rizinDebugToolContracts.ts#L68-L84) promises bounded recent output. The specific byte unit follows the [MAX_RECENT_FRAME_BYTES constant](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/rizin/RizinDebugSessionManager.ts#L45) and its byte-count guard. The fixed label is outside this prefix budget, as the ASCII control demonstrates; no separately documented exact public response-size guarantee was found. ## Impact and scope Large multibyte frames can retain and return up to three times the intended prefix bytes per history entry. The observed CJK case retained 32,768 excess prefix bytes; the emoji case also damaged the final retained character. The same source caps history at 64 entries, but aggregate retention was not measured. The defect is bounded and affects session history. Full command Evidence was preserved in these cases. No crash, target effect, memory-use measurement or token-cost measurement is claimed. Real-provider frequency and a general caller workaround are NOT_ENOUGH_INFO. ## REA version Source revision `3537e58c1ab29309f06eabf50bc52adba9817a0c`; package metadata 6.7.0. ## Runtime and platform Node v26.12.0; macOS 27.0.1 build 26A434; arm64; locked Vitest 5.0.3. The repository pins Node 24.18.0; that runtime was not exercised. The injected provider models Linux protocol behavior on this host. ## Target and artifact context Generated ASCII, CJK and emoji text only. The nominal `fixture.bin` path is recorded by the injected provider workflow; no binary is opened or executed. No installed Rizin version applies to this mocked-boundary result. ## Interface MCP ## Earliest failure stage Capture or observation ## Relevant output or evidence Measured through the registered MCP tools: ```text case input bytes retained prefix bytes bytes including label prefix well formed ascii 20000 16384 16408 true cjk 60000 49152 49176 true surrogate 16487 16386 16410 false ``` Each full command result matched its input. The history's `output_invalid_utf8` remained false because the received frame bytes were valid; that field describes input decoding, so no separate flag-contract defect is claimed. [Frame decoding](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/rizin/RizinDebugSessionManager.ts#L1407-L1414) computes the Buffer byte length. [History retention](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/rizin/RizinDebugSessionManager.ts#L1064-L1078) compares that count to the byte budget, then uses a UTF-16 slice. The [MCP status handler](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/server/registerRizinDebugTools.ts#L108-L124) returns the stored strings. ## Typed outcome or recovery expected Keep `recent_output_truncated: true` for over-budget frames. Bound the retained original-text prefix in UTF-8 bytes and avoid splitting code points; keep the existing complete command output for these under-command-budget inputs. ## Analysis or suspected cause Source-derived cause, consistent with the measurements: `String.slice` counts UTF-16 code units while the truncation guard counts bytes. The 16,384-code-unit CJK prefix therefore occupies 49,152 UTF-8 bytes, and an emoji crossing that code-unit boundary is split. ## Additional context The relevant manager, handler and contract files have no diff between `rea-agents-6.7.0` and the tested main revision. This establishes stable source coverage only; the stable runtime was not rerun. The supplied complete open-and-closed issue snapshot had no matching recent-history byte-budget report. Related [#4015](https://github.com/morluto/rea/issues/4015) concerns interrupted command stdout; [#4016](https://github.com/morluto/rea/issues/4016) concerns completion-marker whitespace. Neither concerns recent-history prefix retention. NOT_ENOUGH_INFO: real Rizin behavior/frequency, Linux descriptor verification, compiled CLI, public stdio transport and pinned Node runtime. The probe establishes the exercised adapter/MCP boundary only.

bsakaguchi closed this issue

2026-10-31 00:36:57 +00:00

bsakaguchi commented

2026-10-31 00:36:58 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

The root cause is truncating by code-point count instead of UTF-8 byte budget — for CJK that multiplies the retained size by the code-unit factor (3 for 雪), and the surrogate split shows the cut also lands mid-grapheme.

I'd suggest truncating by measured byte length (e.g. TextEncoder().encode and walk code points accumulating byte size until the budget), and only cut on a code-point boundary — plus optionally clamp the tail to the last complete grapheme. That fixes both the over-retention and the ill-formed prefix, and keeps the [recent frame truncated] marker accurate.

@XinwangTao commented on GitHub (Oct 19, 2026): The root cause is truncating by code-point count instead of UTF-8 byte budget — for CJK that multiplies the retained size by the code-unit factor (3 for 雪), and the surrogate split shows the cut also lands mid-grapheme. I'd suggest truncating by measured byte length (e.g. TextEncoder().encode and walk code points accumulating byte size until the budget), and only cut on a code-point boundary — plus optionally clamp the tail to the last complete grapheme. That fixes both the over-retention and the ill-formed prefix, and keeps the [recent frame truncated] marker accurate.

bsakaguchi commented

2026-10-31 00:36:58 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 20, 2026):

The #5246 fix addresses exactly the two failure modes from the probe — the CJK case now stays within the 16,384-byte budget instead of retaining 3x, and the emoji surrogate case keeps the retained prefix well-formed. Bounding by UTF-8 bytes with the label kept outside the budget matches the contract's byte unit. Thanks gensokyo-dev; glad this shipped in 6.8.0.

@XinwangTao commented on GitHub (Oct 20, 2026): The #5246 fix addresses exactly the two failure modes from the probe — the CJK case now stays within the 16,384-byte budget instead of retaining 3x, and the emoji surrogate case keeps the retained prefix well-formed. Bounding by UTF-8 bytes with the label kept outside the budget matches the contract's byte unit. Thanks gensokyo-dev; glad this shipped in 6.8.0.

bsakaguchi referenced this issue

2026-10-31 00:40:20 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-10-31 01:08:08 +00:00

[PR #3129] feat(ghidra): admit ELF and thin Mach-O targets on Windows #5634