morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5242] [Bug] Rizin recent-output retains up to 3x its UTF-8 byte budget for CJK frames #3207

New issue

Closed

opened 2026-10-31 00:37:50 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:37:50 +00:00

Owner

Originally created by @Faisal-Fayaz on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5242

Summary

MAX_RECENT_FRAME_BYTES is a byte budget, but the recent-frame slice operates on frame.text, a decoded JS string, where slice counts UTF-16 code units. Every multi-byte character is retained at up to 3× its budget cost, so a CJK or emoji frame retains up to 49,152 bytes against the 16,384-byte cap.

The identical operation 20 lines earlier — #startupCapturedOutput, harness2-style byte budgeting at RizinDebugSessionManager.ts:1046-1052 — slices protocolBytes as a Buffer and walks back off continuation bytes. This path slices the decoded string instead.

Root cause

src/rizin/RizinDebugSessionManager.ts:1068-1078:

const historyFrame =
  frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated
    ? `${frame.text.slice(0, MAX_RECENT_FRAME_BYTES)}[recent frame truncated]`
    : frame.text;

MAX_RECENT_FRAME_BYTES = 16 * 1024 (RizinDebugSessionManager.ts:45). The guard on the left compares frameBytes — a real byte count — against the budget. The slice on the right does not. frame.text is a string (RizinDebugSessionManager.ts:207), and String.prototype.slice takes code units, so 16,384 code units of 3-byte CJK is 49,152 bytes.

The [recent frame truncated] label adds 24 bytes on top, which is why the ASCII case is already 24 over. That part is bounded and intentional-looking. The multiplier is not.

Measured

Transcribing #deliverFrame's slice verbatim (no REA build required):

frame	frame bytes	retained	vs budget
`"A" × 60000`	60,000	16,408	×1.00 (+24 for the label)
`"é" × 30000` (2-byte)	60,000	32,792	×2.00
`"雪" × 20000` (3-byte)	60,000	49,176	×3.00
emoji `× 10000` (4-byte surrogate pair)	40,000	32,792	×2.00

Emoji is worth separating: one code point is two code units, so 10,000 emoji are 16,384 code units — exactly the budget — but 40,000 bytes, so it is truncated at the cap while retaining 32,792 bytes. The multiplier there is 2 because the surrogate pair halves the code units per byte, not because the budget was met.

Aggravating factor: the history ring multiplies it

recent_output retains up to 64 frames (RizinDebugSessionManager.ts:1076-1079). Each CJK frame carries its own 3× overshoot, so the intended 1 MiB of recent output becomes 3 MiB — the ring bounds frame count, and the byte budget is the only thing bounding frame size, and that is the part that fails.

Reproduction

frame.text is produced by bytes.toString("utf8") (RizinDebugSessionManager.ts:1413), so a valid UTF-8 frame of CJK is enough to reach this. No malformed input required, and invalidUtf8 is not involved:

// Transcribes #deliverFrame's slice from
// src/rizin/RizinDebugSessionManager.ts:1068-1078 verbatim.
const MAX_RECENT_FRAME_BYTES = 16 * 1024;

function retain(text, frameBytes, truncated) {
  return frameBytes > MAX_RECENT_FRAME_BYTES || truncated
    ? `${text.slice(0, MAX_RECENT_FRAME_BYTES)}[recent frame truncated]`
    : text;
}

for (const [label, text] of [
  ["ASCII  ", "A".repeat(60000)],
  ["2-byte ", "é".repeat(30000)],
  ["3-byte ", "雪".repeat(20000)],
  ["emoji  ", "\u{1F600}".repeat(10000)],
]) {
  const frameBytes = Buffer.byteLength(text, "utf8");
  const kept = retain(text, frameBytes, false);
  console.log(label, frameBytes, "->", Buffer.byteLength(kept, "utf8"));
}

ASCII    60000 -> 16408
2-byte   60000 -> 32792
3-byte   60000 -> 49176
emoji    40000 -> 32792

The existing coverage at tests/boundary/providers/RizinDebugSessionCoverage.part4.test.ts:90-99 asserts [recent frame truncated] appears for a 17,000-byte ASCII frame. That test passes, because ASCII makes code units and bytes the same number — which is why this has gone unnoticed.

Suggested direction

Slice on the byte side, the way #startupCapturedOutput already does at lines 1046-1052:

const bytes = Buffer.from(frame.text, "utf8");
const kept = bytes.subarray(0, MAX_RECENT_FRAME_BYTES);
// walk back off a split multi-byte sequence
let end = kept.length;
while (end > 0 && (kept[end]! & 0xc0) === 0x80) end -= 1;
const historyFrame =
  frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated
    ? `${kept.subarray(0, end).toString("utf8")}[recent frame truncated]`
    : frame.text;

This keeps the retained prefix under the byte cap for every encoding, and the existing continuation-byte walk handles the split-sequence case without producing replacement characters. Happy to send a patch with a CJK regression case alongside the existing ASCII one, if that is welcome.

Environment

a0edd14, Linux, Node 24.21.0 (repo .nvmrc pins 24.18.0).
Measured by transcribing the slice, not by running the manager: node_modules is not installed in the checkout I used. The transcription is the one-line expression at 1071 with the constant from line 45, so it is the current behaviour; I have not executed it inside a live RizinDebugSessionManager session and am not claiming a runtime reproduction of the MCP status call.

Originally created by @Faisal-Fayaz on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5242 ## Summary `MAX_RECENT_FRAME_BYTES` is a byte budget, but the recent-frame slice operates on `frame.text`, a decoded JS string, where `slice` counts UTF-16 code units. Every multi-byte character is retained at up to 3× its budget cost, so a CJK or emoji frame retains up to 49,152 bytes against the 16,384-byte cap. The identical operation 20 lines earlier — `#startupCapturedOutput`, `harness2`-style byte budgeting at `RizinDebugSessionManager.ts:1046-1052` — slices `protocolBytes` as a `Buffer` and walks back off continuation bytes. This path slices the decoded string instead. ## Root cause `src/rizin/RizinDebugSessionManager.ts:1068-1078`: ```ts const historyFrame = frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated ? `${frame.text.slice(0, MAX_RECENT_FRAME_BYTES)}[recent frame truncated]` : frame.text; ``` `MAX_RECENT_FRAME_BYTES = 16 * 1024` (`RizinDebugSessionManager.ts:45`). The guard on the left compares `frameBytes` — a real byte count — against the budget. The slice on the right does not. `frame.text` is a `string` (`RizinDebugSessionManager.ts:207`), and `String.prototype.slice` takes code units, so 16,384 code units of 3-byte CJK is 49,152 bytes. The `[recent frame truncated]` label adds 24 bytes on top, which is why the ASCII case is already 24 over. That part is bounded and intentional-looking. The multiplier is not. ## Measured Transcribing `#deliverFrame`'s slice verbatim (no REA build required): | frame | frame bytes | retained | vs budget | |---|---|---|---| | `"A" × 60000` | 60,000 | 16,408 | ×1.00 (+24 for the label) | | `"é" × 30000` (2-byte) | 60,000 | 32,792 | **×2.00** | | `"雪" × 20000` (3-byte) | 60,000 | 49,176 | **×3.00** | | emoji `× 10000` (4-byte surrogate pair) | 40,000 | 32,792 | **×2.00** | Emoji is worth separating: one code point is two code units, so 10,000 emoji are 16,384 code units — exactly the budget — but 40,000 bytes, so it is truncated at the cap while retaining 32,792 bytes. The multiplier there is 2 because the surrogate pair halves the code units per byte, not because the budget was met. ## Aggravating factor: the history ring multiplies it `recent_output` retains up to 64 frames (`RizinDebugSessionManager.ts:1076-1079`). Each CJK frame carries its own 3× overshoot, so the intended 1 MiB of recent output becomes 3 MiB — the ring bounds frame *count*, and the byte budget is the only thing bounding frame *size*, and that is the part that fails. ## Reproduction `frame.text` is produced by `bytes.toString("utf8")` (`RizinDebugSessionManager.ts:1413`), so a valid UTF-8 frame of CJK is enough to reach this. No malformed input required, and `invalidUtf8` is not involved: ```js // Transcribes #deliverFrame's slice from // src/rizin/RizinDebugSessionManager.ts:1068-1078 verbatim. const MAX_RECENT_FRAME_BYTES = 16 * 1024; function retain(text, frameBytes, truncated) { return frameBytes > MAX_RECENT_FRAME_BYTES || truncated ? `${text.slice(0, MAX_RECENT_FRAME_BYTES)}[recent frame truncated]` : text; } for (const [label, text] of [ ["ASCII ", "A".repeat(60000)], ["2-byte ", "é".repeat(30000)], ["3-byte ", "雪".repeat(20000)], ["emoji ", "\u{1F600}".repeat(10000)], ]) { const frameBytes = Buffer.byteLength(text, "utf8"); const kept = retain(text, frameBytes, false); console.log(label, frameBytes, "->", Buffer.byteLength(kept, "utf8")); } ``` ``` ASCII 60000 -> 16408 2-byte 60000 -> 32792 3-byte 60000 -> 49176 emoji 40000 -> 32792 ``` The existing coverage at `tests/boundary/providers/RizinDebugSessionCoverage.part4.test.ts:90-99` asserts `[recent frame truncated]` appears for a 17,000-byte ASCII frame. That test passes, because ASCII makes code units and bytes the same number — which is why this has gone unnoticed. ## Suggested direction Slice on the byte side, the way `#startupCapturedOutput` already does at lines 1046-1052: ```ts const bytes = Buffer.from(frame.text, "utf8"); const kept = bytes.subarray(0, MAX_RECENT_FRAME_BYTES); // walk back off a split multi-byte sequence let end = kept.length; while (end > 0 && (kept[end]! & 0xc0) === 0x80) end -= 1; const historyFrame = frameBytes > MAX_RECENT_FRAME_BYTES || frame.truncated ? `${kept.subarray(0, end).toString("utf8")}[recent frame truncated]` : frame.text; ``` This keeps the retained prefix under the byte cap for every encoding, and the existing continuation-byte walk handles the split-sequence case without producing replacement characters. Happy to send a patch with a CJK regression case alongside the existing ASCII one, if that is welcome. ## Environment - `a0edd14`, Linux, Node 24.21.0 (repo `.nvmrc` pins 24.18.0). - Measured by transcribing the slice, not by running the manager: `node_modules` is not installed in the checkout I used. The transcription is the one-line expression at 1071 with the constant from line 45, so it is the current behaviour; I have not executed it inside a live `RizinDebugSessionManager` session and am not claiming a runtime reproduction of the MCP `status` call.

bsakaguchi closed this issue

2026-10-31 00:37:50 +00:00

bsakaguchi commented

2026-10-31 00:37:51 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 19, 2026):

Direct duplicate of canonical bug report #5091 ([Bug] Rizin recent output retains three times its UTF-8 byte budget for CJK frames), opened on October 18, 2026 by @coygeek. Both reports pinpoint the identical defect where recent output truncation in Rizin counts code-point length rather than UTF-8 byte budget, causing CJK multi-byte frames (e.g. 3 bytes per character) to overrun the memory budget by 3x.

Recommend closing this newer duplicate in favor of #5091. @morluto

Indexed under Section 1 (Direct Duplicates) in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 19, 2026): Direct duplicate of canonical bug report #5091 (`[Bug] Rizin recent output retains three times its UTF-8 byte budget for CJK frames`), opened on October 18, 2026 by @coygeek. Both reports pinpoint the identical defect where recent output truncation in Rizin counts code-point length rather than UTF-8 byte budget, causing CJK multi-byte frames (e.g. 3 bytes per character) to overrun the memory budget by 3x. Recommend closing this newer duplicate in favor of #5091. @morluto Indexed under Section 1 (Direct Duplicates) in consolidation tracker #4330.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3207

No description provided.

Rows
Columns