morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5097] [Bug] Frida error descriptions exceed their JSON byte cap for lone high surrogates #3135

New issue

Closed

opened 2026-10-31 00:37:01 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:37:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5097

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

A mocked Frida message delivered through the production manager's subscribed script signal retained an error description larger than its source-defined 32,768-byte JSON-content limit. A description containing 32,765 ASCII characters followed by one lone high surrogate retained 32,771 bytes. Both the field and script truncation flags remained false. The same message appeared in session status.

A second case containing 10,922 lone high surrogates retained 65,532 JSON-content bytes with both flags false. This was a synthetic provider-boundary execution, without native Frida, target instrumentation, or CLI/MCP transport execution.

Reproduction

From the source checkout with lockfile dependencies available, save the following as probe.mts in the checkout root and run node node_modules/tsx/dist/cli.mjs probe.mts. This is the executed probe with its source import made repository-relative. The fake handles do not attach to any actual process.

import assert from "node:assert/strict";
import { FridaInstrumentationManager } from "./src/frida/FridaInstrumentationManager.ts";
const cap = 32 * 1024;
const cases = [
  ["ascii-control", "x".repeat(cap)],
  ["trailing-high", "x".repeat(cap - 3) + "\ud800"],
  ["trailing-low-control", "x".repeat(cap - 3) + "\udc00"],
  ["paired-control", "x".repeat(cap - 4) + "\ud800\udc00"],
  ["repeated-high", "\ud800".repeat(10922)],
];
console.log(JSON.stringify({node:process.version, platform:process.platform, arch:process.arch}));
for (const [name, description] of cases) {
  let listener;
  let destroyed = false;
  let detached = false;
  const script = {
    get isDestroyed() { return destroyed; },
    message: { connect(fn) { listener = fn; }, disconnect() { listener = undefined; } },
    async load() { listener({type:"error", description}, null); },
    async unload() { destroyed = true; },
  };
  const session = {
    pid:512, detached:{connect(){},disconnect(){}}, isDetached:()=>detached,
    async createScript(){ return script; }, async detach(){detached=true;},
  };
  const device = {id:"local",name:"fixture",type:"local",
    async getProcessByPid(){ return {pid:512,name:"fixture"}; },
    async attach(){return session;},
  };
  const manager = new FridaInstrumentationManager({deviceManager:{async getDeviceById(){return device;}}});
  try {
    const started = await manager.startSession({mode:"attach",pid:512,deviceId:"local"});
    assert(started.ok);
    const loaded = await manager.loadScript(started.value.sessionId,{sourceKind:"inline",source:"// synthetic fixture"});
    assert(loaded.ok);
    const message = loaded.value.messages[0];
    assert(message);
    assert.deepEqual(manager.status(started.value.sessionId).messages[0],message);
    const bytes = Buffer.byteLength(JSON.stringify(message.description)) - 2;
    const result = {name, inputUnits:description.length, retainedUnits:message.description.length,
      contentJsonBytes:bytes, cap, exceedsCap:bytes>cap,
      descriptionTruncated:message.description_truncated ?? false, messagesTruncated:loaded.value.messagesTruncated};
    console.log(JSON.stringify(result));
    if (name === "trailing-high") assert.equal(bytes, cap + 3);
    if (name === "repeated-high") assert.equal(bytes,65532);
    if (name === "ascii-control" || name === "paired-control") assert.equal(bytes,cap);
    if (name === "trailing-low-control") {assert.equal(bytes,cap-3); assert.equal(message.description_truncated,true);}
  } finally { await manager.closeAll(); }
  assert(destroyed && detached);
}

All five cases completed successfully. The assertions check the observed faulty behavior, controls, retained status, and synthetic cleanup; they are reproduction assertions rather than expected-fix assertions.

Expected behavior

The description's serialized JSON content should remain within the configured byte cap, and omitted text should set the existing truncation flags. The 32 KiB constant, its description-field application, and the JSON-byte comparison establish this source contract. The 32 KiB per-field limit is not a separately documented public configuration option.

Impact and scope

Lone high surrogates cause the error-description representation bound to be exceeded in retained results. Repeated instances can approximately double this field's retained JSON size. ASCII, valid pairs, and lone-low-surrogate controls respected the cap in this probe. The separate whole-message/session 16 MiB accounting remains present; no unbounded memory growth, crash, or security effect is claimed.

Frequency in real provider messages and a practical workaround are NOT_ENOUGH_INFO. Stack and file-name metadata use the same bounded-string source path but were not exercised. Arbitrary message-type values and send-payload behavior are outside this reproduction.

REA version

Source main 3537e58c1ab29309f06eabf50bc52adba9817a0c, package version 6.7.0. The manager and service files have no diff from rea-agents-6.7.0; the stable installed package was not executed.

Runtime and platform

Node.js v26.12.0; Darwin 27.0.0; arm64. This supported Node version was exercised; the repository-pinned Node 24.18.0 was not. Native Frida was not loaded.

Target and artifact context

Synthetic error messages and in-memory device, session, and script handles. No actual binary, device, service, or target process was used.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

Observed probe output, with JSON-content size excluding the two string quotes:

ascii-control:       bytes=32768 cap=32768 descriptionTruncated=false messagesTruncated=false
trailing-high:       bytes=32771 cap=32768 descriptionTruncated=false messagesTruncated=false
trailing-low-control:bytes=32765 cap=32768 descriptionTruncated=true  messagesTruncated=true
paired-control:      bytes=32768 cap=32768 descriptionTruncated=false messagesTruncated=false
repeated-high:       bytes=65532 cap=32768 descriptionTruncated=false messagesTruncated=false

The subscribed message callback captures the fixture during load and returns the retained message. Service projections forward these messages without another field-size check; that projection is source-derived, not an executed MCP result.

Typed outcome or recovery expected

Use the existing description_truncated and messagesTruncated signals when shortening an oversized description, while keeping its serialized JSON-content bytes within the configured limit.

Analysis or suspected cause

The JSON string byte counter starts with UTF-8 byte length and adds JSON-escape overhead. Its high-surrogate branch checks only the next code unit's range; at the end that lookup is NaN, so both comparisons are false. A lone high surrogate costs six JSON-content bytes but is charged three. The bounded-string loop calls this counter on one code point at a time, so the omission repeats for every lone high surrogate.

The later whole-message charge measures actual JSON bytes but enforces the separate 16 MiB bound rather than correcting individual metadata fields.

Additional context

The complete supplied open-and-closed issue snapshot was searched. #2529 concerns inherited script completeness after older-message eviction; #2530 concerns missing error locations. Neither covers this per-field byte-accounting trigger. Real Frida delivery, installed-release execution, and end-to-end CLI/MCP behavior remain untested.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5097 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Process capture or runtime observation ## Observed behavior A mocked Frida message delivered through the production manager's subscribed script signal retained an error description larger than its source-defined 32,768-byte JSON-content limit. A description containing 32,765 ASCII characters followed by one lone high surrogate retained 32,771 bytes. Both the field and script truncation flags remained false. The same message appeared in session status. A second case containing 10,922 lone high surrogates retained 65,532 JSON-content bytes with both flags false. This was a synthetic provider-boundary execution, without native Frida, target instrumentation, or CLI/MCP transport execution. ## Reproduction From the source checkout with lockfile dependencies available, save the following as `probe.mts` in the checkout root and run `node node_modules/tsx/dist/cli.mjs probe.mts`. This is the executed probe with its source import made repository-relative. The fake handles do not attach to any actual process. ```ts import assert from "node:assert/strict"; import { FridaInstrumentationManager } from "./src/frida/FridaInstrumentationManager.ts"; const cap = 32 * 1024; const cases = [ ["ascii-control", "x".repeat(cap)], ["trailing-high", "x".repeat(cap - 3) + "\ud800"], ["trailing-low-control", "x".repeat(cap - 3) + "\udc00"], ["paired-control", "x".repeat(cap - 4) + "\ud800\udc00"], ["repeated-high", "\ud800".repeat(10922)], ]; console.log(JSON.stringify({node:process.version, platform:process.platform, arch:process.arch})); for (const [name, description] of cases) { let listener; let destroyed = false; let detached = false; const script = { get isDestroyed() { return destroyed; }, message: { connect(fn) { listener = fn; }, disconnect() { listener = undefined; } }, async load() { listener({type:"error", description}, null); }, async unload() { destroyed = true; }, }; const session = { pid:512, detached:{connect(){},disconnect(){}}, isDetached:()=>detached, async createScript(){ return script; }, async detach(){detached=true;}, }; const device = {id:"local",name:"fixture",type:"local", async getProcessByPid(){ return {pid:512,name:"fixture"}; }, async attach(){return session;}, }; const manager = new FridaInstrumentationManager({deviceManager:{async getDeviceById(){return device;}}}); try { const started = await manager.startSession({mode:"attach",pid:512,deviceId:"local"}); assert(started.ok); const loaded = await manager.loadScript(started.value.sessionId,{sourceKind:"inline",source:"// synthetic fixture"}); assert(loaded.ok); const message = loaded.value.messages[0]; assert(message); assert.deepEqual(manager.status(started.value.sessionId).messages[0],message); const bytes = Buffer.byteLength(JSON.stringify(message.description)) - 2; const result = {name, inputUnits:description.length, retainedUnits:message.description.length, contentJsonBytes:bytes, cap, exceedsCap:bytes>cap, descriptionTruncated:message.description_truncated ?? false, messagesTruncated:loaded.value.messagesTruncated}; console.log(JSON.stringify(result)); if (name === "trailing-high") assert.equal(bytes, cap + 3); if (name === "repeated-high") assert.equal(bytes,65532); if (name === "ascii-control" || name === "paired-control") assert.equal(bytes,cap); if (name === "trailing-low-control") {assert.equal(bytes,cap-3); assert.equal(message.description_truncated,true);} } finally { await manager.closeAll(); } assert(destroyed && detached); } ``` All five cases completed successfully. The assertions check the observed faulty behavior, controls, retained status, and synthetic cleanup; they are reproduction assertions rather than expected-fix assertions. ## Expected behavior The description's serialized JSON content should remain within the configured byte cap, and omitted text should set the existing truncation flags. The [32 KiB constant](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L41-L44), its [description-field application](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L968-L972), and the [JSON-byte comparison](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L1209-L1228) establish this source contract. The 32 KiB per-field limit is not a separately documented public configuration option. ## Impact and scope Lone high surrogates cause the error-description representation bound to be exceeded in retained results. Repeated instances can approximately double this field's retained JSON size. ASCII, valid pairs, and lone-low-surrogate controls respected the cap in this probe. The separate whole-message/session 16 MiB accounting remains present; no unbounded memory growth, crash, or security effect is claimed. Frequency in real provider messages and a practical workaround are NOT_ENOUGH_INFO. Stack and file-name metadata use the same bounded-string source path but were not exercised. Arbitrary message-type values and send-payload behavior are outside this reproduction. ## REA version Source main `3537e58c1ab29309f06eabf50bc52adba9817a0c`, package version 6.7.0. The manager and service files have no diff from `rea-agents-6.7.0`; the stable installed package was not executed. ## Runtime and platform Node.js v26.12.0; Darwin 27.0.0; arm64. This supported Node version was exercised; the repository-pinned Node 24.18.0 was not. Native Frida was not loaded. ## Target and artifact context Synthetic error messages and in-memory device, session, and script handles. No actual binary, device, service, or target process was used. ## Interface Other ## Earliest failure stage Capture or observation ## Relevant output or evidence Observed probe output, with JSON-content size excluding the two string quotes: ```text ascii-control: bytes=32768 cap=32768 descriptionTruncated=false messagesTruncated=false trailing-high: bytes=32771 cap=32768 descriptionTruncated=false messagesTruncated=false trailing-low-control:bytes=32765 cap=32768 descriptionTruncated=true messagesTruncated=true paired-control: bytes=32768 cap=32768 descriptionTruncated=false messagesTruncated=false repeated-high: bytes=65532 cap=32768 descriptionTruncated=false messagesTruncated=false ``` The [subscribed message callback](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L424-L452) captures the fixture during load and returns the retained message. [Service projections](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/frida/FridaInstrumentationService.ts#L458-L479) forward these messages without another field-size check; that projection is source-derived, not an executed MCP result. ## Typed outcome or recovery expected Use the existing `description_truncated` and `messagesTruncated` signals when shortening an oversized description, while keeping its serialized JSON-content bytes within the configured limit. ## Analysis or suspected cause The [JSON string byte counter](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L1343-L1375) starts with UTF-8 byte length and adds JSON-escape overhead. Its high-surrogate branch checks only the next code unit's range; at the end that lookup is NaN, so both comparisons are false. A lone high surrogate costs six JSON-content bytes but is charged three. The bounded-string loop calls this counter on one code point at a time, so the omission repeats for every lone high surrogate. The later [whole-message charge](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/frida/FridaInstrumentationManager.ts#L1032-L1036) measures actual JSON bytes but enforces the separate 16 MiB bound rather than correcting individual metadata fields. ## Additional context The complete supplied open-and-closed issue snapshot was searched. [#2529](https://github.com/morluto/rea/issues/2529) concerns inherited script completeness after older-message eviction; [#2530](https://github.com/morluto/rea/issues/2530) concerns missing error locations. Neither covers this per-field byte-accounting trigger. Real Frida delivery, installed-release execution, and end-to-end CLI/MCP behavior remain untested.

bsakaguchi closed this issue

2026-10-31 00:37:01 +00:00

bsakaguchi commented

2026-10-31 00:37:02 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Good catch — a lone high surrogate (0xD800–0xDBFF without a low pair) is not a valid standalone code point, so counting it as 1 byte under-budgets the JSON-escaped \uD800 form which costs more on the wire. The 10,922-surrogate case showing 65,532 bytes confirms the byte counter is treating them as single bytes instead of the escaped length. Counting lone surrogates by their escaped form in the byte budget (as the linked draft #5175 does) closes the mismatch cleanly.

@XinwangTao commented on GitHub (Oct 19, 2026): Good catch — a lone high surrogate (0xD800–0xDBFF without a low pair) is not a valid standalone code point, so counting it as 1 byte under-budgets the JSON-escaped \uD800 form which costs more on the wire. The 10,922-surrogate case showing 65,532 bytes confirms the byte counter is treating them as single bytes instead of the escaped length. Counting lone surrogates by their escaped form in the byte budget (as the linked draft #5175 does) closes the mismatch cleanly.

bsakaguchi commented

2026-10-31 00:37:02 +00:00

Author

Owner

@morluto commented on GitHub (Oct 19, 2026):

Fixed on main in 435efe7af.

Root cause: Frida counted raw UTF-8 plus escape corrections, and charCodeAt beyond a trailing high surrogate returned NaN. The missing correction charged three bytes for a six-byte JSON escape. The bounded-string owner consequently retained too much text and omitted truncation flags.

The fix uses one exact domain-owned JSON string byte counter for Frida, managed metadata projections and Ghidra PDB caveats, removing the divergent counters. Ghidra keeps its cheap length guards; the exact scan is bounded to 1,022 UTF-16 units before serialization. NativeAOT keeps its specialized raw UTF-16 reader so admission still precedes string allocation. No second undercount was found in these related consumers.

Validation on pinned Node24.18.0/npm11.16.0:

Baseline: 4 failed /72 passed, including both reported inputs through subscribed Frida messages and registered MCP handlers.
Final: 296 tests passed across10 files. Description, stack, filename, complete/truncated controls, load Evidence and retained status are covered; managed member accounting and Ghidra locator limits pass. The shared counter agrees with native JSON serialization for all65,536 individual UTF-16 code units and196 adjacent fragment pairs.
npm run check, npm run docs:check and npm run verify:test-discovery passed (all1,640 files discovered exactly once).

The completed simulated-native MCP probe now retains32,766 JSON-content bytes for repeated high surrogates (previously65,532) and sets the truncation flags. Its exactly-fitting valid-pair control remains32,768 bytes without truncation. Three fresh-process medians before→after:0.322→0.341s elapsed,0.554→0.550s CPU,333.5→334.6MB sampled process-family RSS. Selected request/reply/probe-wrapper estimates were150,349→84,846 cl100k tokens; start/close and handshake/catalog frames, actual agent context and model usage were not counted. Runs were nonalternating with warm OS caches; brief RSS peaks may be missed. This is descriptive evidence, not a speedup claim.

Frida native delivery/target instrumentation and Linux/Windows remain unverified. Managed/Ghidra checks use synthetic artifact/profile facts; their full task performance is unmeasured. The representation contract is supported by V8's well-formed JSON documentation and the ECMAScript string-quoting algorithm.

Adapted the regressions reviewed at #5175 (fceee5f49), with verified Git co-author credit; that PR was not merged and its readiness declarations were not changed. Evidence is retained locally under .cache/codebase-audit/frida5097-{baseline-red,focused-final,check,docs,discovery}.log, frida5097-provenance.json, and frida5097-costs.json.

@morluto commented on GitHub (Oct 19, 2026): Fixed on main in [`435efe7af`](https://github.com/morluto/rea/commit/435efe7afcc6a286a9b450dbef682df67252ebf5). Root cause: Frida counted raw UTF-8 plus escape corrections, and `charCodeAt` beyond a trailing high surrogate returned NaN. The missing correction charged three bytes for a six-byte JSON escape. The bounded-string owner consequently retained too much text and omitted truncation flags. The fix uses one exact domain-owned JSON string byte counter for Frida, managed metadata projections and Ghidra PDB caveats, removing the divergent counters. Ghidra keeps its cheap length guards; the exact scan is bounded to 1,022 UTF-16 units before serialization. NativeAOT keeps its specialized raw UTF-16 reader so admission still precedes string allocation. No second undercount was found in these related consumers. Validation on pinned Node24.18.0/npm11.16.0: - Baseline: **4 failed /72 passed**, including both reported inputs through subscribed Frida messages and registered MCP handlers. - Final: **296 tests passed across10 files**. Description, stack, filename, complete/truncated controls, load Evidence and retained status are covered; managed member accounting and Ghidra locator limits pass. The shared counter agrees with native JSON serialization for all65,536 individual UTF-16 code units and196 adjacent fragment pairs. - `npm run check`, `npm run docs:check` and `npm run verify:test-discovery` passed (all1,640 files discovered exactly once). The completed simulated-native MCP probe now retains32,766 JSON-content bytes for repeated high surrogates (previously65,532) and sets the truncation flags. Its exactly-fitting valid-pair control remains32,768 bytes without truncation. Three fresh-process medians before→after:0.322→0.341s elapsed,0.554→0.550s CPU,333.5→334.6MB sampled process-family RSS. Selected request/reply/probe-wrapper estimates were150,349→84,846 cl100k tokens; start/close and handshake/catalog frames, actual agent context and model usage were not counted. Runs were nonalternating with warm OS caches; brief RSS peaks may be missed. This is descriptive evidence, not a speedup claim. Frida native delivery/target instrumentation and Linux/Windows remain unverified. Managed/Ghidra checks use synthetic artifact/profile facts; their full task performance is unmeasured. The representation contract is supported by [V8's well-formed JSON documentation](https://v8.dev/features/well-formed-json-stringify) and the [ECMAScript string-quoting algorithm](https://tc39.es/ecma262/multipage/structured-data.html#sec-quotejsonstring). Adapted the regressions reviewed at #5175 (`fceee5f49`), with verified Git co-author credit; that PR was not merged and its readiness declarations were not changed. Evidence is retained locally under `.cache/codebase-audit/frida5097-{baseline-red,focused-final,check,docs,discovery}.log`, `frida5097-provenance.json`, and `frida5097-costs.json`.

bsakaguchi referenced this issue

2026-10-31 00:41:20 +00:00

[GH-ISSUE #5802] perf(rust): query selected dependency facts from retained Evidence #3513

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471