morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3396] [Bug] Frida script-file Evidence loses original byte identity for non-UTF-8 files #1262

New issue

Closed

opened 2026-10-16 21:03:31 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:03:31 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3396

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

instrument-with-frida --script-file returns successful Evidence whose source_path identifies the selected file but whose source_sha256 does not identify that file's bytes when the file contains invalid UTF-8. Two files with different original bytes produce the same reported digest after their invalid bytes become replacement characters. The result has no limitation or encoding field that distinguishes the original file from the transformed source delivered for loading.

All four public CLI calls exited with status 0 and returned source_kind: "file" and cleanup_error: null. For every input, the reported digest matched the source received by the simulated native module. It failed to match the original file for both invalid-UTF-8 inputs. This is a decoding transformation, not a SHA-256 collision. The successful Evidence's limitations concerned unavailable target identity and platform metadata; they did not disclose a changed source representation.

Separate attempts using the actual native binding did not reach source loading on this host. One startup returned capability_unavailable with an access-denial reason; another timed out during startup and reported incomplete cleanup. Those attempts do not establish real native reproduction of the digest mismatch. The complete reproduction below avoids that host boundary and reproduces the mismatch through the production CLI.

Reproduction

Start from a clean checkout at 8dde6370d0809ad04cda3342d4e6f290f48aa063 and prepare the locked dependencies and runtime. These are clean-checkout prerequisites; the reported execution used existing built runtime artifacts and locked dependencies.

npm ci
npm run build:cached

Run the remaining commands from the repository root. This reproduction substitutes only the native Frida module with a local simulated device. The production CLI, script-file checks, file reader, hashing, service, and Evidence projection execute unchanged. No process is actually attached or instrumented.

Create the isolated inputs and native module replacement:

work_dir=$(mktemp -d)
mkdir "$work_dir/home"
cat > "$work_dir/frida-hook.mjs" <<'JS'
import { registerHooks } from 'node:module';
registerHooks({
  resolve(specifier, context, next) {
    if (specifier === 'frida') {
      return {
        url: new URL('./frida-fake.mjs', import.meta.url).href,
        shortCircuit: true
      };
    }
    return next(specifier, context);
  }
});
JS
cat > "$work_dir/frida-fake.mjs" <<'JS'
import { writeFile } from 'node:fs/promises';
const signal = () => ({ connect() {}, disconnect() {} });
const session = {
  detached: signal(),
  isDetached() { return this.closed === true; },
  async detach() { this.closed = true; },
  async createScript(source) {
    await writeFile(new URL('./loaded-source.txt', import.meta.url), source);
    return {
      message: signal(),
      isDestroyed: false,
      async load() {},
      async unload() { this.isDestroyed = true; }
    };
  }
};
const device = {
  id: 'local', name: 'local fixture', type: 'local',
  async getProcessByPid(pid) { return { pid, name: 'fixture' }; },
  async attach() { return session; },
  async resume() {},
  async spawn() { return 1; }
};
export function getDeviceManager() {
  return { async getDeviceById() { return device; } };
}
JS
python3 - "$work_dir" <<'PY'
from pathlib import Path
import sys
root = Path(sys.argv[1])
(root / 'sample-ff.js').write_bytes(b'//\xff\nsend("marker");\n')
(root / 'sample-fe.js').write_bytes(b'//\xfe\nsend("marker");\n')
(root / 'sample-utf8.js').write_bytes('//\ufffd\nsend("marker");\n'.encode())
(root / 'sample-ascii.js').write_bytes(b'//comment\nsend("marker");\n')
PY

Execute the public command for each input and compare the selected bytes, returned digest, and source received by the simulated native module:

env HOME="$work_dir/home" node --input-type=module - "$work_dir" <<'JS'
import { spawnSync } from 'node:child_process';
import { readFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
const root = process.argv[2];
const hash = (path) => createHash('sha256').update(readFileSync(path)).digest('hex');
for (const kind of ['ff', 'fe', 'utf8', 'ascii']) {
  const path = `${root}/sample-${kind}.js`;
  const child = spawnSync(process.execPath, [
    '--import', `${root}/frida-hook.mjs`,
    'scripts/rea.mjs', 'instrument-with-frida', 'attach', '1',
    '--script-file', path, '--duration-ms', '0', '--format', 'json'
  ], {
    env: { ...process.env, HOME: `${root}/home` },
    encoding: 'utf8', timeout: 10000
  });
  if (child.status !== 0) throw new Error(child.stderr + child.stdout);
  const result = JSON.parse(child.stdout).evidence.normalized_result;
  const selected = hash(path);
  console.log(JSON.stringify({
    kind, exit_status: child.status,
    selected_sha256: selected,
    reported_sha256: result.source_sha256,
    loaded_sha256: hash(`${root}/loaded-source.txt`),
    selected_file_matches: selected === result.source_sha256
  }));
}
JS

The two invalid bytes occur inside comments. The UTF-8 control contains an actual encoded replacement character in the same comment, and the ASCII control supplies an ordinary comment.

Expected behavior

File-source Evidence should preserve the selected file's SHA-256 identity. If loading requires a transformed text representation, the caller must be able to distinguish that representation from the original file. Successful Evidence should not pair the selected file path with an unqualified digest of different bytes. A file that cannot meet the supported source contract should instead receive an explicit input failure.

docs/frida-instrumentation.md:11 states that script source is not stored in Evidence and that its selected path and SHA-256 digest are retained. AGENTS.md:13, under Product and Authority, also requires preserving caller-selected inputs and local evidence, including digests and metadata.

Impact and scope

P2, supported. Successful source Evidence loses the original identity of accepted non-UTF-8 script files. Two distinct selected files become indistinguishable by the reported source digest, which impairs provenance checks. The original files remain unchanged, and the valid UTF-8 and ASCII controls retain correct file hashes.

Workaround available. Independently hash and retain the selected file alongside the observation, and compare it with the source hash. The reproduction calculates those distinct original hashes without modifying the inputs. This preserves the missing file identity outside the returned Evidence until the contract is restored.

REA version

6.4.0 (main at 8dde6370d0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64. The reported execution used existing built runtime artifacts and locked dependencies. The module-hook reproduction simulates native Frida and needs no server, target application, or native attach permission. Real native instrumentation was not verified.

Target and artifact context

Four caller-selected JavaScript files contain invalid bytes 0xff and 0xfe, an actual UTF-8 replacement character, and an ordinary ASCII comment, respectively. Each comment precedes send("marker");. A local simulated native Frida device accepts synthetic PID 1 and records the source delivered for loading. No process is actually attached or instrumented.

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

The observed selected-file and reported digests were:

Input       Selected-file SHA-256
ff          d6c31c67461c0c07b616f441170b4daead0d039fc6fd01a30dcc8cf54fcc487b
fe          8940bb2c848dad6af6a83c86fb1c214f83c9a624a07315f4be7a3f4b5297cd21
utf8        dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6
ascii       0fe349f28aa100e026abaa65f52a5df3c79f2493c1793f8eefa149b87730f9af

Input       Reported source_sha256
ff          dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6
fe          dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6
utf8        dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6
ascii       0fe349f28aa100e026abaa65f52a5df3c79f2493c1793f8eefa149b87730f9af

The CLI result, independently calculated file hashes, and source received by the simulated native module were executed locally. Evidence provenance was observed, local verification reproduced the failure, and reproduction completeness was complete. The real native Frida binding was replaced at its module boundary, so this verifies REA's source acquisition and Evidence behavior rather than real-device compatibility.

Typed outcome or recovery expected

Repeat the same public CLI reproduction. Both invalid-UTF-8 inputs must either receive an explicit input failure or retain their distinct original-file identities with any loaded text representation clearly distinguished. Successful file-source Evidence must not silently attribute the UTF-8 control's digest to either different input file. Keep both valid controls successful with hashes matching their exact bytes; rejecting every script file would not restore the contract.

Analysis or suspected cause

Source-derived reasoning: src/frida/FridaInstrumentationManager.ts:383 reads the file as UTF-8, and src/frida/FridaInstrumentationManager.ts:408 hashes that decoded string before producing the script observation. This explains why different invalid input bytes become replacement characters and share the digest of the valid UTF-8 control. The public CLI reproduction returns a digest different from the selected file for both invalid-UTF-8 fixtures.

Additional context

Related closed #2209 concerns reading an unavailable script file only after starting a Frida session; this report concerns byte identity lost after successfully reading an existing file.
Related closed #2213 concerns missing argument details on rejected Frida input; this report concerns successful Evidence for accepted script files.
Related closed #2529 concerns per-script messages inheriting a session-wide truncation flag; this report concerns script-source digests after UTF-8 decoding.
Related closed #2530 concerns dropped provider-supplied script-error positions; this report concerns selected-file byte identity rather than message metadata.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3396 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `instrument-with-frida --script-file` returns successful Evidence whose `source_path` identifies the selected file but whose `source_sha256` does not identify that file's bytes when the file contains invalid UTF-8. Two files with different original bytes produce the same reported digest after their invalid bytes become replacement characters. The result has no limitation or encoding field that distinguishes the original file from the transformed source delivered for loading. All four public CLI calls exited with status 0 and returned `source_kind: "file"` and `cleanup_error: null`. For every input, the reported digest matched the source received by the simulated native module. It failed to match the original file for both invalid-UTF-8 inputs. This is a decoding transformation, not a SHA-256 collision. The successful Evidence's limitations concerned unavailable target identity and platform metadata; they did not disclose a changed source representation. Separate attempts using the actual native binding did not reach source loading on this host. One startup returned `capability_unavailable` with an access-denial reason; another timed out during startup and reported incomplete cleanup. Those attempts do not establish real native reproduction of the digest mismatch. The complete reproduction below avoids that host boundary and reproduces the mismatch through the production CLI. ### Reproduction 1. Start from a clean checkout at `8dde6370d0809ad04cda3342d4e6f290f48aa063` and prepare the locked dependencies and runtime. These are clean-checkout prerequisites; the reported execution used existing built runtime artifacts and locked dependencies. ```sh npm ci npm run build:cached ``` Run the remaining commands from the repository root. This reproduction substitutes only the native Frida module with a local simulated device. The production CLI, script-file checks, file reader, hashing, service, and Evidence projection execute unchanged. No process is actually attached or instrumented. 2. Create the isolated inputs and native module replacement: ```sh work_dir=$(mktemp -d) mkdir "$work_dir/home" cat > "$work_dir/frida-hook.mjs" <<'JS' import { registerHooks } from 'node:module'; registerHooks({ resolve(specifier, context, next) { if (specifier === 'frida') { return { url: new URL('./frida-fake.mjs', import.meta.url).href, shortCircuit: true }; } return next(specifier, context); } }); JS cat > "$work_dir/frida-fake.mjs" <<'JS' import { writeFile } from 'node:fs/promises'; const signal = () => ({ connect() {}, disconnect() {} }); const session = { detached: signal(), isDetached() { return this.closed === true; }, async detach() { this.closed = true; }, async createScript(source) { await writeFile(new URL('./loaded-source.txt', import.meta.url), source); return { message: signal(), isDestroyed: false, async load() {}, async unload() { this.isDestroyed = true; } }; } }; const device = { id: 'local', name: 'local fixture', type: 'local', async getProcessByPid(pid) { return { pid, name: 'fixture' }; }, async attach() { return session; }, async resume() {}, async spawn() { return 1; } }; export function getDeviceManager() { return { async getDeviceById() { return device; } }; } JS python3 - "$work_dir" <<'PY' from pathlib import Path import sys root = Path(sys.argv[1]) (root / 'sample-ff.js').write_bytes(b'//\xff\nsend("marker");\n') (root / 'sample-fe.js').write_bytes(b'//\xfe\nsend("marker");\n') (root / 'sample-utf8.js').write_bytes('//\ufffd\nsend("marker");\n'.encode()) (root / 'sample-ascii.js').write_bytes(b'//comment\nsend("marker");\n') PY ``` 3. Execute the public command for each input and compare the selected bytes, returned digest, and source received by the simulated native module: ```sh env HOME="$work_dir/home" node --input-type=module - "$work_dir" <<'JS' import { spawnSync } from 'node:child_process'; import { readFileSync } from 'node:fs'; import { createHash } from 'node:crypto'; const root = process.argv[2]; const hash = (path) => createHash('sha256').update(readFileSync(path)).digest('hex'); for (const kind of ['ff', 'fe', 'utf8', 'ascii']) { const path = `${root}/sample-${kind}.js`; const child = spawnSync(process.execPath, [ '--import', `${root}/frida-hook.mjs`, 'scripts/rea.mjs', 'instrument-with-frida', 'attach', '1', '--script-file', path, '--duration-ms', '0', '--format', 'json' ], { env: { ...process.env, HOME: `${root}/home` }, encoding: 'utf8', timeout: 10000 }); if (child.status !== 0) throw new Error(child.stderr + child.stdout); const result = JSON.parse(child.stdout).evidence.normalized_result; const selected = hash(path); console.log(JSON.stringify({ kind, exit_status: child.status, selected_sha256: selected, reported_sha256: result.source_sha256, loaded_sha256: hash(`${root}/loaded-source.txt`), selected_file_matches: selected === result.source_sha256 })); } JS ``` The two invalid bytes occur inside comments. The UTF-8 control contains an actual encoded replacement character in the same comment, and the ASCII control supplies an ordinary comment. ### Expected behavior File-source Evidence should preserve the selected file's SHA-256 identity. If loading requires a transformed text representation, the caller must be able to distinguish that representation from the original file. Successful Evidence should not pair the selected file path with an unqualified digest of different bytes. A file that cannot meet the supported source contract should instead receive an explicit input failure. `docs/frida-instrumentation.md:11` states that script source is not stored in Evidence and that its selected path and SHA-256 digest are retained. `AGENTS.md:13`, under Product and Authority, also requires preserving caller-selected inputs and local evidence, including digests and metadata. ### Impact and scope P2, supported. Successful source Evidence loses the original identity of accepted non-UTF-8 script files. Two distinct selected files become indistinguishable by the reported source digest, which impairs provenance checks. The original files remain unchanged, and the valid UTF-8 and ASCII controls retain correct file hashes. Workaround available. Independently hash and retain the selected file alongside the observation, and compare it with the source hash. The reproduction calculates those distinct original hashes without modifying the inputs. This preserves the missing file identity outside the returned Evidence until the contract is restored. ### REA version 6.4.0 (main at 8dde6370d0809ad04cda3342d4e6f290f48aa063) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64. The reported execution used existing built runtime artifacts and locked dependencies. The module-hook reproduction simulates native Frida and needs no server, target application, or native attach permission. Real native instrumentation was not verified. ### Target and artifact context Four caller-selected JavaScript files contain invalid bytes `0xff` and `0xfe`, an actual UTF-8 replacement character, and an ordinary ASCII comment, respectively. Each comment precedes `send("marker");`. A local simulated native Frida device accepts synthetic PID 1 and records the source delivered for loading. No process is actually attached or instrumented. ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence The observed selected-file and reported digests were: ```text Input Selected-file SHA-256 ff d6c31c67461c0c07b616f441170b4daead0d039fc6fd01a30dcc8cf54fcc487b fe 8940bb2c848dad6af6a83c86fb1c214f83c9a624a07315f4be7a3f4b5297cd21 utf8 dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6 ascii 0fe349f28aa100e026abaa65f52a5df3c79f2493c1793f8eefa149b87730f9af Input Reported source_sha256 ff dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6 fe dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6 utf8 dd405a3895c2261039689c23d60bf5941e46817c0cda22e2acb62d624e190fa6 ascii 0fe349f28aa100e026abaa65f52a5df3c79f2493c1793f8eefa149b87730f9af ``` The CLI result, independently calculated file hashes, and source received by the simulated native module were executed locally. Evidence provenance was observed, local verification reproduced the failure, and reproduction completeness was complete. The real native Frida binding was replaced at its module boundary, so this verifies REA's source acquisition and Evidence behavior rather than real-device compatibility. ### Typed outcome or recovery expected Repeat the same public CLI reproduction. Both invalid-UTF-8 inputs must either receive an explicit input failure or retain their distinct original-file identities with any loaded text representation clearly distinguished. Successful file-source Evidence must not silently attribute the UTF-8 control's digest to either different input file. Keep both valid controls successful with hashes matching their exact bytes; rejecting every script file would not restore the contract. ### Analysis or suspected cause Source-derived reasoning: `src/frida/FridaInstrumentationManager.ts:383` reads the file as UTF-8, and `src/frida/FridaInstrumentationManager.ts:408` hashes that decoded string before producing the script observation. This explains why different invalid input bytes become replacement characters and share the digest of the valid UTF-8 control. The public CLI reproduction returns a digest different from the selected file for both invalid-UTF-8 fixtures. ### Additional context - Related closed [#2209](https://github.com/morluto/rea/issues/2209) concerns reading an unavailable script file only after starting a Frida session; this report concerns byte identity lost after successfully reading an existing file. - Related closed [#2213](https://github.com/morluto/rea/issues/2213) concerns missing argument details on rejected Frida input; this report concerns successful Evidence for accepted script files. - Related closed [#2529](https://github.com/morluto/rea/issues/2529) concerns per-script messages inheriting a session-wide truncation flag; this report concerns script-source digests after UTF-8 decoding. - Related closed [#2530](https://github.com/morluto/rea/issues/2530) concerns dropped provider-supplied script-error positions; this report concerns selected-file byte identity rather than message metadata.

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:09:54 +00:00

[PR #1276] [MERGED] fix(android): support owned JADX stdio on Windows #2759

bsakaguchi closed this issue

2026-10-31 00:30:20 +00:00

bsakaguchi commented

2026-10-31 00:30:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 19, 2026):

Fixed on main in 048d70755e.

The file reader decoded bytes before hashing, so distinct invalid UTF-8 files collapsed to the same identity. It now hashes the selected bytes and decodes that same read. Existing replacement decoding stays accepted; malformed inputs carry a typed producer fact projected to raw_result.source_representation with the loaded-string digest and an explicit limitation. Persistent loads, cancelled partial Evidence and one-shot instrumentation share that projection. Normalized result schemas remain compatible. PR #3507 proposes the separate strict-rejection policy.

Validation: 103 focused tests pass, including public CLI FF/FE, actual U+FFFD and ASCII controls, plus a cancelled native-rejection MCP load retaining both original bytes' identity and decoded representation. npm run check, npm run docs:check and normal hooks pass. Controlled bindings only; real Frida engine/device behavior remains unverified.

Alternating completed CLI workflows (valid and malformed 1 MiB files, three repeated processes per version after warmup) had median elapsed 0.39→0.38s, CPU 0.55→0.53s and process peak RSS 368.2→370.4 MB. Full replies grew from 817→934 estimated cl100k tokens to retain the decoding receipt. These small simulated measurements establish no speedup; true cold runs, actual agent context and real engine costs remain unmeasured.

@morluto commented on GitHub (Oct 19, 2026): Fixed on main in 048d70755ea9ca1d0d159ac78285d62a7ae00b21. The file reader decoded bytes before hashing, so distinct invalid UTF-8 files collapsed to the same identity. It now hashes the selected bytes and decodes that same read. Existing replacement decoding stays accepted; malformed inputs carry a typed producer fact projected to `raw_result.source_representation` with the loaded-string digest and an explicit limitation. Persistent loads, cancelled partial Evidence and one-shot instrumentation share that projection. Normalized result schemas remain compatible. PR #3507 proposes the separate strict-rejection policy. Validation: 103 focused tests pass, including public CLI FF/FE, actual U+FFFD and ASCII controls, plus a cancelled native-rejection MCP load retaining both original bytes' identity and decoded representation. `npm run check`, `npm run docs:check` and normal hooks pass. Controlled bindings only; real Frida engine/device behavior remains unverified. Alternating completed CLI workflows (valid and malformed 1 MiB files, three repeated processes per version after warmup) had median elapsed 0.39→0.38s, CPU 0.55→0.53s and process peak RSS 368.2→370.4 MB. Full replies grew from 817→934 estimated cl100k tokens to retain the decoding receipt. These small simulated measurements establish no speedup; true cold runs, actual agent context and real engine costs remain unmeasured.

bsakaguchi referenced this issue

2026-10-31 00:50:58 +00:00

[GH-ISSUE #7657] [Bug] Aegis latest-release walkthrough omits native Windows x64 setup #4581

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1262

No description provided.

Rows
Columns