[GH-ISSUE #6693] [Bug] Sigrok Evidence silently replaces invalid UTF-8 and loses encoding warnings #4066

New issue

Closed

opened 2026-10-31 00:46:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:32 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6693

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production sigrok provider/service returns successful confidence: "observed" Evidence after replacing malformed producer bytes with U+FFFD. Three distinct stdout byte sequences—FF, FE, and valid UTF-8 EF BF BD—produce identical annotation text and Evidence IDs. Evidence contains no encoding-loss warning or retained original bytes. Decoder catalog names and help lines also lose the warning.

This is a production-boundary reproduction with a simulated sigrok executable, real process supervision, real Linux prlimit, and isolated home configuration. It does not establish that stock sigrok emits malformed text.

Reproduction

From the tested checkout with its locked dependencies already installed, run work=$(mktemp -d), save this as $work/repro.mts, then run the command below. The fixture emits a supported version banner but deliberately substitutes fixed output for decoding.

import { chmod, mkdir, writeFile } from 'node:fs/promises';
import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';
const root = resolve(process.argv[2]);
await mkdir(root, { recursive: true });
const load = (file: string) => import(pathToFileURL(resolve(file)).href);
const { SigrokRecordingProvider } = await load('src/signals/SigrokRecordingProvider.ts');
const { SignalRecordingService } = await load('src/application/signals/SignalRecordingService.ts');
await writeFile(root+'/producer', `#!/usr/bin/python3
import os,sys
if '--version' in sys.argv:
    os.write(1,b'sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3\\n')
else:
    token=bytes.fromhex(os.environ['FIXTURE_HEX'])
    os.write(1,b'10-20 uart-1: '+token+b'\\n')
`);
await chmod(root+'/producer', 0o700);
await writeFile(root+'/input.vcd', '$comment benign local fixture $end\n');
for (const hex of ['ff','fe','efbfbd']) {
  const service = new SignalRecordingService(new SigrokRecordingProvider({
    PATH: process.env.PATH, HOME: root, XDG_CONFIG_HOME: root+'/.config',
    REA_SIGROK_COMMAND: root+'/producer', FIXTURE_HEX: hex,
  }));
  try {
    const r = await service.decode({ path: root+'/input.vcd', decoders: 'uart', line_limit: null });
    if (!r.ok) throw r.error;
    const e = r.value;
    console.log(JSON.stringify({hex,ok:r.ok,confidence:e.confidence,text:e.normalized_result.annotations_text,id:e.evidence_id,encodingWarning:/invalidUtf8|invalid UTF-?8|encoding|replacement/i.test(JSON.stringify(e))}));
  } finally { await service.close(); }
}

node --import ./node_modules/tsx/dist/loader.mjs "$work/repro.mts" "$work"

Expected behavior

Preserve the distinction between malformed bytes and a literal U+FFFD. If the adapter requires UTF-8, return a typed unreadable-output result; alternatively retain source bytes and explicitly report encoding loss. Do not silently describe substituted text as verbatim observation.

The pinned recording contract promises unchanged selected text and verbatim retention. The process snapshot contract explicitly identifies invalidUtf8 as inability to represent retained bytes exactly.

Impact and scope

Reproduced for both invalid byte values in annotations, catalog names, and decoder help. The literal replacement-character control remains indistinguishable. Only the returned observation loses data; the recording remains unchanged. Stock-provider incidence, stderr/version-banner behavior, and public CLI/MCP transport were not tested. Normal UTF-8 paging preserved CRLF, blank lines, and non-ASCII text; 156 focused paging comparisons passed.

REA version

Package 6.7.0; source main 8c4f96ef4759fd78da66c4b9726825ca71d7ee0c.

Runtime and platform

Node.js v26.12.1; Linux x86_64. No Hopper/Ghidra or real sigrok installation used.

Target and artifact context

Benign local recording fixture; controlled executable output containing malformed UTF-8, compared with literal U+FFFD.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

Actual minimal reproduction output:

{"hex":"ff","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false}
{"hex":"fe","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false}
{"hex":"efbfbd","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false}

An additional boundary probe confirmed stdout.invalidUtf8: true for the malformed bytes before the sigrok adapter discarded the flag. Its annotation, catalog, and help Evidence contained no encoding warning.

Typed outcome or recovery expected

An encoding-aware output failure or explicitly qualified, recoverable partial observation; valid literal U+FFFD must remain distinguishable from replacement caused by decoding failure.

Analysis or suspected cause

Source-derived: ProviderProcess detects invalid UTF-8 while creating replacement-decoded text. SigrokRecordingProvider copies only text into captured output; report construction likewise consumes the text without checking the flag.

Additional context

Searched open and closed sigrok/decoder reports plus sigrok UTF-8, invalidUtf8, and producer replacement reports. Related #5984 concerns failed version probes; #6671 concerns missing channel targets. Neither describes encoding loss. No fixes, builds, installations, or whole test suites performed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The observed production boundary silently changes producer output and collapses distinguishable observations. The defect is bounded to malformed text; source recordings remain intact, and stock-provider frequency is unknown.
Workaround status: unverified
Workaround basis: Saving direct sigrok stdout separately could preserve original bytes, but no real sigrok installation was available to verify that workflow.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6693 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The production sigrok provider/service returns successful `confidence: "observed"` Evidence after replacing malformed producer bytes with U+FFFD. Three distinct stdout byte sequences—`FF`, `FE`, and valid UTF-8 `EF BF BD`—produce identical annotation text and Evidence IDs. Evidence contains no encoding-loss warning or retained original bytes. Decoder catalog names and help lines also lose the warning. This is a production-boundary reproduction with a simulated sigrok executable, real process supervision, real Linux `prlimit`, and isolated home configuration. It does not establish that stock sigrok emits malformed text. ### Reproduction From the tested checkout with its locked dependencies already installed, run `work=$(mktemp -d)`, save this as `$work/repro.mts`, then run the command below. The fixture emits a supported version banner but deliberately substitutes fixed output for decoding. ```ts import { chmod, mkdir, writeFile } from 'node:fs/promises'; import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; const root = resolve(process.argv[2]); await mkdir(root, { recursive: true }); const load = (file: string) => import(pathToFileURL(resolve(file)).href); const { SigrokRecordingProvider } = await load('src/signals/SigrokRecordingProvider.ts'); const { SignalRecordingService } = await load('src/application/signals/SignalRecordingService.ts'); await writeFile(root+'/producer', `#!/usr/bin/python3 import os,sys if '--version' in sys.argv: os.write(1,b'sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3\\n') else: token=bytes.fromhex(os.environ['FIXTURE_HEX']) os.write(1,b'10-20 uart-1: '+token+b'\\n') `); await chmod(root+'/producer', 0o700); await writeFile(root+'/input.vcd', '$comment benign local fixture $end\n'); for (const hex of ['ff','fe','efbfbd']) { const service = new SignalRecordingService(new SigrokRecordingProvider({ PATH: process.env.PATH, HOME: root, XDG_CONFIG_HOME: root+'/.config', REA_SIGROK_COMMAND: root+'/producer', FIXTURE_HEX: hex, })); try { const r = await service.decode({ path: root+'/input.vcd', decoders: 'uart', line_limit: null }); if (!r.ok) throw r.error; const e = r.value; console.log(JSON.stringify({hex,ok:r.ok,confidence:e.confidence,text:e.normalized_result.annotations_text,id:e.evidence_id,encodingWarning:/invalidUtf8|invalid UTF-?8|encoding|replacement/i.test(JSON.stringify(e))})); } finally { await service.close(); } } ``` ```sh node --import ./node_modules/tsx/dist/loader.mjs "$work/repro.mts" "$work" ``` ### Expected behavior Preserve the distinction between malformed bytes and a literal U+FFFD. If the adapter requires UTF-8, return a typed unreadable-output result; alternatively retain source bytes and explicitly report encoding loss. Do not silently describe substituted text as verbatim observation. The pinned [recording contract](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/docs/signal-recordings.md#L46-L68) promises unchanged selected text and verbatim retention. The [process snapshot contract](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/process/ProviderProcess.ts#L108-L127) explicitly identifies `invalidUtf8` as inability to represent retained bytes exactly. ### Impact and scope Reproduced for both invalid byte values in annotations, catalog names, and decoder help. The literal replacement-character control remains indistinguishable. Only the returned observation loses data; the recording remains unchanged. Stock-provider incidence, stderr/version-banner behavior, and public CLI/MCP transport were not tested. Normal UTF-8 paging preserved CRLF, blank lines, and non-ASCII text; 156 focused paging comparisons passed. ### REA version Package 6.7.0; source main `8c4f96ef4759fd78da66c4b9726825ca71d7ee0c`. ### Runtime and platform Node.js v26.12.1; Linux x86_64. No Hopper/Ghidra or real sigrok installation used. ### Target and artifact context Benign local recording fixture; controlled executable output containing malformed UTF-8, compared with literal U+FFFD. ### Interface Other ### Earliest failure stage Capture or observation ### Relevant output or evidence Actual minimal reproduction output: ```jsonl {"hex":"ff","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false} {"hex":"fe","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false} {"hex":"efbfbd","ok":true,"confidence":"observed","text":"10-20 uart-1: �\n","id":"ev_da79f42bea46d057ec5b06f5cc1a7302af2823d5233bc351ab839d65db5176ea","encodingWarning":false} ``` An additional boundary probe confirmed `stdout.invalidUtf8: true` for the malformed bytes before the sigrok adapter discarded the flag. Its annotation, catalog, and help Evidence contained no encoding warning. ### Typed outcome or recovery expected An encoding-aware output failure or explicitly qualified, recoverable partial observation; valid literal U+FFFD must remain distinguishable from replacement caused by decoding failure. ### Analysis or suspected cause Source-derived: [ProviderProcess](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/process/ProviderProcess.ts#L875-L890) detects invalid UTF-8 while creating replacement-decoded text. [SigrokRecordingProvider](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/signals/SigrokRecordingProvider.ts#L212-L218) copies only text into captured output; report construction likewise consumes the text without checking the flag. ### Additional context Searched open and closed sigrok/decoder reports plus `sigrok UTF-8`, `invalidUtf8`, and producer replacement reports. Related #5984 concerns failed version probes; #6671 concerns missing channel targets. Neither describes encoding loss. No fixes, builds, installations, or whole test suites performed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The observed production boundary silently changes producer output and collapses distinguishable observations. The defect is bounded to malformed text; source recordings remain intact, and stock-provider frequency is unknown. - Workaround status: unverified - Workaround basis: Saving direct sigrok stdout separately could preserve original bytes, but no real sigrok installation was available to verify that workflow.

bsakaguchi closed this issue

2026-10-31 00:46:32 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:20 +00:00

[PR #4066] [CLOSED] fix(kuna): run the inventory with the configured decode lanes #6033