[GH-ISSUE #6774] [Bug] compare_functions reports undecodable string displays as unchanged without uncertainty #4131

New issue

Closed

opened 2026-10-31 00:47:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:02 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6774

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The registered compare_functions MCP tool reports strings_names.status: "unchanged" and an overall status: "unchanged" when both input dossiers contain a referenced string with decoding: {available: false, reason: "The native string bytes are unavailable"}. The dimension has no limitations, and the session records no residual unknown. The equal value fields are explicitly unavailable-decoding display text, so they do not establish equal decoded strings. A decoded-string control returns unchanged as expected. A supplemental pair whose decoded text stays equal but whose recorded termination and byte length change also returns unchanged without any string-dimension qualification.

Reproduction

With the source checkout's locked dependencies installed, save the following as probe.mts outside the checkout. This is the executed probe, including its decoded control and supplemental termination pair. It uses the production server, production session and registered MCP handler through the SDK's in-memory transport. No provider observations are fabricated by a mock provider: the inputs are explicitly synthetic, schema-valid normalized Evidence accepted by the public comparison contract. No native analysis provider or target is started.

import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';
const repo = resolve(process.argv[2]);
const load = (path:string) => import(pathToFileURL(resolve(repo,'src',path+'.ts')).href);
const require = createRequire(resolve(repo,'package.json'));
const { Client, InMemoryTransport } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client')).href);
const { BinarySession } = await load('application/binary/BinarySession');
const { SessionProviderRouter } = await load('application/binary/SessionProviderRouter');
const { AnalysisProviderRegistry } = await load('application/binary/AnalysisProviderRegistry');
const { createServer } = await load('server/createServer');
const { createEvidence } = await load('domain/evidence');
const { functionDossierSchema } = await load('domain/hopperValues');
const base = functionDossierSchema.parse({
 procedure:{address:'0x1000',name:'main',classification:null,body:{available:false,reason:'Synthetic fixture does not observe body ranges'},signature:'int main(void)',locals:[]},
 pseudocode:'return 0;',assembly:['ret'],comments:[],callers:[],callees:[],incoming_references:[],outgoing_references:[],referenced_strings:[],referenced_names:[],
 basic_blocks:[{start:'0x1000',end:'0x1001',successors:[]}],native_api:null,native_value_flow:null,
 facet_availability:Object.fromEntries(['pseudocode','assembly','comments','callers','callees','references','strings_names','cfg'].map(x=>[x,{available:true}])),limitations:[]
});
const observe = (digit:string,row:any) => createEvidence({path:'/tmp/synthetic-function-'+digit,sha256:digit.repeat(64),format:'mach-o'},
 {id:'synthetic',name:'Synthetic normalized observations',version:'1'},
 {predicateType:'rea.analysis',operation:'analyze_function',parameters:{procedure:'main'},result:{...base,referenced_strings:[{address:'0x2000',source_address:'0x1000',...row}]},confidence:'observed',authority:'shipped-artifact',relationshipCoverage:{status:'complete'}});
const good={value:'hello',provider_value:'hello',string:{encoding:'utf-8',encoding_status:'observed',termination:'present_or_not_required',byte_length:6}};
const unknown={value:'display preview',provider_value:'display preview',decoding:{available:false,reason:'The native string bytes are unavailable'}};
const cases=[['decoded-control',observe('1',good),observe('2',good)],['undecoded-display',observe('3',unknown),observe('4',unknown)],['termination-change',observe('5',good),observe('6',{...good,string:{...good.string,termination:'missing',byte_length:5}})]];
const session = new BinarySession(SessionProviderRouter.selectable(new AnalysisProviderRegistry([]),[]));
const server = createServer({kind:'session',session});
const client = new Client({name:'synthetic-comparison-research',version:'1'});
const [ct,st]=InMemoryTransport.createLinkedPair();
try{
 await server.connect(st); await client.connect(ct);
 for (const [name,left,right] of cases) {
  const reply=await client.callTool({name:'compare_functions',arguments:{left,right}});
  const result=(reply.structuredContent as any)?.normalized_result;
  console.log(JSON.stringify({name,isError:reply.isError??false,status:result?.status,strings_names:result?.dimensions?.find((x:any)=>x.dimension==='strings_names'),error:result===undefined?reply:undefined}));
 }
 console.log(JSON.stringify({unknowns:session.listUnknowns({domain:'function-comparison'}).length,node:process.version,platform:process.platform,arch:process.arch}));
}finally{await Promise.allSettled([client.close(),server.close(),session.close()]);}

Run from the checkout with the pinned Node runtime selected on PATH, replacing only the two example script paths:

repo="$PWD"
work="$(mktemp -d)"
mkdir -p "$work/home" "$work/config" "$work/cache"
HOME="$work/home" XDG_CONFIG_HOME="$work/config" XDG_CACHE_HOME="$work/cache" \
  node --import "$repo/node_modules/tsx/dist/loader.mjs" /path/to/probe.mts "$repo"

The probe ran once, with one call for each of the three cases, and exited 0. No probe was rerun after this draft was written.

Expected behavior

A string explicitly marked undecodable must retain that uncertainty when consumed by comparison. Equal provider display previews must not become an unqualified equality claim about the referenced strings or the whole function. The string schema says that value retains native display text when typed bytes could not be decoded. The native-analysis guide documents this uncertainty and says function dossiers preserve the same string evidence. The comparison handoff accepts complete analyze_function Evidence directly.

Impact and scope

The observed consequence is a false unqualified equality result for an accepted normalized observation with unavailable string decoding. Source inputs remain available, but an analyst or downstream workflow relying on the comparison does not see the missing fact and receives no residual unknown. The probe establishes this result for synthetic normalized Evidence through the production MCP handler; it does not establish frequency with real Hopper targets or effects in a reconstruction verifier. Workaround: callers can inspect the original referenced_strings rows for decoding.available: false and preserve that uncertainty themselves instead of treating the returned equality as established; the executed input objects expose that flag and reason.

REA version

REA 6.7.0 at source revision 99d2d7ad22bcbd780919760008aba6a88e7d1077, using locked dependencies; 6.7.0 was the latest npm release checked for this investigation.

Runtime and platform

Node.js v24.22.1; Linux x64; repository-pinned tsx loader. No native provider was used.

Target and artifact context

Synthetic normalized Mach-O function Evidence for the same named function and relative string-reference site in two distinct artifact identities. All comparison facets are declared available, relationship coverage is complete, and only the string observations differ between the control and failure cases. The /tmp/synthetic-function-* paths are synthetic subject labels and are never opened.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: The typed string contract and native-analysis guide linked above distinguish undecodable display text from decoded string contents.
Failure source: Executed client.callTool({name: 'compare_functions', arguments: {left, right}}) responses from the registered production handler.
Evidence provenance: observed; source-derived reasoning is labeled below.
Local verification: reproduced.
Reproduction completeness: complete for the synthetic normalized-Evidence MCP workflow; no real-provider reproduction is claimed.

The smallest relevant projection of the actual successful replies was:

{"name":"decoded-control","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804","right_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804"}}
{"name":"undecoded-display","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"976011ffbb6280b19a2d62e3a924cf4f012fe9c393faf9cab0e233f6a4798ada","right_digest":"976011ffbb6280b19a2d62e3a924cf4f012fe9c393faf9cab0e233f6a4798ada"}}
{"name":"termination-change","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804","right_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804"}}
{"unknowns":0,"node":"v24.22.1","platform":"linux","arch":"x64"}

The equal dimension digests for the decoded control and termination-change pair additionally show that these typed qualifications do not participate in the comparison. This supplemental observation is not a claim that different string encodings necessarily establish different program behavior.

Typed outcome or recovery expected

For undecoded-display, preserve the decoding-unavailable reason and leave string-content equality unresolved; with every other dimension unchanged, the overall result must not claim unqualified equality. Restoration criterion: the unchanged probe retains its successful decoded control while the unavailable-decoding case returns an unknown or otherwise explicitly limited string comparison and corresponding overall uncertainty. A fix that merely omits the string rows would not restore the uncertainty contract.

Analysis or suspected cause

Source-derived: stringAndNameProjection projects each referenced string to its source offset and value, discarding decoding and string qualifications. compareStringsAndNames evaluates Swift-literal coverage but never checks ordinary string-decoding availability. The facet availability check concerns whether the collection was observed; it does not restore each row's missing decoding fact.

Additional context

Related #5503 concerns trace_feature dropping qualifications while projecting search matches. This report concerns the separate function-comparison normalization and its false equality conclusion; restoring tracing alone does not restore this comparison. Searches across open and closed issues for compare_functions with decoding, strings_names, encoding, and stringAndNameProjection found no matching report. No last-known-good version was established.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: An accepted synthetic MCP comparison produces an unqualified equality conclusion despite explicit unavailable string decoding; the demonstrated effect is incorrect analysis evidence, without observed severe downstream harm.
Workaround status: available
Workaround basis: The supplied original Evidence retains each string's decoding-unavailable flag and reason, so callers can preserve that uncertainty themselves instead of trusting the comparison's equality conclusion.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6774 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The registered `compare_functions` MCP tool reports `strings_names.status: "unchanged"` and an overall `status: "unchanged"` when both input dossiers contain a referenced string with `decoding: {available: false, reason: "The native string bytes are unavailable"}`. The dimension has no limitations, and the session records no residual unknown. The equal `value` fields are explicitly unavailable-decoding display text, so they do not establish equal decoded strings. A decoded-string control returns unchanged as expected. A supplemental pair whose decoded text stays equal but whose recorded termination and byte length change also returns unchanged without any string-dimension qualification. ## Reproduction With the source checkout's locked dependencies installed, save the following as `probe.mts` outside the checkout. This is the executed probe, including its decoded control and supplemental termination pair. It uses the production server, production session and registered MCP handler through the SDK's in-memory transport. No provider observations are fabricated by a mock provider: the inputs are explicitly synthetic, schema-valid normalized Evidence accepted by the public comparison contract. No native analysis provider or target is started. ```typescript import { createRequire } from 'node:module'; import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; const repo = resolve(process.argv[2]); const load = (path:string) => import(pathToFileURL(resolve(repo,'src',path+'.ts')).href); const require = createRequire(resolve(repo,'package.json')); const { Client, InMemoryTransport } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client')).href); const { BinarySession } = await load('application/binary/BinarySession'); const { SessionProviderRouter } = await load('application/binary/SessionProviderRouter'); const { AnalysisProviderRegistry } = await load('application/binary/AnalysisProviderRegistry'); const { createServer } = await load('server/createServer'); const { createEvidence } = await load('domain/evidence'); const { functionDossierSchema } = await load('domain/hopperValues'); const base = functionDossierSchema.parse({ procedure:{address:'0x1000',name:'main',classification:null,body:{available:false,reason:'Synthetic fixture does not observe body ranges'},signature:'int main(void)',locals:[]}, pseudocode:'return 0;',assembly:['ret'],comments:[],callers:[],callees:[],incoming_references:[],outgoing_references:[],referenced_strings:[],referenced_names:[], basic_blocks:[{start:'0x1000',end:'0x1001',successors:[]}],native_api:null,native_value_flow:null, facet_availability:Object.fromEntries(['pseudocode','assembly','comments','callers','callees','references','strings_names','cfg'].map(x=>[x,{available:true}])),limitations:[] }); const observe = (digit:string,row:any) => createEvidence({path:'/tmp/synthetic-function-'+digit,sha256:digit.repeat(64),format:'mach-o'}, {id:'synthetic',name:'Synthetic normalized observations',version:'1'}, {predicateType:'rea.analysis',operation:'analyze_function',parameters:{procedure:'main'},result:{...base,referenced_strings:[{address:'0x2000',source_address:'0x1000',...row}]},confidence:'observed',authority:'shipped-artifact',relationshipCoverage:{status:'complete'}}); const good={value:'hello',provider_value:'hello',string:{encoding:'utf-8',encoding_status:'observed',termination:'present_or_not_required',byte_length:6}}; const unknown={value:'display preview',provider_value:'display preview',decoding:{available:false,reason:'The native string bytes are unavailable'}}; const cases=[['decoded-control',observe('1',good),observe('2',good)],['undecoded-display',observe('3',unknown),observe('4',unknown)],['termination-change',observe('5',good),observe('6',{...good,string:{...good.string,termination:'missing',byte_length:5}})]]; const session = new BinarySession(SessionProviderRouter.selectable(new AnalysisProviderRegistry([]),[])); const server = createServer({kind:'session',session}); const client = new Client({name:'synthetic-comparison-research',version:'1'}); const [ct,st]=InMemoryTransport.createLinkedPair(); try{ await server.connect(st); await client.connect(ct); for (const [name,left,right] of cases) { const reply=await client.callTool({name:'compare_functions',arguments:{left,right}}); const result=(reply.structuredContent as any)?.normalized_result; console.log(JSON.stringify({name,isError:reply.isError??false,status:result?.status,strings_names:result?.dimensions?.find((x:any)=>x.dimension==='strings_names'),error:result===undefined?reply:undefined})); } console.log(JSON.stringify({unknowns:session.listUnknowns({domain:'function-comparison'}).length,node:process.version,platform:process.platform,arch:process.arch})); }finally{await Promise.allSettled([client.close(),server.close(),session.close()]);} ``` Run from the checkout with the pinned Node runtime selected on `PATH`, replacing only the two example script paths: ```sh repo="$PWD" work="$(mktemp -d)" mkdir -p "$work/home" "$work/config" "$work/cache" HOME="$work/home" XDG_CONFIG_HOME="$work/config" XDG_CACHE_HOME="$work/cache" \ node --import "$repo/node_modules/tsx/dist/loader.mjs" /path/to/probe.mts "$repo" ``` The probe ran once, with one call for each of the three cases, and exited 0. No probe was rerun after this draft was written. ## Expected behavior A string explicitly marked undecodable must retain that uncertainty when consumed by comparison. Equal provider display previews must not become an unqualified equality claim about the referenced strings or the whole function. The [string schema](https://github.com/morluto/rea/blob/99d2d7ad22bcbd780919760008aba6a88e7d1077/src/domain/hopperValues.ts#L59-L86) says that `value` retains native display text when typed bytes could not be decoded. The [native-analysis guide](https://github.com/morluto/rea/blob/99d2d7ad22bcbd780919760008aba6a88e7d1077/docs/native-investigation.md#L203-L210) documents this uncertainty and says function dossiers preserve the same string evidence. The [comparison handoff](https://github.com/morluto/rea/blob/99d2d7ad22bcbd780919760008aba6a88e7d1077/docs/mcp-contracts.md#L16-L26) accepts complete `analyze_function` Evidence directly. ## Impact and scope The observed consequence is a false unqualified equality result for an accepted normalized observation with unavailable string decoding. Source inputs remain available, but an analyst or downstream workflow relying on the comparison does not see the missing fact and receives no residual unknown. The probe establishes this result for synthetic normalized Evidence through the production MCP handler; it does not establish frequency with real Hopper targets or effects in a reconstruction verifier. Workaround: callers can inspect the original `referenced_strings` rows for `decoding.available: false` and preserve that uncertainty themselves instead of treating the returned equality as established; the executed input objects expose that flag and reason. ## REA version REA 6.7.0 at source revision `99d2d7ad22bcbd780919760008aba6a88e7d1077`, using locked dependencies; 6.7.0 was the latest npm release checked for this investigation. ## Runtime and platform Node.js v24.22.1; Linux x64; repository-pinned tsx loader. No native provider was used. ## Target and artifact context Synthetic normalized Mach-O function Evidence for the same named function and relative string-reference site in two distinct artifact identities. All comparison facets are declared available, relationship coverage is complete, and only the string observations differ between the control and failure cases. The `/tmp/synthetic-function-*` paths are synthetic subject labels and are never opened. ## Interface MCP ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence - Expected source: The typed string contract and native-analysis guide linked above distinguish undecodable display text from decoded string contents. - Failure source: Executed `client.callTool({name: 'compare_functions', arguments: {left, right}})` responses from the registered production handler. - Evidence provenance: observed; source-derived reasoning is labeled below. - Local verification: reproduced. - Reproduction completeness: complete for the synthetic normalized-Evidence MCP workflow; no real-provider reproduction is claimed. The smallest relevant projection of the actual successful replies was: ```json {"name":"decoded-control","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804","right_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804"}} {"name":"undecoded-display","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"976011ffbb6280b19a2d62e3a924cf4f012fe9c393faf9cab0e233f6a4798ada","right_digest":"976011ffbb6280b19a2d62e3a924cf4f012fe9c393faf9cab0e233f6a4798ada"}} {"name":"termination-change","isError":false,"status":"unchanged","strings_names":{"status":"unchanged","limitations":[],"left_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804","right_digest":"eb9a852edb5471ab448647de2179b79d8575ba798864f67361a4fb40771ff804"}} {"unknowns":0,"node":"v24.22.1","platform":"linux","arch":"x64"} ``` The equal dimension digests for the decoded control and termination-change pair additionally show that these typed qualifications do not participate in the comparison. This supplemental observation is not a claim that different string encodings necessarily establish different program behavior. ## Typed outcome or recovery expected For `undecoded-display`, preserve the decoding-unavailable reason and leave string-content equality unresolved; with every other dimension unchanged, the overall result must not claim unqualified equality. Restoration criterion: the unchanged probe retains its successful decoded control while the unavailable-decoding case returns an unknown or otherwise explicitly limited string comparison and corresponding overall uncertainty. A fix that merely omits the string rows would not restore the uncertainty contract. ## Analysis or suspected cause Source-derived: [stringAndNameProjection](https://github.com/morluto/rea/blob/99d2d7ad22bcbd780919760008aba6a88e7d1077/src/domain/functionComparisonNormalization.ts#L79-L109) projects each referenced string to its source offset and `value`, discarding `decoding` and `string` qualifications. [compareStringsAndNames](https://github.com/morluto/rea/blob/99d2d7ad22bcbd780919760008aba6a88e7d1077/src/domain/functionComparisonDimensions.ts#L200-L243) evaluates Swift-literal coverage but never checks ordinary string-decoding availability. The facet availability check concerns whether the collection was observed; it does not restore each row's missing decoding fact. ## Additional context Related [#5503](https://github.com/morluto/rea/issues/5503) concerns `trace_feature` dropping qualifications while projecting search matches. This report concerns the separate function-comparison normalization and its false equality conclusion; restoring tracing alone does not restore this comparison. Searches across open and closed issues for `compare_functions` with `decoding`, `strings_names`, `encoding`, and `stringAndNameProjection` found no matching report. No last-known-good version was established. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: An accepted synthetic MCP comparison produces an unqualified equality conclusion despite explicit unavailable string decoding; the demonstrated effect is incorrect analysis evidence, without observed severe downstream harm. - Workaround status: available - Workaround basis: The supplied original Evidence retains each string's decoding-unavailable flag and reason, so callers can preserve that uncertainty themselves instead of trusting the comparison's equality conclusion.

bsakaguchi closed this issue

2026-10-31 00:47:02 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:41 +00:00

[PR #4131] [MERGED] fix(artifacts): never treat a directory named *.map as a source map #6076