morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #5109] [Bug] Late-cancelled document selection drops the known analysis profile from Evidence #3147

New issue

Closed

opened 2026-10-31 00:37:10 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 00:37:10 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5109

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Executed the shared binary_overview application workflow with a mocked provider boundary. After inventory-summary capability fallback, the provider returns a valid current_document execution with a concrete analysis profile and aborts the signal immediately before returning. The resulting AnalysisCancelledError.partialObservation retains the selected document but has analysis_profile: null. A control projecting the same observation with its profile preserves the profile and has a different Evidence ID.

Reproduction

From a source checkout at the revision below with its locked dependencies available, save this as $WORK/probe.mts in a disposable directory. It uses only authored in-memory observations and starts no analysis engine.

import assert from 'node:assert/strict';
import { pathToFileURL } from 'node:url';
const repo = process.argv[2];
const load = (file: string) => import(pathToFileURL(`${repo}/${file}`).href);
const { EnhancedTools } = await load('src/application/EnhancedTools.ts');
const { createAnalysisExecution } = await load('src/application/AnalysisProvider.ts');
const { createAnalysisProfile } = await load('src/domain/analysisProfile.ts');
const { createEvidence } = await load('src/domain/evidence.ts');
const { AnalysisCapabilityUnavailableError, AnalysisCancelledError } = await load('src/domain/analysisErrorCore.ts');
const { ok, err } = await load('src/domain/result.ts');
const provider = { id: 'controlled', name: 'Controlled provider', version: '1' };
const profile = createAnalysisProfile(provider, { mode: 'fixture' });
const execution = createAnalysisExecution('selected', provider, { analysisProfile: profile });
const control = createEvidence(undefined, provider, {
  operation: 'current_document', parameters: {}, result: execution.result,
  rawResult: execution.rawResult, limitations: execution.limitations,
  locations: execution.locations, confidence: 'observed', analysisProfile: profile,
});
assert.deepEqual(control.analysis_profile, profile);
const controller = new AbortController();
const reason = { request: 'late selection' };
const calls: string[] = [];
const tools = new EnhancedTools({ execute: async (operation: string) => {
  calls.push(operation);
  if (operation === 'read_binary_inventory_summary')
    return err(new AnalysisCapabilityUnavailableError(provider.id, operation, 'Use document fallback'));
  assert.equal(operation, 'current_document');
  controller.abort(reason);
  return ok(execution);
}});
const result = await tools.execute('binary_overview', {}, controller.signal);
assert.equal(result.ok, false);
assert.ok(result.error instanceof AnalysisCancelledError);
assert.equal(result.error.cause, reason);
const partial = result.error.partialObservation;
assert.ok(partial);
assert.equal(partial.normalized_result, 'selected');
assert.equal(partial.analysis_profile, null);
assert.notEqual(partial.evidence_id, control.evidence_id);
assert.deepEqual(calls, ['read_binary_inventory_summary', 'current_document']);
console.log(JSON.stringify({ calls, error: result.error._tag,
  execution_profile_present: !!execution.analysisProfile,
  control_profile_present: !!control.analysis_profile,
  partial_profile: partial.analysis_profile,
  retained_result: partial.normalized_result,
  evidence_ids_differ: partial.evidence_id !== control.evidence_id }, null, 2));

Run from the checkout root, with $WORK naming the disposable directory:

mkdir -p "$WORK/home"
env -i HOME="$WORK/home" PATH="$PATH" node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$PWD"

Two executions reproduced the profile loss; the final run used the empty environment shown above. Assertions characterize the current defective behavior.

Expected behavior

The retained Evidence should preserve the valid profile already attached to the completed provider execution. The repository's provenance invariants require preserving identity and source facts across boundaries. The execution-to-Evidence contract promises identical completed facts, and MCP contracts establish that analysis_profile participates in semantic identity. Null remains appropriate when the producer has no profile.

Impact and scope

Observed bounded loss of analysis-configuration provenance and a different semantic Evidence ID for an otherwise identical completed observation. The selected document and cancellation error remain available. This is a mocked application-boundary reproduction; NOT_ENOUGH_INFO for naturally occurring cancellation frequency, real-provider behavior, compiled CLI/MCP delivery or a practical caller workflow restoring the missing profile.

REA version

Source package version 6.7.0, commit 3537e58c1ab29309f06eabf50bc52adba9817a0c. Relevant source files match tag rea-agents-6.7.0; the published package was not executed.

Runtime and platform

Node.js v26.12.0; macOS, darwin arm64; kernel 27.0.0. No optional provider application was run.

Target and artifact context

No binary or provider document was opened. The fixture supplies an in-memory selected-document string and a valid matching profile generated by createAnalysisProfile.

Interface

Other

Shared application workflow, EnhancedTools.execute('binary_overview', ...).

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Observed output from the application-boundary probe:

{
  "calls": [
    "read_binary_inventory_summary",
    "current_document"
  ],
  "error": "AnalysisCancelledError",
  "execution_profile_present": true,
  "control_profile_present": true,
  "partial_profile": null,
  "retained_result": "selected",
  "evidence_ids_differ": true
}

The probe asserts that exactly the inventory-summary and current-document operations ran, that the caller's cancellation reason survives by identity, and that the partial observation retains the result. The profile-preserving control uses the same result, provider, operation, parameters and other projection fields.

Typed outcome or recovery expected

Return the cancellation error with the same reason and completed observation, carrying the execution's original analysis profile and digest in partialObservation.analysis_profile.

Analysis or suspected cause

Source-derived cause: the inventory fallback reaches late-selection completion, which uses createAnalysisExecutionEvidence. That projector omits analysisProfile; Evidence normalization defaults the omission to null. The ordinary official MCP projector explicitly forwards profiles. Hopper's producer projection can supply them, but no real Hopper execution was performed.

Additional context

Related closed #4934 covers cancellation/fallback classification and retaining completed selection observations; this reproduction reaches the retained observation but finds its known profile omitted. Closed #3992 covers a separate IDA partial-Evidence constructor after failed decompilation. No matching report was found in the searched open/closed snapshot. Last known good version and first affected release: NOT_ENOUGH_INFO.

Freshness review against 29910d16b98fe4b607a7aa15928789a73614b0b9: the intervening EnhancedTools edits concern call-graph direction selection. The binary_overview fallback, late-cancellation selection path and execution-to-Evidence projector remain unchanged, including the omitted analysis profile. This is a source freshness check, not a new application, compiled-adapter or real-provider runtime reproduction. The original tested SHA and runtime scope above remain unchanged.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5109 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Executed the shared `binary_overview` application workflow with a mocked provider boundary. After inventory-summary capability fallback, the provider returns a valid `current_document` execution with a concrete analysis profile and aborts the signal immediately before returning. The resulting `AnalysisCancelledError.partialObservation` retains the selected document but has `analysis_profile: null`. A control projecting the same observation with its profile preserves the profile and has a different Evidence ID. ## Reproduction From a source checkout at the revision below with its locked dependencies available, save this as `$WORK/probe.mts` in a disposable directory. It uses only authored in-memory observations and starts no analysis engine. ```ts import assert from 'node:assert/strict'; import { pathToFileURL } from 'node:url'; const repo = process.argv[2]; const load = (file: string) => import(pathToFileURL(`${repo}/${file}`).href); const { EnhancedTools } = await load('src/application/EnhancedTools.ts'); const { createAnalysisExecution } = await load('src/application/AnalysisProvider.ts'); const { createAnalysisProfile } = await load('src/domain/analysisProfile.ts'); const { createEvidence } = await load('src/domain/evidence.ts'); const { AnalysisCapabilityUnavailableError, AnalysisCancelledError } = await load('src/domain/analysisErrorCore.ts'); const { ok, err } = await load('src/domain/result.ts'); const provider = { id: 'controlled', name: 'Controlled provider', version: '1' }; const profile = createAnalysisProfile(provider, { mode: 'fixture' }); const execution = createAnalysisExecution('selected', provider, { analysisProfile: profile }); const control = createEvidence(undefined, provider, { operation: 'current_document', parameters: {}, result: execution.result, rawResult: execution.rawResult, limitations: execution.limitations, locations: execution.locations, confidence: 'observed', analysisProfile: profile, }); assert.deepEqual(control.analysis_profile, profile); const controller = new AbortController(); const reason = { request: 'late selection' }; const calls: string[] = []; const tools = new EnhancedTools({ execute: async (operation: string) => { calls.push(operation); if (operation === 'read_binary_inventory_summary') return err(new AnalysisCapabilityUnavailableError(provider.id, operation, 'Use document fallback')); assert.equal(operation, 'current_document'); controller.abort(reason); return ok(execution); }}); const result = await tools.execute('binary_overview', {}, controller.signal); assert.equal(result.ok, false); assert.ok(result.error instanceof AnalysisCancelledError); assert.equal(result.error.cause, reason); const partial = result.error.partialObservation; assert.ok(partial); assert.equal(partial.normalized_result, 'selected'); assert.equal(partial.analysis_profile, null); assert.notEqual(partial.evidence_id, control.evidence_id); assert.deepEqual(calls, ['read_binary_inventory_summary', 'current_document']); console.log(JSON.stringify({ calls, error: result.error._tag, execution_profile_present: !!execution.analysisProfile, control_profile_present: !!control.analysis_profile, partial_profile: partial.analysis_profile, retained_result: partial.normalized_result, evidence_ids_differ: partial.evidence_id !== control.evidence_id }, null, 2)); ``` Run from the checkout root, with `$WORK` naming the disposable directory: ```sh mkdir -p "$WORK/home" env -i HOME="$WORK/home" PATH="$PATH" node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$PWD" ``` Two executions reproduced the profile loss; the final run used the empty environment shown above. Assertions characterize the current defective behavior. ## Expected behavior The retained Evidence should preserve the valid profile already attached to the completed provider execution. The repository's [provenance invariants](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/AGENTS.md#L22-L28) require preserving identity and source facts across boundaries. The [execution-to-Evidence contract](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/AnalysisProvider.ts#L130-L148) promises identical completed facts, and [MCP contracts](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/docs/mcp-contracts.md#L11-L18) establish that analysis_profile participates in semantic identity. Null remains appropriate when the producer has no profile. ## Impact and scope Observed bounded loss of analysis-configuration provenance and a different semantic Evidence ID for an otherwise identical completed observation. The selected document and cancellation error remain available. This is a mocked application-boundary reproduction; NOT_ENOUGH_INFO for naturally occurring cancellation frequency, real-provider behavior, compiled CLI/MCP delivery or a practical caller workflow restoring the missing profile. ## REA version Source package version 6.7.0, commit `3537e58c1ab29309f06eabf50bc52adba9817a0c`. Relevant source files match tag `rea-agents-6.7.0`; the published package was not executed. ## Runtime and platform Node.js v26.12.0; macOS, darwin arm64; kernel 27.0.0. No optional provider application was run. ## Target and artifact context No binary or provider document was opened. The fixture supplies an in-memory selected-document string and a valid matching profile generated by createAnalysisProfile. ## Interface Other Shared application workflow, `EnhancedTools.execute('binary_overview', ...)`. ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Observed output from the application-boundary probe: ```json { "calls": [ "read_binary_inventory_summary", "current_document" ], "error": "AnalysisCancelledError", "execution_profile_present": true, "control_profile_present": true, "partial_profile": null, "retained_result": "selected", "evidence_ids_differ": true } ``` The probe asserts that exactly the inventory-summary and current-document operations ran, that the caller's cancellation reason survives by identity, and that the partial observation retains the result. The profile-preserving control uses the same result, provider, operation, parameters and other projection fields. ## Typed outcome or recovery expected Return the cancellation error with the same reason and completed observation, carrying the execution's original analysis profile and digest in `partialObservation.analysis_profile`. ## Analysis or suspected cause Source-derived cause: the [inventory fallback](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/EnhancedTools.ts#L618-L635) reaches [late-selection completion](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/AnalysisDocument.ts#L99-L118), which uses [createAnalysisExecutionEvidence](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/AnalysisProvider.ts#L134-L148). That projector omits analysisProfile; [Evidence normalization](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/domain/evidence.ts#L410-L420) defaults the omission to null. The ordinary [official MCP projector](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/server/registerOfficialTools.ts#L80-L98) explicitly forwards profiles. [Hopper's producer projection](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/hopper/HopperProvider.ts#L271-L279) can supply them, but no real Hopper execution was performed. ## Additional context Related closed [#4934](https://github.com/morluto/rea/issues/4934) covers cancellation/fallback classification and retaining completed selection observations; this reproduction reaches the retained observation but finds its known profile omitted. Closed [#3992](https://github.com/morluto/rea/issues/3992) covers a separate IDA partial-Evidence constructor after failed decompilation. No matching report was found in the searched open/closed snapshot. Last known good version and first affected release: NOT_ENOUGH_INFO. Freshness review against `29910d16b98fe4b607a7aa15928789a73614b0b9`: the intervening EnhancedTools edits concern call-graph direction selection. The [binary_overview fallback](https://github.com/morluto/rea/blob/29910d16b98fe4b607a7aa15928789a73614b0b9/src/application/EnhancedTools.ts#L621-L638), late-cancellation selection path and [execution-to-Evidence projector](https://github.com/morluto/rea/blob/29910d16b98fe4b607a7aa15928789a73614b0b9/src/application/AnalysisProvider.ts#L130-L148) remain unchanged, including the omitted analysis profile. This is a source freshness check, not a new application, compiled-adapter or real-provider runtime reproduction. The original tested SHA and runtime scope above remain unchanged.

bsakaguchi closed this issue

2026-10-31 00:37:10 +00:00

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@Varshith-Kali commented on GitHub (Oct 19, 2026):

Hi, I'd like to take this on.

I reproduced it on current upstream main (dfe8378cf) with the supplied probe: the aborted binary_overview flow yields AnalysisCancelledError.partialObservation with the selected document intact but analysis_profile: null, while a control projecting the same completed execution preserves the profile. The completed provider execution carries a valid profile, so the loss happens when the partial observation is constructed on the cancellation path in EnhancedTools.

My plan: carry the completed execution's analysis profile through to the partial observation (null stays reserved for producers that genuinely have no profile), keeping the NFC logical identity and Evidence semantics unchanged otherwise. I'll add regression tests covering the late-cancel path with and without a profile.

@Varshith-Kali commented on GitHub (Oct 19, 2026): Hi, I'd like to take this on. I reproduced it on current upstream main (dfe8378cf) with the supplied probe: the aborted `binary_overview` flow yields `AnalysisCancelledError.partialObservation` with the selected document intact but `analysis_profile: null`, while a control projecting the same completed execution preserves the profile. The completed provider execution carries a valid profile, so the loss happens when the partial observation is constructed on the cancellation path in EnhancedTools. My plan: carry the completed execution's analysis profile through to the partial observation (null stays reserved for producers that genuinely have no profile), keeping the NFC logical identity and Evidence semantics unchanged otherwise. I'll add regression tests covering the late-cancel path with and without a profile.

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Confirmed — dropping a known analysis_profile because cancellation arrived late is a provenance regression: the caller had already selected a document and the provider had produced a valid profile, so the Evidence should record it even when the run is cancelled. Losing it makes partial observations inconsistent with control projections of the same data.

I'd support routing the cancellation path through the same execution-to-Evidence projector (Draft PR #5156's direction) rather than special-casing, and adding a regression test that asserts a late-cancelled observation preserves analysis_profile and keeps the same Evidence ID shape as a completed projection.

@XinwangTao commented on GitHub (Oct 19, 2026): Confirmed — dropping a known analysis_profile because cancellation arrived late is a provenance regression: the caller had already selected a document and the provider had produced a valid profile, so the Evidence should record it even when the run is cancelled. Losing it makes partial observations inconsistent with control projections of the same data. I'd support routing the cancellation path through the same execution-to-Evidence projector (Draft PR #5156's direction) rather than special-casing, and adding a regression test that asserts a late-cancelled observation preserves analysis_profile and keeps the same Evidence ID shape as a completed projection.

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 19, 2026):

Cross-referencing with sibling report #3992 ([Bug] Failed IDA decompilation drops the known profile from partial Evidence). Both issues pinpoint the Evidence builder dropping the caller-selected or resolved analysis_profile whenever an operation aborts mid-flight or completes with partial observations.

Consolidated under Cluster AE (Partial Evidence Omission of Known Analysis Profile on Cancellation / Failure) in tracker #4330.

@matbrgz commented on GitHub (Oct 19, 2026): Cross-referencing with sibling report #3992 (`[Bug] Failed IDA decompilation drops the known profile from partial Evidence`). Both issues pinpoint the Evidence builder dropping the caller-selected or resolved `analysis_profile` whenever an operation aborts mid-flight or completes with partial observations. Consolidated under **Cluster AE (Partial Evidence Omission of Known Analysis Profile on Cancellation / Failure)** in tracker #4330.

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

binary_overview still reaches resolveAnalysisDocument after inventory-summary capability fallback, and the late-selection cancellation path now uses the domain-owned createAnalysisExecutionEvidence projector. That projector forwards an existing analysisProfile as well as the other completed facts, preserving the same semantic Evidence identity as a completed control. The parameterized regression aborts immediately before returning current_document or list_documents, then asserts retained profile and equal evidence_id; a separate no-profile case asserts null.

Implementation: EnhancedTools.ts, AnalysisDocument.ts.

The relevant regression checks in AnalysisDocument.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. binary_overview still reaches resolveAnalysisDocument after inventory-summary capability fallback, and the late-selection cancellation path now uses the domain-owned createAnalysisExecutionEvidence projector. That projector forwards an existing analysisProfile as well as the other completed facts, preserving the same semantic Evidence identity as a completed control. The parameterized regression aborts immediately before returning current_document or list_documents, then asserts retained profile and equal evidence_id; a separate no-profile case asserts null. Implementation: [EnhancedTools.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/EnhancedTools.ts#L734), [AnalysisDocument.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/AnalysisDocument.ts#L99). The relevant regression checks in [AnalysisDocument.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/AnalysisDocument.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:08:13 +00:00

[PR #3147] [CLOSED] fix(javascript): decode literal file URL paths #5643

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3147

No description provided.

Rows
Columns