morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3220] [Bug] Prompt completion omits inventories after the first imported artifact inspection substep #1156

New issue

Closed

opened 2026-10-15 18:31:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:31:35 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3220

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

After importing an Evidence bundle containing one artifact inspection with two schema-valid inventory substeps, guided-prompt completion offered identifiers from only the first inventory. The CLI-started stdio MCP server accepted the bundle through import_evidence_bundle. Requests for compare_application_versions arguments left_manifest_id and focus_occurrence_id, each with an empty prefix, returned one identifier with total: 1 and hasMore: false, omitting the second retained manifest and occurrence.

The same session accepted the omitted identifiers when supplied directly to prompts/get. Importing the two inventories as separate top-level Evidence records then made both manifests and both occurrences appear exactly once in completion. This demonstrates a discovery failure for the imported inspection representation.

Reproduction

From a clean checkout of revision 58300885207133422af235e0f2b10658cf958b03, prepare the locked dependencies, build the runtime, and create isolated client directories. The verifier used an existing baseline build and dependencies; install/build prerequisites were not rerun during verification.

nvm use
npm ci
npm run build:cached
mkdir -p .repro/home/config .repro/home/cache .repro/home/data

Create and run this inline fixture. It validates both advertised comparison inventories, assembles a schema-valid inspection and authenticated Evidence bundle, starts the real stdio MCP entry point, imports only the inspection record, and requests manifest and occurrence completions. The child server receives isolated HOME and XDG directories without changing the calling shell's environment.

cat > .repro/reproduce.mjs <<'EOF'
import assert from 'node:assert/strict';
import { writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const root = resolve(process.argv[2]);
const scratch = resolve(process.argv[3]);
const load = path => import(pathToFileURL(resolve(root, path)).href);
const { ARTIFACT_COMPARISON_EXAMPLE: example } = await load('dist/contracts/artifactComparisonExample.js');
const { createArtifactInspection, artifactInspectionResultSchema } = await load('dist/domain/artifactInspection.js');
const { artifactInventoryResultSchema } = await load('dist/domain/artifactGraph.js');
const { createEvidence } = await load('dist/domain/evidence.js');
const { createEvidenceBundle, serializeEvidenceBundle, parseEvidenceBundle } = await load('dist/domain/evidenceBundle.js');
const { digestCanonicalValue } = await load('dist/domain/canonicalDigest.js');
const { Client } = await load('node_modules/@modelcontextprotocol/client/dist/index.mjs');
const { StdioClientTransport } = await load('node_modules/@modelcontextprotocol/client/dist/stdio.mjs');
const inventories = [example.left, example.right];
const expected = inventories.map(e => artifactInventoryResultSchema.parse(e.normalized_result));
const { inspection_id, ...base } = createArtifactInspection(inventories[0]);
const semantic = { ...base, substeps: inventories.flatMap(e => createArtifactInspection(e).substeps), evidence_links: inventories.map(e => e.evidence_id) };
const inspection = artifactInspectionResultSchema.parse({ ...semantic, inspection_id: `ai_${digestCanonicalValue(semantic, 'Artifact inspection')}` });
const inspectionEvidence = createEvidence(undefined, {id:'fixture', name:'Fixture', version:'1'}, {operation:'inspect_artifact', parameters:{}, result:inspection});
const bundle = createEvidenceBundle([inspectionEvidence]);
parseEvidenceBundle(bundle);
const bundlePath = resolve(scratch, 'bundle.json');
writeFileSync(bundlePath, serializeEvidenceBundle(bundle));
console.log(JSON.stringify({schema_valid:true, top_level_records:bundle.records.length, substeps:inspection.substeps.length, coverage_substeps_completed:inspection.coverage.substeps_completed, expected:expected.map(i => ({manifest:i.manifest.manifest_id, occurrences:i.occurrences.map(o => o.occurrence_id)}))}));
const transport = new StdioClientTransport({command:process.execPath, args:[resolve(root,'scripts/rea.mjs'), 'mcp'], cwd:root, env:{...process.env, HOME:resolve(scratch,'home'), XDG_CONFIG_HOME:resolve(scratch,'home/config'), XDG_CACHE_HOME:resolve(scratch,'home/cache'), XDG_DATA_HOME:resolve(scratch,'home/data')}, stderr:'pipe'});
const client = new Client({name:'artifact-completion-verifier',version:'1'});
let stderr = '';
transport.stderr?.on('data', chunk => {stderr += chunk});
try {
  await client.connect(transport);
  const imported = await client.callTool({name:'import_evidence_bundle', arguments:{path:bundlePath}});
  console.log(JSON.stringify({phase:'import', result:imported}));
  assert.notEqual(imported.isError, true);
  for (const [argument, wanted] of [
    ['left_manifest_id', expected.map(i => i.manifest.manifest_id).sort()],
    ['focus_occurrence_id', expected.flatMap(i => i.occurrences.map(o => o.occurrence_id)).sort()]
  ]) {
    const actual = await client.complete({ref:{type:'ref/prompt', name:'compare_application_versions'},argument:{name:argument, value:''}});
    console.log(JSON.stringify({phase:'inspection-only', argument, expected:wanted, actual}));
    console.log(JSON.stringify({missing:wanted.filter(id => !actual.completion.values.includes(id))}));
  }
  const prompt = await client.getPrompt({name:'compare_application_versions',arguments:{left_target_path:resolve(scratch,'left.bin'),right_target_path:resolve(scratch,'right.bin'),left_manifest_id:expected[1].manifest.manifest_id,focus_occurrence_id:expected[1].occurrences[0].occurrence_id}});
  console.log(JSON.stringify({phase:'manual-argument-control', accepted:prompt.messages.length > 0}));
  const controlPath = resolve(scratch,'control.json');
  writeFileSync(controlPath,serializeEvidenceBundle(createEvidenceBundle(inventories)));
  const control = await client.callTool({name:'import_evidence_bundle', arguments:{path:controlPath}});
  assert.notEqual(control.isError,true);
  console.log(JSON.stringify({phase:'control-import', result:control}));
  for (const [argument,wanted] of [
    ['left_manifest_id', expected.map(i => i.manifest.manifest_id).sort()],
    ['focus_occurrence_id', expected.flatMap(i => i.occurrences.map(o => o.occurrence_id)).sort()]
  ]) {
    const actual = await client.complete({ref:{type:'ref/prompt', name:'compare_application_versions'},argument:{name:argument,value:''}});
    console.log(JSON.stringify({phase:'top-level-control',argument,expected:wanted,actual}));
    assert.deepEqual(actual.completion.values,wanted);
  }
  console.log('Control assertions passed. Inspect the inspection-only completion results above.');
} finally {
  await client.close();
  if(stderr) console.error(stderr);
}
EOF
node .repro/reproduce.mjs . .repro

In the two inspection-only responses, observe that each completion returns one value and prints the second inventory's identifier under missing. The manual-argument-control reports acceptance. After the control-import, the top-level-control responses contain both sorted identifiers exactly once and their assertions pass.

The fixture preserves coverage.substeps_completed: 1, which the input schema requires, while the accepted substeps array contains two completed inventory entries. This report concerns completion of the accepted imported representation. It does not assert that the current producer emits multiple substeps or report coverage counting as this defect.

Expected behavior

Both retained manifest IDs and both retained occurrence IDs should be offered. docs/mcp-prompts.md:107-115 defines schema-valid retained artifact inventories and inventory pages as the completion sources and promises every matching value, sorted deterministically and deduplicated. An empty prefix matches both inventories. src/contracts/promptContracts.ts:170-181 assigns those completion families to the tested prompt arguments.

Impact and scope

P2. The reproduced failure affects identifier discovery from accepted imported inspections containing multiple inventory substeps. The current inspection producer emits one substep. No lost Evidence, incorrect comparison result, or execution impact was established.

Two workarounds were demonstrated: supply the retained identifiers directly as prompt arguments, or import the same inventories as separate top-level Evidence records. Both preserve validation and require no code change. No analysis engine, external service, or security permission is required for the reproduction.

REA version

6.4.0 (main at 5830088520)

Runtime and platform

Node.js 26.10.0; macOS; pinned MCP client SDK 2.3.1. The verifier did not record the OS version or architecture. Node.js 26.10.0 satisfies the declared supported range; the checkout's exact .nvmrc runtime, 24.18.0, was unavailable to the verifier.

Target and artifact context

An inline Evidence bundle with one top-level inspect_artifact record and two completed inventory_artifact Evidence substeps, built from ARTIFACT_COMPARISON_EXAMPLE.left and .right. Both inventory payloads, the inspection, and the bundle pass the baseline's validation. No target is executed.

Interface

MCP

Earliest failure stage

Transport or result projection

Relevant output or evidence

Input: schema_valid=true, top_level_records=1, substeps=2, coverage_substeps_completed=1
Initial import: {"result":{"imported":1,"unknowns_added":0,"total":1}}
left_manifest_id completion: total=1, hasMore=false
  returned: agm_727bc5fdb1d6903999f625d154e51878703ba15a815325cf0e0d5713d50ac51f
  missing:  agm_9ff809a884f3419d241f1618bff671e983c50dd72153f6b3b4bdc3766bb91fcf
focus_occurrence_id completion: total=1, hasMore=false
  returned: occ_273bef4da5057c50b27edf034ffce68d5d9c0c5795778fc3ff41e4e6b3ef5586
  missing:  occ_4257a159555da426a3dd9e420ba59795ef3d87d0b1fd93e5cc7d969539519364
Manual prompt arguments: accepted=true
Control import: {"result":{"imported":2,"unknowns_added":0,"total":3}}
After control import: both completions contain both expected sorted identifiers exactly once.

Typed outcome or recovery expected

Before the top-level control import, completion/complete should return both expected manifests for left_manifest_id and both expected occurrences for focus_occurrence_id, with total: 2 and hasMore: false. After the control import, each identifier should still appear exactly once. Continue validating inventory payloads and returning no candidates for malformed inventory data, so the repair does not offer unverified identifiers. These are restoration criteria; no repair was performed during verification.

Analysis or suspected cause

Source-derived at baseline 58300885207133422af235e0f2b10658cf958b03: src/server/promptCompletion.ts:183-200 parses an inspection, selects only inspection.data.substeps[0] at line 192, validates that inventory, and collects its manifest or occurrence IDs. Later retained inventories do not contribute candidates through this branch.

src/domain/artifactInspection.ts:71-81 admits an array of completed inventory substeps with a minimum of one and no maximum. Line 89 requires the coverage literal retained in the fixture. Lines 107-110 explicitly preserve historical/imported input parsing while specializing current producer output. This explains why the accepted imported representation can expose the first-substep selection even though the current producer constructs one substep.

Additional context

The supported P2 assessment follows the bounded identifier-discovery failure and the two demonstrated workarounds. The controls distinguish missing nested-inventory enumeration from invalid payloads, prefix filtering, and completion size limits.

Related: #3171 reports a separate failure in the same area; the two have independent fixes.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3220 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Observed behavior After importing an Evidence bundle containing one artifact inspection with two schema-valid inventory substeps, guided-prompt completion offered identifiers from only the first inventory. The CLI-started stdio MCP server accepted the bundle through `import_evidence_bundle`. Requests for `compare_application_versions` arguments `left_manifest_id` and `focus_occurrence_id`, each with an empty prefix, returned one identifier with `total: 1` and `hasMore: false`, omitting the second retained manifest and occurrence. The same session accepted the omitted identifiers when supplied directly to `prompts/get`. Importing the two inventories as separate top-level Evidence records then made both manifests and both occurrences appear exactly once in completion. This demonstrates a discovery failure for the imported inspection representation. ### Reproduction 1. From a clean checkout of revision `58300885207133422af235e0f2b10658cf958b03`, prepare the locked dependencies, build the runtime, and create isolated client directories. The verifier used an existing baseline build and dependencies; install/build prerequisites were not rerun during verification. ```sh nvm use npm ci npm run build:cached mkdir -p .repro/home/config .repro/home/cache .repro/home/data ``` 2. Create and run this inline fixture. It validates both advertised comparison inventories, assembles a schema-valid inspection and authenticated Evidence bundle, starts the real stdio MCP entry point, imports only the inspection record, and requests manifest and occurrence completions. The child server receives isolated HOME and XDG directories without changing the calling shell's environment. ```sh cat > .repro/reproduce.mjs <<'EOF' import assert from 'node:assert/strict'; import { writeFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; const root = resolve(process.argv[2]); const scratch = resolve(process.argv[3]); const load = path => import(pathToFileURL(resolve(root, path)).href); const { ARTIFACT_COMPARISON_EXAMPLE: example } = await load('dist/contracts/artifactComparisonExample.js'); const { createArtifactInspection, artifactInspectionResultSchema } = await load('dist/domain/artifactInspection.js'); const { artifactInventoryResultSchema } = await load('dist/domain/artifactGraph.js'); const { createEvidence } = await load('dist/domain/evidence.js'); const { createEvidenceBundle, serializeEvidenceBundle, parseEvidenceBundle } = await load('dist/domain/evidenceBundle.js'); const { digestCanonicalValue } = await load('dist/domain/canonicalDigest.js'); const { Client } = await load('node_modules/@modelcontextprotocol/client/dist/index.mjs'); const { StdioClientTransport } = await load('node_modules/@modelcontextprotocol/client/dist/stdio.mjs'); const inventories = [example.left, example.right]; const expected = inventories.map(e => artifactInventoryResultSchema.parse(e.normalized_result)); const { inspection_id, ...base } = createArtifactInspection(inventories[0]); const semantic = { ...base, substeps: inventories.flatMap(e => createArtifactInspection(e).substeps), evidence_links: inventories.map(e => e.evidence_id) }; const inspection = artifactInspectionResultSchema.parse({ ...semantic, inspection_id: `ai_${digestCanonicalValue(semantic, 'Artifact inspection')}` }); const inspectionEvidence = createEvidence(undefined, {id:'fixture', name:'Fixture', version:'1'}, {operation:'inspect_artifact', parameters:{}, result:inspection}); const bundle = createEvidenceBundle([inspectionEvidence]); parseEvidenceBundle(bundle); const bundlePath = resolve(scratch, 'bundle.json'); writeFileSync(bundlePath, serializeEvidenceBundle(bundle)); console.log(JSON.stringify({schema_valid:true, top_level_records:bundle.records.length, substeps:inspection.substeps.length, coverage_substeps_completed:inspection.coverage.substeps_completed, expected:expected.map(i => ({manifest:i.manifest.manifest_id, occurrences:i.occurrences.map(o => o.occurrence_id)}))})); const transport = new StdioClientTransport({command:process.execPath, args:[resolve(root,'scripts/rea.mjs'), 'mcp'], cwd:root, env:{...process.env, HOME:resolve(scratch,'home'), XDG_CONFIG_HOME:resolve(scratch,'home/config'), XDG_CACHE_HOME:resolve(scratch,'home/cache'), XDG_DATA_HOME:resolve(scratch,'home/data')}, stderr:'pipe'}); const client = new Client({name:'artifact-completion-verifier',version:'1'}); let stderr = ''; transport.stderr?.on('data', chunk => {stderr += chunk}); try { await client.connect(transport); const imported = await client.callTool({name:'import_evidence_bundle', arguments:{path:bundlePath}}); console.log(JSON.stringify({phase:'import', result:imported})); assert.notEqual(imported.isError, true); for (const [argument, wanted] of [ ['left_manifest_id', expected.map(i => i.manifest.manifest_id).sort()], ['focus_occurrence_id', expected.flatMap(i => i.occurrences.map(o => o.occurrence_id)).sort()] ]) { const actual = await client.complete({ref:{type:'ref/prompt', name:'compare_application_versions'},argument:{name:argument, value:''}}); console.log(JSON.stringify({phase:'inspection-only', argument, expected:wanted, actual})); console.log(JSON.stringify({missing:wanted.filter(id => !actual.completion.values.includes(id))})); } const prompt = await client.getPrompt({name:'compare_application_versions',arguments:{left_target_path:resolve(scratch,'left.bin'),right_target_path:resolve(scratch,'right.bin'),left_manifest_id:expected[1].manifest.manifest_id,focus_occurrence_id:expected[1].occurrences[0].occurrence_id}}); console.log(JSON.stringify({phase:'manual-argument-control', accepted:prompt.messages.length > 0})); const controlPath = resolve(scratch,'control.json'); writeFileSync(controlPath,serializeEvidenceBundle(createEvidenceBundle(inventories))); const control = await client.callTool({name:'import_evidence_bundle', arguments:{path:controlPath}}); assert.notEqual(control.isError,true); console.log(JSON.stringify({phase:'control-import', result:control})); for (const [argument,wanted] of [ ['left_manifest_id', expected.map(i => i.manifest.manifest_id).sort()], ['focus_occurrence_id', expected.flatMap(i => i.occurrences.map(o => o.occurrence_id)).sort()] ]) { const actual = await client.complete({ref:{type:'ref/prompt', name:'compare_application_versions'},argument:{name:argument,value:''}}); console.log(JSON.stringify({phase:'top-level-control',argument,expected:wanted,actual})); assert.deepEqual(actual.completion.values,wanted); } console.log('Control assertions passed. Inspect the inspection-only completion results above.'); } finally { await client.close(); if(stderr) console.error(stderr); } EOF node .repro/reproduce.mjs . .repro ``` 3. In the two `inspection-only` responses, observe that each completion returns one value and prints the second inventory's identifier under `missing`. The `manual-argument-control` reports acceptance. After the `control-import`, the `top-level-control` responses contain both sorted identifiers exactly once and their assertions pass. The fixture preserves `coverage.substeps_completed: 1`, which the input schema requires, while the accepted `substeps` array contains two completed inventory entries. This report concerns completion of the accepted imported representation. It does not assert that the current producer emits multiple substeps or report coverage counting as this defect. ### Expected behavior Both retained manifest IDs and both retained occurrence IDs should be offered. `docs/mcp-prompts.md:107-115` defines schema-valid retained artifact inventories and inventory pages as the completion sources and promises every matching value, sorted deterministically and deduplicated. An empty prefix matches both inventories. `src/contracts/promptContracts.ts:170-181` assigns those completion families to the tested prompt arguments. ### Impact and scope P2. The reproduced failure affects identifier discovery from accepted imported inspections containing multiple inventory substeps. The current inspection producer emits one substep. No lost Evidence, incorrect comparison result, or execution impact was established. Two workarounds were demonstrated: supply the retained identifiers directly as prompt arguments, or import the same inventories as separate top-level Evidence records. Both preserve validation and require no code change. No analysis engine, external service, or security permission is required for the reproduction. ### REA version 6.4.0 (main at 58300885207133422af235e0f2b10658cf958b03) ### Runtime and platform Node.js 26.10.0; macOS; pinned MCP client SDK 2.3.1. The verifier did not record the OS version or architecture. Node.js 26.10.0 satisfies the declared supported range; the checkout's exact `.nvmrc` runtime, 24.18.0, was unavailable to the verifier. ### Target and artifact context An inline Evidence bundle with one top-level `inspect_artifact` record and two completed `inventory_artifact` Evidence substeps, built from `ARTIFACT_COMPARISON_EXAMPLE.left` and `.right`. Both inventory payloads, the inspection, and the bundle pass the baseline's validation. No target is executed. ### Interface MCP ### Earliest failure stage Transport or result projection ### Relevant output or evidence ```text Input: schema_valid=true, top_level_records=1, substeps=2, coverage_substeps_completed=1 Initial import: {"result":{"imported":1,"unknowns_added":0,"total":1}} left_manifest_id completion: total=1, hasMore=false returned: agm_727bc5fdb1d6903999f625d154e51878703ba15a815325cf0e0d5713d50ac51f missing: agm_9ff809a884f3419d241f1618bff671e983c50dd72153f6b3b4bdc3766bb91fcf focus_occurrence_id completion: total=1, hasMore=false returned: occ_273bef4da5057c50b27edf034ffce68d5d9c0c5795778fc3ff41e4e6b3ef5586 missing: occ_4257a159555da426a3dd9e420ba59795ef3d87d0b1fd93e5cc7d969539519364 Manual prompt arguments: accepted=true Control import: {"result":{"imported":2,"unknowns_added":0,"total":3}} After control import: both completions contain both expected sorted identifiers exactly once. ``` ### Typed outcome or recovery expected Before the top-level control import, `completion/complete` should return both expected manifests for `left_manifest_id` and both expected occurrences for `focus_occurrence_id`, with `total: 2` and `hasMore: false`. After the control import, each identifier should still appear exactly once. Continue validating inventory payloads and returning no candidates for malformed inventory data, so the repair does not offer unverified identifiers. These are restoration criteria; no repair was performed during verification. ### Analysis or suspected cause Source-derived at baseline `58300885207133422af235e0f2b10658cf958b03`: `src/server/promptCompletion.ts:183-200` parses an inspection, selects only `inspection.data.substeps[0]` at line 192, validates that inventory, and collects its manifest or occurrence IDs. Later retained inventories do not contribute candidates through this branch. `src/domain/artifactInspection.ts:71-81` admits an array of completed inventory substeps with a minimum of one and no maximum. Line 89 requires the coverage literal retained in the fixture. Lines 107-110 explicitly preserve historical/imported input parsing while specializing current producer output. This explains why the accepted imported representation can expose the first-substep selection even though the current producer constructs one substep. ### Additional context The supported P2 assessment follows the bounded identifier-discovery failure and the two demonstrated workarounds. The controls distinguish missing nested-inventory enumeration from invalid payloads, prefix filtering, and completion size limits. Related: #3171 reports a separate failure in the same area; the two have independent fixes.

bsakaguchi

2026-10-15 18:31:35 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-15 18:31:37 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Delivered on main in fffb07419.

Completion read only the first inventory substep, and its later per-child repair bypassed inspection-level identity checks. Completion now consumes the same authenticated, assembled inspection as comparison and analysis views. Invalid carriers are rejected; valid later pages contribute identifiers.

Validation: MCP prompt completion tests include a real paged inventory and malformed child, substep-link, count, declaration and carrier-subject controls. Full exact-revision qualification passed.

@morluto commented on GitHub (Oct 14, 2026): Delivered on main in [fffb07419](https://github.com/morluto/rea/commit/fffb0741936067530d1ce9777461cd53e87398c2). Completion read only the first inventory substep, and its later per-child repair bypassed inspection-level identity checks. Completion now consumes the same authenticated, assembled inspection as comparison and analysis views. Invalid carriers are rejected; valid later pages contribute identifiers. Validation: MCP prompt completion tests include a real paged inventory and malformed child, substep-link, count, declaration and carrier-subject controls. Full exact-revision [qualification](https://github.com/morluto/rea/actions/runs/38629313022) passed.

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:09:08 +00:00

[PR #1156] [MERGED] fix(cli): cancel routed JavaScript analysis in rea analyze #2691