morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5916] [Bug] Finite-trace validation accepts duplicates with reordered unordered members #3592

New issue

Closed

opened 2026-10-31 00:42:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:12 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5916

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

The finite-trace specification accepts two variants whose only difference is the order of event IDs inside an unordered token. Both variants accept the same traces, but the existing duplicate-trace validation does not reject them.

I executed the built compare-process-captures CLI with complete fixture captures in opposite terminal/process journal orders. It accepted the specification and returned comparison Evidence with trace verdict equivalent; both sides reported matched_variant: "forward" and satisfied_constraints: ["variant:forward"]. The second variant, reverse, cannot be selected because the first accepts exactly the same observations.

A control with two structurally identical traces was rejected by the same schema with finite trace variants must not duplicate a trace. Permuting an unordered token's members bypasses that existing invariant.

Reproduction

Dependencies and compiled runtime were already prepared at the source revision below. No build or installation was rerun. These inputs use the repository's synthetic capture fixtures and production Evidence constructors; no live target or provider is required.

Save this focused test as tests/boundary/mcp/unorderedDuplicate.test.ts:

import { execFileSync } from "node:child_process";
import { writeFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { expect, it } from "vitest";
import { capture, values, partialSpecification } from "../../../src/domain/process/processTraceComparison.fixture.js";
import { processTraceSpecificationSchema } from "../../../src/domain/process/processTraceSpecification.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js";
import { jsonValueSchema } from "../../../src/domain/jsonValue.js";

it("accepts reordered unordered duplicates through the CLI", () => {
  const root = process.env.REA_VERIFY_SCRATCH!;
  const spec = {
    events: partialSpecification().events,
    language: { kind: "finite_traces", variants: [
      { id: "forward", trace: [{ unordered: ["ready", "worker"] }] },
      { id: "reverse", trace: [{ unordered: ["worker", "ready"] }] },
    ] },
  };
  expect(processTraceSpecificationSchema.safeParse(spec).success).toBe(true);
  const exactDuplicate = structuredClone(spec);
  exactDuplicate.language.variants[1]!.trace = structuredClone(spec.language.variants[0]!.trace);
  const control = processTraceSpecificationSchema.safeParse(exactDuplicate);
  expect(control.success).toBe(false);
  if (control.success) throw new Error("exact duplicate accepted");
  expect(control.error.issues).toContainEqual(expect.objectContaining({
    path: ["language", "variants", 1, "trace"],
    message: "finite trace variants must not duplicate a trace",
  }));
  const files = ["left.json", "right.json", "spec.json"].map(name => join(root, name));
  try {
    for (const [index, order] of [values(["terminal", "process"]), values(["process", "terminal"])].entries()) {
      const evidence = createEvidence(undefined, PROCESS_PROVIDER, {
        predicateType: "rea.process-capture", operation: "capture_process_scenario",
        parameters: {}, result: jsonValueSchema.parse(capture(order)),
        confidence: "observed", authority: "controlled-replay",
      });
      writeFileSync(files[index]!, JSON.stringify(evidence));
    }
    writeFileSync(files[2]!, JSON.stringify(spec));
    const stdout = execFileSync(process.execPath, ["scripts/rea.mjs", "compare-process-captures", ...files, "--format", "json"], { encoding: "utf8" });
    writeFileSync(join(root, "cli-output.json"), stdout);
    const result = JSON.parse(stdout);
    console.log("CLI trace:", JSON.stringify(result.normalized_result?.trace ?? result));
    expect(result.normalized_result.trace).toMatchObject({
      verdict: "equivalent",
      left: { status: "pass", matched_variant: "forward", satisfied_constraints: ["variant:forward"] },
      right: { status: "pass", matched_variant: "forward", satisfied_constraints: ["variant:forward"] },
    });
  } finally {
    for (const file of files) rmSync(file, { force: true });
  }
});

Run only that test, with a writable temporary directory for its disposable capture files and retained CLI output:

export REA_VERIFY_SCRATCH="$(mktemp -d)"
node node_modules/vitest/vitest.mjs run tests/boundary/mcp/unorderedDuplicate.test.ts --maxWorkers 1

The test invokes this real CLI command internally:

node scripts/rea.mjs compare-process-captures left.json right.json spec.json --format json

It asserts both the observed acceptance of reordered unordered duplicates and rejection of an exact structural duplicate, then checks the comparison Evidence returned by the CLI. The test passed under Node 26.12.0 and again under the repository's pinned Node 24.18.0.

Expected behavior

Reject the second variant as a duplicate trace when it differs only by the order of members within the same unordered token. Keep the token's position and event multiplicities significant.

The existing duplicate diagnostic is at processTraceSpecification.ts, lines 575–582. The matching implementation already treats the unordered members as a multiset, so reversing their list does not describe a distinct trace. The negative control establishes that changing the variant ID alone does not make an otherwise identical trace valid.

This is a narrow duplicate-normalization expectation, not a request to reject every possible overlap between different finite languages.

Impact and scope

The reproduced scope is finite-trace variants with the same token sequence and member multiplicities but permuted unordered members. The validation gap permits redundant variants that can never appear as matched_variant, and makes duplicate detection depend on an ordering the evaluator explicitly ignores.

The demonstrated equivalence verdict is appropriate for the accepted language; the failure is admission of the redundant specification. No capture mutation, data loss, or security boundary crossing was observed. A practical workaround is to remove the redundant variant, which preserves the accepted trace language. Population frequency and downstream effects were not measured. This is a bounded P2 validation defect.

REA version

rea-agents 6.7.0, source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; reproduced with Node 26.12.0 and Node 24.18.0; Vitest 5.0.3.

Target and artifact context

Synthetic, fully parsed process-capture Evidence constructed from the repository's existing terminal/process trace fixtures. Each capture contains the same terminal frame and process sample, with opposite cross-source journal ordering. No live process capture or analysis provider ran.

Interface

CLI

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Both focused executions passed one test. The actual CLI subprocess exited 0 and produced these fields:

trace.verdict = equivalent
trace.left.raw_trace event IDs = [ready, worker]
trace.right.raw_trace event IDs = [worker, ready]
trace.left.status = pass
trace.right.status = pass
trace.left.matched_variant = forward
trace.right.matched_variant = forward
trace.left.satisfied_constraints = [variant:forward]
trace.right.satisfied_constraints = [variant:forward]

The schema control rejected the structural duplicate at:

path: ["language", "variants", 1, "trace"]
message: finite trace variants must not duplicate a trace

The CLI registration accepts the trace file as its third positional argument. The application workflow parses capture Evidence and the trace specification before creating comparison Evidence. The documentation describes that public comparison entry point.

Typed outcome or recovery expected

Return the existing invalid-input outcome, with finite trace variants must not duplicate a trace at trace_spec.language.variants[1].trace, before creating comparison Evidence for this specification. The direct schema path omits the trace_spec prefix; the public CLI adds it in parseTraceSpecification. Removing the redundant variant should leave one accepted language.

Analysis or suspected cause

Source-derived: validateFiniteTraces keys traces with canonicalTraceJson(variant.trace). That helper uses JSON canonicalization, which preserves array order. The evaluator instead counts unordered members and returns the first matching variant. A trace-specific duplicate key that normalizes only each unordered member list, retaining multiplicities and token positions, would align validation with these matching semantics.

The helper is also used outside finite-trace duplicate checking, so changing global array canonicalization would require separate justification.

Additional context

The relevant specification, evaluator, CLI adapter, and MCP input-schema files were unchanged after fetching upstream main at efab044f303ae3d957d977730bb1628063223236. MCP source uses the same schema, but MCP transport behavior was not executed. The full suite and live target capture were not run.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5916 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior The finite-trace specification accepts two variants whose only difference is the order of event IDs inside an `unordered` token. Both variants accept the same traces, but the existing duplicate-trace validation does not reject them. I executed the built `compare-process-captures` CLI with complete fixture captures in opposite terminal/process journal orders. It accepted the specification and returned comparison Evidence with trace verdict `equivalent`; both sides reported `matched_variant: "forward"` and `satisfied_constraints: ["variant:forward"]`. The second variant, `reverse`, cannot be selected because the first accepts exactly the same observations. A control with two structurally identical traces was rejected by the same schema with `finite trace variants must not duplicate a trace`. Permuting an unordered token's members bypasses that existing invariant. ### Reproduction Dependencies and compiled runtime were already prepared at the source revision below. No build or installation was rerun. These inputs use the repository's synthetic capture fixtures and production Evidence constructors; no live target or provider is required. Save this focused test as `tests/boundary/mcp/unorderedDuplicate.test.ts`: ```ts import { execFileSync } from "node:child_process"; import { writeFileSync, rmSync } from "node:fs"; import { join } from "node:path"; import { expect, it } from "vitest"; import { capture, values, partialSpecification } from "../../../src/domain/process/processTraceComparison.fixture.js"; import { processTraceSpecificationSchema } from "../../../src/domain/process/processTraceSpecification.js"; import { createEvidence } from "../../../src/domain/evidence.js"; import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js"; import { jsonValueSchema } from "../../../src/domain/jsonValue.js"; it("accepts reordered unordered duplicates through the CLI", () => { const root = process.env.REA_VERIFY_SCRATCH!; const spec = { events: partialSpecification().events, language: { kind: "finite_traces", variants: [ { id: "forward", trace: [{ unordered: ["ready", "worker"] }] }, { id: "reverse", trace: [{ unordered: ["worker", "ready"] }] }, ] }, }; expect(processTraceSpecificationSchema.safeParse(spec).success).toBe(true); const exactDuplicate = structuredClone(spec); exactDuplicate.language.variants[1]!.trace = structuredClone(spec.language.variants[0]!.trace); const control = processTraceSpecificationSchema.safeParse(exactDuplicate); expect(control.success).toBe(false); if (control.success) throw new Error("exact duplicate accepted"); expect(control.error.issues).toContainEqual(expect.objectContaining({ path: ["language", "variants", 1, "trace"], message: "finite trace variants must not duplicate a trace", })); const files = ["left.json", "right.json", "spec.json"].map(name => join(root, name)); try { for (const [index, order] of [values(["terminal", "process"]), values(["process", "terminal"])].entries()) { const evidence = createEvidence(undefined, PROCESS_PROVIDER, { predicateType: "rea.process-capture", operation: "capture_process_scenario", parameters: {}, result: jsonValueSchema.parse(capture(order)), confidence: "observed", authority: "controlled-replay", }); writeFileSync(files[index]!, JSON.stringify(evidence)); } writeFileSync(files[2]!, JSON.stringify(spec)); const stdout = execFileSync(process.execPath, ["scripts/rea.mjs", "compare-process-captures", ...files, "--format", "json"], { encoding: "utf8" }); writeFileSync(join(root, "cli-output.json"), stdout); const result = JSON.parse(stdout); console.log("CLI trace:", JSON.stringify(result.normalized_result?.trace ?? result)); expect(result.normalized_result.trace).toMatchObject({ verdict: "equivalent", left: { status: "pass", matched_variant: "forward", satisfied_constraints: ["variant:forward"] }, right: { status: "pass", matched_variant: "forward", satisfied_constraints: ["variant:forward"] }, }); } finally { for (const file of files) rmSync(file, { force: true }); } }); ``` Run only that test, with a writable temporary directory for its disposable capture files and retained CLI output: ```sh export REA_VERIFY_SCRATCH="$(mktemp -d)" node node_modules/vitest/vitest.mjs run tests/boundary/mcp/unorderedDuplicate.test.ts --maxWorkers 1 ``` The test invokes this real CLI command internally: ```sh node scripts/rea.mjs compare-process-captures left.json right.json spec.json --format json ``` It asserts both the observed acceptance of reordered unordered duplicates and rejection of an exact structural duplicate, then checks the comparison Evidence returned by the CLI. The test passed under Node 26.12.0 and again under the repository's pinned Node 24.18.0. ### Expected behavior Reject the second variant as a duplicate trace when it differs only by the order of members within the same unordered token. Keep the token's position and event multiplicities significant. The existing duplicate diagnostic is at [processTraceSpecification.ts, lines 575–582](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/process/processTraceSpecification.ts#L575-L582). The [matching implementation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/process/processTraceEvaluation.ts#L164-L198) already treats the unordered members as a multiset, so reversing their list does not describe a distinct trace. The negative control establishes that changing the variant ID alone does not make an otherwise identical trace valid. This is a narrow duplicate-normalization expectation, not a request to reject every possible overlap between different finite languages. ### Impact and scope The reproduced scope is finite-trace variants with the same token sequence and member multiplicities but permuted unordered members. The validation gap permits redundant variants that can never appear as `matched_variant`, and makes duplicate detection depend on an ordering the evaluator explicitly ignores. The demonstrated equivalence verdict is appropriate for the accepted language; the failure is admission of the redundant specification. No capture mutation, data loss, or security boundary crossing was observed. A practical workaround is to remove the redundant variant, which preserves the accepted trace language. Population frequency and downstream effects were not measured. This is a bounded P2 validation defect. ### REA version rea-agents 6.7.0, source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; reproduced with Node 26.12.0 and Node 24.18.0; Vitest 5.0.3. ### Target and artifact context Synthetic, fully parsed process-capture Evidence constructed from the repository's existing terminal/process trace fixtures. Each capture contains the same terminal frame and process sample, with opposite cross-source journal ordering. No live process capture or analysis provider ran. ### Interface CLI ### Earliest failure stage Request parsing or validation ### Relevant output or evidence Both focused executions passed one test. The actual CLI subprocess exited 0 and produced these fields: ```text trace.verdict = equivalent trace.left.raw_trace event IDs = [ready, worker] trace.right.raw_trace event IDs = [worker, ready] trace.left.status = pass trace.right.status = pass trace.left.matched_variant = forward trace.right.matched_variant = forward trace.left.satisfied_constraints = [variant:forward] trace.right.satisfied_constraints = [variant:forward] ``` The schema control rejected the structural duplicate at: ```text path: ["language", "variants", 1, "trace"] message: finite trace variants must not duplicate a trace ``` The [CLI registration](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/cli/processCommands.ts#L31-L56) accepts the trace file as its third positional argument. The [application workflow](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/process/ProcessCli.ts#L120-L166) parses capture Evidence and the trace specification before creating comparison Evidence. The [documentation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/process-capture.md#L72-L95) describes that public comparison entry point. ### Typed outcome or recovery expected Return the existing invalid-input outcome, with `finite trace variants must not duplicate a trace` at `trace_spec.language.variants[1].trace`, before creating comparison Evidence for this specification. The direct schema path omits the `trace_spec` prefix; the public CLI adds it in [parseTraceSpecification](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/process/ProcessCli.ts#L214-L226). Removing the redundant variant should leave one accepted language. ### Analysis or suspected cause Source-derived: `validateFiniteTraces` keys traces with `canonicalTraceJson(variant.trace)`. That helper uses JSON canonicalization, which preserves array order. The evaluator instead counts unordered members and returns the first matching variant. A trace-specific duplicate key that normalizes only each unordered member list, retaining multiplicities and token positions, would align validation with these matching semantics. The helper is also used outside finite-trace duplicate checking, so changing global array canonicalization would require separate justification. ### Additional context The relevant specification, evaluator, CLI adapter, and MCP input-schema files were unchanged after fetching upstream main at `efab044f303ae3d957d977730bb1628063223236`. MCP source uses the same schema, but MCP transport behavior was not executed. The full suite and live target capture were not run.

bsakaguchi closed this issue

2026-10-31 00:42:13 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:58 +00:00

[PR #3622] [CLOSED] feat(analysis-view): report value-free request header facets in API surfaces #5858

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471