morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6312] [Bug] Invalid CArchive compression-flag diagnostics omit the TOC entry index #3813

New issue

Closed

opened 2026-10-31 00:44:18 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:18 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6312

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

For each corrupted index, open_binary succeeds and inspect_artifact returns isError: true with code: "artifact_operation_failed", details.reason: "format" and details.detail: "CArchive compression flag 2 is not stored or zlib". The entire error reply lacks the table index or a TOC offset. The unmodified control inventories all three member indices [0,1,2].

Reproduction

From the source baseline below with its locked dependencies already installed and Node v24.18.0 on PATH, save this complete test as tests/boundary/mcp/tocIndexRepro.test.ts:

import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect } from "vitest";
import { artifactMcpCall } from "../../../scripts/lib/artifact-e2e.mjs";
import { cArchiveExecutable } from "../../../src/artifacts/pyinstaller/CArchive.fixture.js";
import { createArtifactBinarySession } from "../../../src/composition/binary.js";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { artifactInspectionResultSchema } from "../../../src/domain/artifactInspection.js";
import { createServer } from "../../../src/server/createServer.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { mcpTest } from "../../support/mcp/mcpFixture.js";

for (const index of [null, 0, 1, 2]) {
  mcpTest(`CArchive compression flag at index ${index}`, async ({ mcp }) => {
    const bytes = cArchiveExecutable({
      layout: "2.1",
      pythonVersion: 309,
      members: ["a.txt", "b.txt", "c.txt"].map((name) => ({
        name, typeCode: "x", payload: Buffer.from(name), compress: false,
      })),
    });
    let record = 64 + bytes.readUInt32BE(bytes.length - 88 + 12);
    if (index !== null) {
      for (let row = 0; row < index; row++) record += bytes.readUInt32BE(record);
      bytes[record + 16] = 2;
    }
    const directory = await createTestTempDirectory("toc-index-repro-");
    const path = join(directory, "carrier");
    await writeFile(path, bytes);
    const session = await createArtifactBinarySession(process.env);
    const client = await mcp.connect(createServer({ kind: "session", session }));
    const call = (name: string, args: Record<string, unknown>) => artifactMcpCall(client, { name, arguments: args });
    const opened = await call("open_binary", { path });
    expect(opened.isError).not.toBe(true);
    const reply = await call("inspect_artifact", {});
    if (index === null) {
      expect(reply.isError).not.toBe(true);
      const inspection = artifactInspectionResultSchema.parse(parseEvidence(reply.structuredContent).normalized_result);
      const inventory = inspection.substeps[0]!.evidence.normalized_result;
      const rows = inventory.occurrences.filter((row) => row.format_metadata?.kind === "pyinstaller-entry");
      expect(rows.map((row) => row.format_metadata.index)).toEqual([0, 1, 2]);
      console.log(JSON.stringify({ case: "valid", indices: rows.map((row) => row.format_metadata.index) }));
    } else {
      expect(reply.isError).toBe(true);
      console.log(JSON.stringify({ corruptedIndex: index, reply }));
      const text = reply.content.find((block) => block.type === "text");
      const error = JSON.parse((text as { text: string }).text).error;
      expect(error.code).toBe("artifact_operation_failed");
      expect(error.details).toEqual({ operation: "inspect_artifact", reason: "format", detail: "CArchive compression flag 2 is not stored or zlib" });
      expect(JSON.stringify(reply)).not.toContain("table index");
    }
  });
}

Run with an isolated configuration and a single worker:

repro_root="$(mktemp -d)"
mkdir -p "$repro_root/home" "$repro_root/config" "$repro_root/cache" "$repro_root/temp"
env -i PATH="$PATH" HOME="$repro_root/home" DSH_HOME="$repro_root/home" XDG_CONFIG_HOME="$repro_root/config" XDG_CACHE_HOME="$repro_root/cache" TMPDIR="$repro_root/temp" node node_modules/vitest/vitest.mjs run tests/boundary/mcp/tocIndexRepro.test.ts --maxWorkers=1

The test generates a valid ELF carrier with a PyInstaller 2.1 CArchive containing three stored data members, then tests independent copies with only the compression flag at index 0, 1 or 2 changed to 2. It calls the registered open_binary and inspect_artifact tools through the SDK in-memory transport and validates advertised output schemas. Its assertions characterize the current bug, so all four cases pass while the index is absent. Remove the test and disposable configuration directory after use.

The exact test content ran twice in isolated configurations: the valid control succeeded, and all three corrupted-index cases reproduced the missing identifier on both runs. No target code, native provider, CLI or stdio server was executed.

Expected behavior

Malformed CArchive TOC diagnostics should identify the damaged entry by its table index, as the Python packaged applications guide promises. The three malformed cases should therefore name indices 0, 1 and 2 respectively while retaining the format refusal.

The Python packaged applications guide explicitly states: "Malformed TOC diagnostics name the table index".

Impact and scope

P3, diagnostics-only. The malformed compression flag is still rejected. The observed difference is missing row-identifying text, with no incorrect successful inventory, changed failure classification or security-boundary crossing demonstrated. The other five parse failures, extraction, CLI, stdio transport, other platforms and real-world frequency were not independently tested. Manually decoding TOC boundaries is a possible workaround, but an alternate end-user diagnostic workflow was not verified.

REA version

Source commit 055d1a9e1b (main)

Runtime and platform

Node v24.18.0; macOS 27.0.1 arm64; repository-pinned Vitest 5.0.3; locked MCP SDK. Node matches the repository pin. No external analysis engine was used.

Target and artifact context

Synthetic ELF prefix plus PyInstaller 2.1 CArchive, Python version field 309, containing stored type-x members a.txt, b.txt and c.txt whose payloads equal their names. Each malformed copy changes one compression-flag byte to 2; no archive offsets or lengths change.

Interface

MCP

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The valid control prints {"case":"valid","indices":[0,1,2]}. For each corruptedIndex value 0, 1 and 2, the complete public error object is identical:

{
  "code": "artifact_operation_failed",
  "category": "execution_failure",
  "message": "Artifact is invalid or has changed. Get a fresh copy and try again.",
  "retryable": false,
  "remediation": {
    "action": "Artifact is invalid or has changed. Get a fresh copy and try again."
  },
  "details": {
    "operation": "inspect_artifact",
    "reason": "format",
    "detail": "CArchive compression flag 2 is not stored or zlib"
  }
}

One test file and all four characterization cases passed. The complete MCP replies contain no row identifier. These observations establish this compression-flag case only.

Typed outcome or recovery expected

Keep the existing malformed-input format refusal and include the corresponding table index in the diagnostic. The valid control must continue to inventory all three members. Moving the invalid flag between entries must change the reported index.

Analysis or suspected cause

Source-derived: the compression-flag validation constructs the message before appending the entry and does not include its current index. This explains the observed public detail. No repaired software was tested.

Additional context

Open and closed issues and PRs were searched with PyInstaller TOC, CArchive diagnostic, table index, CArchive, malformed TOC and compression flag. Related #2433 and #3678 describe the original capability; #6120 concerns negative PYZ sizes, #5836 capacity recovery, #6067 changed-source admission, and #6147 a separate proposed extractor. No same-behavior duplicate was found, including the original PR's diagnostic review comments. Those comments concern oversized names and path previews.

Fetched main at 3a42a9e356da45106640b4d84468f552c18bcf78 has no relevant changes to the parser, reader, artifact-session composition or guide since the tested baseline. Current upstream was source-inspected rather than executed. No known-good revision was established.

Originally created by @coygeek on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6312 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior For each corrupted index, `open_binary` succeeds and `inspect_artifact` returns `isError: true` with `code: "artifact_operation_failed"`, `details.reason: "format"` and `details.detail: "CArchive compression flag 2 is not stored or zlib"`. The entire error reply lacks the table index or a TOC offset. The unmodified control inventories all three member indices `[0,1,2]`. ### Reproduction From the source baseline below with its locked dependencies already installed and Node v24.18.0 on PATH, save this complete test as `tests/boundary/mcp/tocIndexRepro.test.ts`: ```ts import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; import { artifactMcpCall } from "../../../scripts/lib/artifact-e2e.mjs"; import { cArchiveExecutable } from "../../../src/artifacts/pyinstaller/CArchive.fixture.js"; import { createArtifactBinarySession } from "../../../src/composition/binary.js"; import { parseEvidence } from "../../../src/domain/evidence.js"; import { artifactInspectionResultSchema } from "../../../src/domain/artifactInspection.js"; import { createServer } from "../../../src/server/createServer.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { mcpTest } from "../../support/mcp/mcpFixture.js"; for (const index of [null, 0, 1, 2]) { mcpTest(`CArchive compression flag at index ${index}`, async ({ mcp }) => { const bytes = cArchiveExecutable({ layout: "2.1", pythonVersion: 309, members: ["a.txt", "b.txt", "c.txt"].map((name) => ({ name, typeCode: "x", payload: Buffer.from(name), compress: false, })), }); let record = 64 + bytes.readUInt32BE(bytes.length - 88 + 12); if (index !== null) { for (let row = 0; row < index; row++) record += bytes.readUInt32BE(record); bytes[record + 16] = 2; } const directory = await createTestTempDirectory("toc-index-repro-"); const path = join(directory, "carrier"); await writeFile(path, bytes); const session = await createArtifactBinarySession(process.env); const client = await mcp.connect(createServer({ kind: "session", session })); const call = (name: string, args: Record<string, unknown>) => artifactMcpCall(client, { name, arguments: args }); const opened = await call("open_binary", { path }); expect(opened.isError).not.toBe(true); const reply = await call("inspect_artifact", {}); if (index === null) { expect(reply.isError).not.toBe(true); const inspection = artifactInspectionResultSchema.parse(parseEvidence(reply.structuredContent).normalized_result); const inventory = inspection.substeps[0]!.evidence.normalized_result; const rows = inventory.occurrences.filter((row) => row.format_metadata?.kind === "pyinstaller-entry"); expect(rows.map((row) => row.format_metadata.index)).toEqual([0, 1, 2]); console.log(JSON.stringify({ case: "valid", indices: rows.map((row) => row.format_metadata.index) })); } else { expect(reply.isError).toBe(true); console.log(JSON.stringify({ corruptedIndex: index, reply })); const text = reply.content.find((block) => block.type === "text"); const error = JSON.parse((text as { text: string }).text).error; expect(error.code).toBe("artifact_operation_failed"); expect(error.details).toEqual({ operation: "inspect_artifact", reason: "format", detail: "CArchive compression flag 2 is not stored or zlib" }); expect(JSON.stringify(reply)).not.toContain("table index"); } }); } ``` Run with an isolated configuration and a single worker: ```sh repro_root="$(mktemp -d)" mkdir -p "$repro_root/home" "$repro_root/config" "$repro_root/cache" "$repro_root/temp" env -i PATH="$PATH" HOME="$repro_root/home" DSH_HOME="$repro_root/home" XDG_CONFIG_HOME="$repro_root/config" XDG_CACHE_HOME="$repro_root/cache" TMPDIR="$repro_root/temp" node node_modules/vitest/vitest.mjs run tests/boundary/mcp/tocIndexRepro.test.ts --maxWorkers=1 ``` The test generates a valid ELF carrier with a PyInstaller 2.1 CArchive containing three stored data members, then tests independent copies with only the compression flag at index 0, 1 or 2 changed to 2. It calls the registered `open_binary` and `inspect_artifact` tools through the SDK in-memory transport and validates advertised output schemas. Its assertions characterize the current bug, so all four cases pass while the index is absent. Remove the test and disposable configuration directory after use. The exact test content ran twice in isolated configurations: the valid control succeeded, and all three corrupted-index cases reproduced the missing identifier on both runs. No target code, native provider, CLI or stdio server was executed. ### Expected behavior Malformed CArchive TOC diagnostics should identify the damaged entry by its table index, as the Python packaged applications guide promises. The three malformed cases should therefore name indices 0, 1 and 2 respectively while retaining the format refusal. The [Python packaged applications guide](https://github.com/morluto/rea/blob/055d1a9e1bd95f6dc7f49e5b37cbdd245b07b792/docs/python-packaged-applications.md#L47-L48) explicitly states: "Malformed TOC diagnostics name the table index". ### Impact and scope P3, diagnostics-only. The malformed compression flag is still rejected. The observed difference is missing row-identifying text, with no incorrect successful inventory, changed failure classification or security-boundary crossing demonstrated. The other five parse failures, extraction, CLI, stdio transport, other platforms and real-world frequency were not independently tested. Manually decoding TOC boundaries is a possible workaround, but an alternate end-user diagnostic workflow was not verified. ### REA version Source commit 055d1a9e1bd95f6dc7f49e5b37cbdd245b07b792 (main) ### Runtime and platform Node v24.18.0; macOS 27.0.1 arm64; repository-pinned Vitest 5.0.3; locked MCP SDK. Node matches the repository pin. No external analysis engine was used. ### Target and artifact context Synthetic ELF prefix plus PyInstaller 2.1 CArchive, Python version field 309, containing stored type-x members a.txt, b.txt and c.txt whose payloads equal their names. Each malformed copy changes one compression-flag byte to 2; no archive offsets or lengths change. ### Interface MCP ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The valid control prints `{"case":"valid","indices":[0,1,2]}`. For each `corruptedIndex` value 0, 1 and 2, the complete public error object is identical: ```json { "code": "artifact_operation_failed", "category": "execution_failure", "message": "Artifact is invalid or has changed. Get a fresh copy and try again.", "retryable": false, "remediation": { "action": "Artifact is invalid or has changed. Get a fresh copy and try again." }, "details": { "operation": "inspect_artifact", "reason": "format", "detail": "CArchive compression flag 2 is not stored or zlib" } } ``` One test file and all four characterization cases passed. The complete MCP replies contain no row identifier. These observations establish this compression-flag case only. ### Typed outcome or recovery expected Keep the existing malformed-input format refusal and include the corresponding table index in the diagnostic. The valid control must continue to inventory all three members. Moving the invalid flag between entries must change the reported index. ### Analysis or suspected cause Source-derived: [the compression-flag validation](https://github.com/morluto/rea/blob/055d1a9e1bd95f6dc7f49e5b37cbdd245b07b792/src/artifacts/pyinstaller/CArchiveLayout.ts#L169-L186) constructs the message before appending the entry and does not include its current index. This explains the observed public detail. No repaired software was tested. ### Additional context Open and closed issues and PRs were searched with `PyInstaller TOC`, `CArchive diagnostic`, `table index`, `CArchive`, `malformed TOC` and `compression flag`. Related #2433 and #3678 describe the original capability; #6120 concerns negative PYZ sizes, #5836 capacity recovery, #6067 changed-source admission, and #6147 a separate proposed extractor. No same-behavior duplicate was found, including the original PR's diagnostic review comments. Those comments concern oversized names and path previews. Fetched main at `3a42a9e356da45106640b4d84468f552c18bcf78` has no relevant changes to the parser, reader, artifact-session composition or guide since the tested baseline. Current upstream was source-inspected rather than executed. No known-good revision was established.

bsakaguchi closed this issue

2026-10-31 00:44:19 +00:00

bsakaguchi commented

2026-10-31 00:44:19 +00:00

Author

Owner

@morluto commented on GitHub (Oct 23, 2026):

Resolved on main in 8425c36da. All TOC row format failures use one index/offset owner. The recorded integration controls explicitly pass for corrupted compression flags at indices0,1,2 and first verify the valid three-member reader. Validation: src/artifacts/pyinstaller/PyInstallerFormatBoundaries.test.ts — identifies malformed CArchive compression at table index0/1/2; recorded execution at e1d9ebd3c. Three exact controls passed even though the file/group failed on a different drive-prefixed diagnostic bound. Later selected rerun covered that different failure; do not call this a complete green PyInstaller suite.

@morluto commented on GitHub (Oct 23, 2026): Resolved on main in [8425c36da](https://github.com/morluto/rea/commit/8425c36da2dbcf9dffc6d40140604d5108e437eb). All TOC row format failures use one index/offset owner. The recorded integration controls explicitly pass for corrupted compression flags at indices0,1,2 and first verify the valid three-member reader. Validation: `src/artifacts/pyinstaller/PyInstallerFormatBoundaries.test.ts` — identifies malformed CArchive compression at table index0/1/2; recorded execution at `e1d9ebd3c`. Three exact controls passed even though the file/group failed on a different drive-prefixed diagnostic bound. Later selected rerun covered that different failure; do not call this a complete green PyInstaller suite.

bsakaguchi referenced this issue

2026-10-31 01:11:10 +00:00

[PR #4041] [MERGED] perf(catalog): record default catalog digests at build time #6011

bsakaguchi referenced this issue

2026-10-31 01:11:25 +00:00

[PR #4083] perf(cli): skip command schemas for standalone version queries #6045

bsakaguchi referenced this issue

2026-11-01 21:10:53 +00:00

[GH-ISSUE #3582] [Tracker] Ready to close or merge: issues and pull requests (October 2026) #1330

bsakaguchi referenced this issue

2026-11-04 00:49:08 +00:00

[GH-ISSUE #3813] perf: CLI startup and doctor catalog digests make clientProfileLifecycle time out on CI #1475