[GH-ISSUE #6547] [Bug] Python 3.8–3.10 code keys with different line tables produce incorrect container counts #3937

New issue

Closed

opened 2026-10-31 00:45:24 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:24 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6547

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed call to the shared production Python inspection service reports a frozenset containing two otherwise identical code objects as length 2, and a dictionary using those code objects as keys as length 2, when only their line tables differ. It returns successful Evidence with complete coverage. The identical-line-table controls report length 1. CPython 3.8 and 3.10 source establishes that those code objects compare equal and have equal hashes because line-table bytes participate in neither equality nor hashing; their containers should therefore contain one entry. The CPython semantic oracle here is source-derived, not an executed matching interpreter.

Reproduction

Use the source revision below with its locked dependencies already installed. Save the following program as probe.mjs outside the checkout. It creates only inert synthetic pyc files and invokes the same production service used by CLI and MCP; it does not execute bytecode. This complete eight-case program was run once with isolated HOME, configuration, and cache directories. The CLI and MCP transports were not executed.

import { mkdir, writeFile } from 'node:fs/promises';
import { resolve, join } from 'node:path';
import { pathToFileURL } from 'node:url';

const repository = resolve(process.argv[2]);
const root = resolve(process.argv[3]);
const { createPythonPycService } = await import(pathToFileURL(join(repository, 'src/composition/pythonPyc.ts')));
const { marshalCode, pythonPycHeader } = await import(pathToFileURL(join(repository, 'src/python/PythonPyc.fixture.ts')));
await mkdir(root, { recursive: true });
const u32 = n => { const b = Buffer.alloc(4); b.writeUInt32LE(n); return b; };
const service = createPythonPycService();
try {
  for (const [version, layout, magic] of [
    ['3.7', 'py37', 3394], ['3.8', 'py38', 3413],
    ['3.10', 'py38', 3439], ['3.12', 'py311', 3531],
  ]) {
    for (const different of [false, true]) {
      const code = linetable => marshalCode(layout, {
        name: 'f', filename: 'fixture.py', firstlineno: 1, linetable,
      });
      const first = code(Buffer.from([2, 1]));
      const second = code(Buffer.from([2, different ? 2 : 1]));
      const frozen = Buffer.concat([Buffer.from([0x3e]), u32(2), first, second]);
      const dict = Buffer.concat([Buffer.from([0x7b]), first, Buffer.from([0x4e]), second, Buffer.from([0x54, 0x30])]);
      const bytes = Buffer.concat([pythonPycHeader(magic), marshalCode(layout, { consts: [frozen, dict] })]);
      const path = join(root, `py${version}-${different ? 'different' : 'same'}.pyc`);
      await writeFile(path, bytes);
      const result = await service.inspect({ path });
      if (!result.ok) throw result.error;
      const report = result.value.normalized_result;
      console.log(JSON.stringify({ version, different_line_table: different,
        frozenset: report.code_objects[0].consts[0],
        dict: report.code_objects[0].consts[1],
        coverage: report.coverage.status }));
    }
  }
} finally { await service.close(); }

From the checkout root, run the saved program using the checkout's locked TypeScript loader and a scratch directory: node --import ./node_modules/tsx/dist/loader.mjs /absolute/scratch/probe.mjs "$PWD" /absolute/scratch/fixtures. The three arguments identify the program, checkout, and fixture directory; use fresh scratch home/config/cache directories for the process. No Python or native analysis provider is required.

Expected behavior

The 3.8 and 3.10 cases with differing line tables should report length 1 for both containers, just like their identical-line-table controls. CPython's 3.8 equality and hash implementation and 3.10 equality and hash implementation omit co_lnotab/co_linetable; the constructor accepts the byte-string line table without interpreting it for equality. Later versions need their own version-specific rules: 3.13 explicitly compares the line table, so simply removing it for every version would be incorrect.

Impact and scope

The demonstrated defect is a bounded incorrect count in successful observed Evidence for synthetic CPython 3.8 and 3.10 inputs containing code objects in hash-based containers. Dictionary key replacement and which nested code objects remain reachable can consequently differ from the target interpreter's semantics. Normal compiler output rarely needs such duplicate code keys; real-world frequency is unknown. The shared py38 layout also covers 3.9, but no 3.9 fixture was executed. No input mutation or target execution occurred. A matching CPython interpreter could provide an external semantic check, but that workaround was not tested.

REA version

Source main: 1635eadfcc49bbb80178838ab74667c2981de383 (package manifest 6.7.0). Latest published release rea-agents-6.7.0 was inspected first and does not contain the Python pyc capability; this finding concerns the capability added on main, not the published release.

Runtime and platform

Node.js v26.12.1; Linux x86_64. The locked tsx loader executed TypeScript source. No Hopper, Ghidra, IDA, or matching CPython runtime was used.

Target and artifact context

Synthetic timestamp-header pyc files with magic 3413 (3.8) or 3439 (3.10), a top-level code object, and frozenset/dictionary constants containing two code records that differ only in line-table bytes 02 01 versus 02 02. Matching-line-table controls and additional 3.7/3.12 fixtures were also inspected. The fixtures were never executed.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The executed production-service matrix returned these relevant rows; different_line_table: false is the control. Each case was executed once.

{"version":"3.8","different_line_table":false,"frozenset":{"kind":"frozenset","length":1},"dict":{"kind":"dict","length":1},"coverage":"complete"}
{"version":"3.8","different_line_table":true,"frozenset":{"kind":"frozenset","length":2},"dict":{"kind":"dict","length":2},"coverage":"complete"}
{"version":"3.10","different_line_table":false,"frozenset":{"kind":"frozenset","length":1},"dict":{"kind":"dict","length":1},"coverage":"complete"}
{"version":"3.10","different_line_table":true,"frozenset":{"kind":"frozenset","length":2},"dict":{"kind":"dict","length":2},"coverage":"complete"}

The production equality key unconditionally includes the line-table digest, which explains the executed distinction. The public tool contract admits these versions without an external provider. The CLI calls the same service, and the service validates the result before marking it observed Evidence; the fixture requests passed those guards.

Evidence provenance: executed REA production-service results plus source-derived CPython semantics. NOT_ENOUGH_INFO for an independent matching-CPython load, CLI/MCP transport reproduction, and real compiler-produced affected artifacts. Searches of the open-issue snapshot and all-state GitHub queries for co_lnotab, co_linetable, Python code line table equality, and marshal frozenset code constants found no existing report of this behavior.

Analysis or suspected cause

MarshalReader.#codeKey applies the same equality fields to all supported layouts. It includes material.linetable for legacy layouts, causing equal CPython keys to receive different REA tokens. Set deduplication and dictionary replacement both consume those tokens. Expected version-specific behavior is established by the pinned CPython source references above; no fix was attempted.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed shared production service returns bounded incorrect container counts as complete observed Evidence for the reduced supported-version inputs; the expected equality is established by the corresponding CPython implementations.
Workaround status: unverified
Workaround basis: Inspection with a matching CPython runtime is a plausible external check, but no matching interpreter was available or run, and no REA workaround was demonstrated.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6547 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed call to the shared production Python inspection service reports a frozenset containing two otherwise identical code objects as length 2, and a dictionary using those code objects as keys as length 2, when only their line tables differ. It returns successful Evidence with complete coverage. The identical-line-table controls report length 1. CPython 3.8 and 3.10 source establishes that those code objects compare equal and have equal hashes because line-table bytes participate in neither equality nor hashing; their containers should therefore contain one entry. The CPython semantic oracle here is source-derived, not an executed matching interpreter. ### Reproduction Use the source revision below with its locked dependencies already installed. Save the following program as `probe.mjs` outside the checkout. It creates only inert synthetic pyc files and invokes the same production service used by CLI and MCP; it does not execute bytecode. This complete eight-case program was run once with isolated HOME, configuration, and cache directories. The CLI and MCP transports were not executed. ```js import { mkdir, writeFile } from 'node:fs/promises'; import { resolve, join } from 'node:path'; import { pathToFileURL } from 'node:url'; const repository = resolve(process.argv[2]); const root = resolve(process.argv[3]); const { createPythonPycService } = await import(pathToFileURL(join(repository, 'src/composition/pythonPyc.ts'))); const { marshalCode, pythonPycHeader } = await import(pathToFileURL(join(repository, 'src/python/PythonPyc.fixture.ts'))); await mkdir(root, { recursive: true }); const u32 = n => { const b = Buffer.alloc(4); b.writeUInt32LE(n); return b; }; const service = createPythonPycService(); try { for (const [version, layout, magic] of [ ['3.7', 'py37', 3394], ['3.8', 'py38', 3413], ['3.10', 'py38', 3439], ['3.12', 'py311', 3531], ]) { for (const different of [false, true]) { const code = linetable => marshalCode(layout, { name: 'f', filename: 'fixture.py', firstlineno: 1, linetable, }); const first = code(Buffer.from([2, 1])); const second = code(Buffer.from([2, different ? 2 : 1])); const frozen = Buffer.concat([Buffer.from([0x3e]), u32(2), first, second]); const dict = Buffer.concat([Buffer.from([0x7b]), first, Buffer.from([0x4e]), second, Buffer.from([0x54, 0x30])]); const bytes = Buffer.concat([pythonPycHeader(magic), marshalCode(layout, { consts: [frozen, dict] })]); const path = join(root, `py${version}-${different ? 'different' : 'same'}.pyc`); await writeFile(path, bytes); const result = await service.inspect({ path }); if (!result.ok) throw result.error; const report = result.value.normalized_result; console.log(JSON.stringify({ version, different_line_table: different, frozenset: report.code_objects[0].consts[0], dict: report.code_objects[0].consts[1], coverage: report.coverage.status })); } } } finally { await service.close(); } ``` From the checkout root, run the saved program using the checkout's locked TypeScript loader and a scratch directory: `node --import ./node_modules/tsx/dist/loader.mjs /absolute/scratch/probe.mjs "$PWD" /absolute/scratch/fixtures`. The three arguments identify the program, checkout, and fixture directory; use fresh scratch home/config/cache directories for the process. No Python or native analysis provider is required. ### Expected behavior The 3.8 and 3.10 cases with differing line tables should report length 1 for both containers, just like their identical-line-table controls. CPython's [3.8 equality and hash implementation](https://github.com/python/cpython/blob/v3.8.0/Objects/codeobject.c#L801-L908) and [3.10 equality and hash implementation](https://github.com/python/cpython/blob/v3.10.0/Objects/codeobject.c#L914-L1023) omit `co_lnotab`/`co_linetable`; the constructor accepts the byte-string line table without interpreting it for equality. Later versions need their own version-specific rules: [3.13 explicitly compares the line table](https://github.com/python/cpython/blob/v3.13.0/Objects/codeobject.c#L1991-L2002), so simply removing it for every version would be incorrect. ### Impact and scope The demonstrated defect is a bounded incorrect count in successful observed Evidence for synthetic CPython 3.8 and 3.10 inputs containing code objects in hash-based containers. Dictionary key replacement and which nested code objects remain reachable can consequently differ from the target interpreter's semantics. Normal compiler output rarely needs such duplicate code keys; real-world frequency is unknown. The shared `py38` layout also covers 3.9, but no 3.9 fixture was executed. No input mutation or target execution occurred. A matching CPython interpreter could provide an external semantic check, but that workaround was not tested. ### REA version Source main: `1635eadfcc49bbb80178838ab74667c2981de383` (package manifest 6.7.0). Latest published release `rea-agents-6.7.0` was inspected first and does not contain the Python pyc capability; this finding concerns the capability added on main, not the published release. ### Runtime and platform Node.js v26.12.1; Linux x86_64. The locked `tsx` loader executed TypeScript source. No Hopper, Ghidra, IDA, or matching CPython runtime was used. ### Target and artifact context Synthetic timestamp-header pyc files with magic 3413 (3.8) or 3439 (3.10), a top-level code object, and frozenset/dictionary constants containing two code records that differ only in line-table bytes `02 01` versus `02 02`. Matching-line-table controls and additional 3.7/3.12 fixtures were also inspected. The fixtures were never executed. ### Interface Other ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The executed production-service matrix returned these relevant rows; `different_line_table: false` is the control. Each case was executed once. ```json {"version":"3.8","different_line_table":false,"frozenset":{"kind":"frozenset","length":1},"dict":{"kind":"dict","length":1},"coverage":"complete"} {"version":"3.8","different_line_table":true,"frozenset":{"kind":"frozenset","length":2},"dict":{"kind":"dict","length":2},"coverage":"complete"} {"version":"3.10","different_line_table":false,"frozenset":{"kind":"frozenset","length":1},"dict":{"kind":"dict","length":1},"coverage":"complete"} {"version":"3.10","different_line_table":true,"frozenset":{"kind":"frozenset","length":2},"dict":{"kind":"dict","length":2},"coverage":"complete"} ``` The [production equality key unconditionally includes the line-table digest](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/python/PythonPycParser.ts#L1130-L1168), which explains the executed distinction. The [public tool contract](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/contracts/python/pythonPycToolContracts.ts#L10-L29) admits these versions without an external provider. The [CLI calls the same service](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/cli/pythonPycCommands.ts#L43-L67), and the [service validates the result before marking it observed Evidence](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/application/python/PythonPycService.ts#L55-L85); the fixture requests passed those guards. Evidence provenance: executed REA production-service results plus source-derived CPython semantics. NOT_ENOUGH_INFO for an independent matching-CPython load, CLI/MCP transport reproduction, and real compiler-produced affected artifacts. Searches of the open-issue snapshot and all-state GitHub queries for `co_lnotab`, `co_linetable`, `Python code line table equality`, and `marshal frozenset code constants` found no existing report of this behavior. ### Analysis or suspected cause `MarshalReader.#codeKey` applies the same equality fields to all supported layouts. It includes `material.linetable` for legacy layouts, causing equal CPython keys to receive different REA tokens. Set deduplication and dictionary replacement both consume those tokens. Expected version-specific behavior is established by the pinned CPython source references above; no fix was attempted. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed shared production service returns bounded incorrect container counts as complete observed Evidence for the reduced supported-version inputs; the expected equality is established by the corresponding CPython implementations. - Workaround status: unverified - Workaround basis: Inspection with a matching CPython runtime is a plausible external check, but no matching interpreter was available or run, and no REA workaround was demonstrated.

bsakaguchi closed this issue

2026-10-31 00:45:24 +00:00

bsakaguchi referenced this issue

2026-10-31 00:47:35 +00:00

[GH-ISSUE #6854] [Bug] Obligation ledger verifies a changed owner using proof for the previous implementation #4193

bsakaguchi referenced this issue

2026-10-31 01:12:30 +00:00

[PR #4337] [CLOSED] fix(domain): fail closed on conflicting reconstruction proofs and evaluate required claim surfaces #6181