[GH-ISSUE #6560] [Bug] Flagged marshal singletons shift pyc reference indexes and change reported constants #3951

New issue

Closed

opened 2026-10-31 00:45:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:32 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6560

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed call to the shared production Python inspection service reports constants [None, "x", None] for a marshal graph that CPython decodes as [None, "x", "x"]. Setting FLAG_REF on the first TYPE_NONE causes REA to allocate a reference-table slot that CPython does not allocate. A later TYPE_REF 0 consequently names a different object. The service succeeds with confidence: "observed" and coverage.status: "complete"; the unflagged control reports the correct string.

Reproduction

The following preserves the fixture construction and shared service calls actually executed in an external .mts program. This inline command was not separately run. Use the source checkout with locked dependencies already installed; it writes only synthetic pyc files under a fresh scratch directory and never executes target bytecode.

WORK="$(mktemp -d)" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS'
import {writeFile} from 'node:fs/promises';
import {join} from 'node:path';
import {marshalCode,pythonPycHeader} from './src/python/PythonPyc.fixture.ts';
import {createPythonPycService} from './src/composition/pythonPyc.ts';
const service = createPythonPycService();
try {
  for (const [name, hex] of [
    ['control', '29034efa01787200000000'],
    ['flagged-none', '2903cefa01787200000000'],
  ]) {
    const consts = Buffer.from(hex, 'hex');
    const bytes = Buffer.concat([pythonPycHeader(3531),
      marshalCode('py311', {constsObject: consts})]);
    const path = join(process.env.WORK, name + '.pyc');
    await writeFile(path, bytes);
    const result = await service.inspect({path});
    if (!result.ok) throw result.error;
    console.log(JSON.stringify({name, confidence: result.value.confidence,
      consts: result.value.normalized_result.code_objects[0].consts,
      coverage: result.value.normalized_result.coverage}));
  }
} finally { await service.close(); }
JS

Both cases were executed once with isolated home/config/cache/temp directories. The retained program also saved each primitive constants tuple separately and ran python3 -I -S with marshal.loads() on it. That oracle returned (None, "x", "x") for both byte sequences. It decoded primitive marshal values only; a matching CPython interpreter was not used to load the complete pyc. The public CLI/MCP adapters were not run.

Expected behavior

A marshal reference must name the same object as CPython's reference table. CPython's reader ignores the flag for singleton tags such as TYPE_NONE; only the flagged string in this fixture occupies slot 0. Both inputs should therefore report the third constant as the string "x". The CPython source for the declared pyc family explicitly implements that behavior.

Impact and scope

The inspector silently changes a decoded constant's kind and value while attesting complete observed coverage. This is a bounded correctness defect for accepted marshal inputs containing a flagged singleton before subsequent back-references. Neither ordinary compiler emission of the encoding nor real-world frequency was established; the synthetic input is accepted by CPython's primitive marshal decoder. No execution, security, or resource-exhaustion claim is made.

REA version

Main source 6c4d880f2dd9ca7d42593be916f90ec19b8a09d7, package metadata 6.7.0. Main-only finding: the Python pyc inspection source is absent from the published release tag; no published-runtime reproduction is claimed.

Runtime and platform

Node.js v26.12.1 and CPython 3.14.8, Linux x64. Source service calls used the locked tsx loader with ambient Node rather than the repository-pinned Node toolchain. The fixture declares CPython 3.12; its singleton/reference semantics are established from that family's source, while the executed Python oracle covers only the compatible primitive constants tuple.

Target and artifact context

Synthetic CPython pyc with magic 3531 and one inert code object. The constants bytes are a small tuple of three items: ce (flagged None), fa 01 78 (flagged short ASCII "x"), and 72 00 00 00 00 (reference 0). The control changes ce to unflagged 4e and leaves all other bytes identical.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

{"case":"control","cpython_consts":[null,"x","x"],"rea_consts":[{"kind":"none"},{"kind":"str","bytes":1,"truncated":false,"value":"x"},{"kind":"str","bytes":1,"truncated":false,"value":"x"}],"confidence":"observed","coverage":{"status":"complete","objects":14,"code_objects":1,"marshal_bytes":93,"marshal_depth":2}}
{"case":"flagged-none","cpython_consts":[null,"x","x"],"rea_consts":[{"kind":"none"},{"kind":"str","bytes":1,"truncated":false,"value":"x"},{"kind":"none"}],"confidence":"observed","coverage":{"status":"complete","objects":14,"code_objects":1,"marshal_bytes":93,"marshal_depth":2}}

Expected source: CPython's singleton cases return without reference registration; its string case registers the flagged string.
Failure source: The executed shared-service results above; REA reserves a reference slot before distinguishing singleton tags, then returns a None value from the singleton branch.
Evidence provenance: observed, with declared-family semantics additionally established by CPython source.
Local verification: reproduced through the production service and its real file reader; no CLI/MCP transport or full-pyc CPython execution.
Reproduction completeness: complete for the decoded-constant discrepancy.

Additional context

This concerns reference-table registration for singleton tags, independently of code-object equality/container counts (#6547), code-object flag validation (#6487), or unhashable code keys (#6415). The fixture contains no set, dictionary, code key, negative code flags, or line-table comparison.

Restoration check (not executed): Both provided byte sequences must retain the third constant as {"kind":"str","bytes":1,"truncated":false,"value":"x"} with complete coverage. The control must continue succeeding. No specific implementation is prescribed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed production service returns an incorrect constant kind/value with complete observed Evidence, and the one-byte control agrees with the primitive CPython oracle.
Workaround status: unverified
Workaround basis: No supported inspection option changes reference registration; manually decoding the primitive graph can reveal the discrepancy, but no existing end-to-end workaround was tested.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6560 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed call to the shared production Python inspection service reports constants `[None, "x", None]` for a marshal graph that CPython decodes as `[None, "x", "x"]`. Setting `FLAG_REF` on the first `TYPE_NONE` causes REA to allocate a reference-table slot that CPython does not allocate. A later `TYPE_REF 0` consequently names a different object. The service succeeds with `confidence: "observed"` and `coverage.status: "complete"`; the unflagged control reports the correct string. ### Reproduction The following preserves the fixture construction and shared service calls actually executed in an external `.mts` program. This inline command was not separately run. Use the source checkout with locked dependencies already installed; it writes only synthetic pyc files under a fresh scratch directory and never executes target bytecode. ```sh WORK="$(mktemp -d)" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS' import {writeFile} from 'node:fs/promises'; import {join} from 'node:path'; import {marshalCode,pythonPycHeader} from './src/python/PythonPyc.fixture.ts'; import {createPythonPycService} from './src/composition/pythonPyc.ts'; const service = createPythonPycService(); try { for (const [name, hex] of [ ['control', '29034efa01787200000000'], ['flagged-none', '2903cefa01787200000000'], ]) { const consts = Buffer.from(hex, 'hex'); const bytes = Buffer.concat([pythonPycHeader(3531), marshalCode('py311', {constsObject: consts})]); const path = join(process.env.WORK, name + '.pyc'); await writeFile(path, bytes); const result = await service.inspect({path}); if (!result.ok) throw result.error; console.log(JSON.stringify({name, confidence: result.value.confidence, consts: result.value.normalized_result.code_objects[0].consts, coverage: result.value.normalized_result.coverage})); } } finally { await service.close(); } JS ``` Both cases were executed once with isolated home/config/cache/temp directories. The retained program also saved each primitive constants tuple separately and ran `python3 -I -S` with `marshal.loads()` on it. That oracle returned `(None, "x", "x")` for both byte sequences. It decoded primitive marshal values only; a matching CPython interpreter was not used to load the complete pyc. The public CLI/MCP adapters were not run. ### Expected behavior A marshal reference must name the same object as CPython's reference table. CPython's reader ignores the flag for singleton tags such as `TYPE_NONE`; only the flagged string in this fixture occupies slot 0. Both inputs should therefore report the third constant as the string `"x"`. The CPython source for the declared pyc family explicitly implements that behavior. ### Impact and scope The inspector silently changes a decoded constant's kind and value while attesting complete observed coverage. This is a bounded correctness defect for accepted marshal inputs containing a flagged singleton before subsequent back-references. Neither ordinary compiler emission of the encoding nor real-world frequency was established; the synthetic input is accepted by CPython's primitive marshal decoder. No execution, security, or resource-exhaustion claim is made. ### REA version Main source `6c4d880f2dd9ca7d42593be916f90ec19b8a09d7`, package metadata 6.7.0. Main-only finding: the Python pyc inspection source is absent from the published release tag; no published-runtime reproduction is claimed. ### Runtime and platform Node.js v26.12.1 and CPython 3.14.8, Linux x64. Source service calls used the locked `tsx` loader with ambient Node rather than the repository-pinned Node toolchain. The fixture declares CPython 3.12; its singleton/reference semantics are established from that family's source, while the executed Python oracle covers only the compatible primitive constants tuple. ### Target and artifact context Synthetic CPython pyc with magic 3531 and one inert code object. The constants bytes are a small tuple of three items: `ce` (flagged None), `fa 01 78` (flagged short ASCII `"x"`), and `72 00 00 00 00` (reference 0). The control changes `ce` to unflagged `4e` and leaves all other bytes identical. ### Interface Other ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence ```json {"case":"control","cpython_consts":[null,"x","x"],"rea_consts":[{"kind":"none"},{"kind":"str","bytes":1,"truncated":false,"value":"x"},{"kind":"str","bytes":1,"truncated":false,"value":"x"}],"confidence":"observed","coverage":{"status":"complete","objects":14,"code_objects":1,"marshal_bytes":93,"marshal_depth":2}} {"case":"flagged-none","cpython_consts":[null,"x","x"],"rea_consts":[{"kind":"none"},{"kind":"str","bytes":1,"truncated":false,"value":"x"},{"kind":"none"}],"confidence":"observed","coverage":{"status":"complete","objects":14,"code_objects":1,"marshal_bytes":93,"marshal_depth":2}} ``` - Expected source: [CPython's singleton cases return without reference registration](https://github.com/python/cpython/blob/v3.12.12/Python/marshal.c#L956-L985); [its string case registers the flagged string](https://github.com/python/cpython/blob/v3.12.12/Python/marshal.c#L1087-L1112). - Failure source: The executed shared-service results above; [REA reserves a reference slot before distinguishing singleton tags](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/python/PythonPycParser.ts#L309-L320), then [returns a None value from the singleton branch](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/python/PythonPycParser.ts#L329-L341). - Evidence provenance: observed, with declared-family semantics additionally established by CPython source. - Local verification: reproduced through the production service and its real file reader; no CLI/MCP transport or full-pyc CPython execution. - Reproduction completeness: complete for the decoded-constant discrepancy. ### Additional context This concerns reference-table registration for singleton tags, independently of code-object equality/container counts (#6547), code-object flag validation (#6487), or unhashable code keys (#6415). The fixture contains no set, dictionary, code key, negative code flags, or line-table comparison. Restoration check (not executed): Both provided byte sequences must retain the third constant as `{"kind":"str","bytes":1,"truncated":false,"value":"x"}` with complete coverage. The control must continue succeeding. No specific implementation is prescribed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed production service returns an incorrect constant kind/value with complete observed Evidence, and the one-byte control agrees with the primitive CPython oracle. - Workaround status: unverified - Workaround basis: No supported inspection option changes reference registration; manually decoding the primitive graph can reveal the discrepancy, but no existing end-to-end workaround was tested.

bsakaguchi closed this issue

2026-10-31 00:45:32 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:04 +00:00

[PR #4026] [MERGED] fix(contracts): complete canonical mobile graph inventory examples #5999

bsakaguchi referenced this issue

2026-10-31 01:11:08 +00:00

[PR #4037] [CLOSED] fix(contracts): restore canonical mobile graph examples #6007