morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6609] [Bug] inspect-python-pyc rejects backward-compatible TYPE_INT64 marshal constants #3994

New issue

Closed

opened 2026-10-31 00:45:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:45:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6609

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The shared Python pyc service rejects a 109-byte synthetic CPython 3.11-layout file containing the backward-compatible marshal TYPE_INT64 tag (0x49) for integer 42. It returns AnalysisInputError with reason: "invalid_format" and Unsupported marshal type 0x49. Replacing only that integer encoding with TYPE_INT produces a 105-byte control that succeeds with complete coverage and constant {"kind":"int","bits":6,"value":"42"}.

Both bodies load with the installed CPython 3.14.8 interpreter and yield co_consts == (42,). This is supplementary decoding evidence, not a matching CPython 3.11 runtime test. Published CPython 3.11 and 3.13 sources explicitly retain the TYPE_INT64 reader for backward compatibility.

Reproduction

From the source revision below with locked dependencies already present and Node v24.22.1 on PATH, save this executed probe as /tmp/pyc-int64-probe.ts. The temporary paths below are normalized. The probe creates two tiny local files, calls the real parser/provider/application composition, retains its responses, and closes the service. It does not execute bytecode.

mkdir -p /tmp/pyc-int64-fixtures
node --import ./node_modules/tsx/dist/loader.mjs /tmp/pyc-int64-probe.ts "$PWD" /tmp/pyc-int64-fixtures

import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import { writeFile } from 'node:fs/promises';
import { createHash } from 'node:crypto';

async function main() {
  const repo = process.argv[2];
  const out = process.argv[3];
  const { marshalCode, pythonPycHeader } = await import(pathToFileURL(`${repo}/src/python/PythonPyc.fixture.ts`).href);
  const { createPythonPycService } = await import(pathToFileURL(`${repo}/src/composition/pythonPyc.ts`).href);
  const i64 = Buffer.alloc(9); i64[0] = 0x49; i64.writeBigInt64LE(42n, 1);
  const int = Buffer.alloc(5); int[0] = 0x69; int.writeInt32LE(42, 1);
  for (const [name, constant] of [['int64', i64], ['int32-control', int]] as const) {
    const bytes = Buffer.concat([pythonPycHeader(3495), marshalCode('py311', { consts: [constant] })]);
    const file = `${out}/${name}.pyc`;
    await writeFile(file, bytes);
    const service = createPythonPycService();
    try {
      const result = await service.inspect({ path: file });
      await writeFile(`${out}/${name}-service.json`, JSON.stringify(result, null, 2));
      console.log(JSON.stringify({case:name, bytes:bytes.length, sha256:createHash('sha256').update(bytes).digest('hex'), ok:result.ok, ...(result.ok ? {result:result.value.normalized_result} : {error:result.error.toJSON?.() ?? result.error})}));
    } finally { await service.close(); }
  }
}
main().catch(error => { console.error(error); process.exitCode = 1; });

The following oracle was also executed with CPython 3.14.8 against those same generated files (paths normalized):

python3 -I -S /tmp/pyc-int64-oracle.py /tmp/pyc-int64-fixtures

import hashlib, json, marshal, pathlib, sys
root = pathlib.Path(sys.argv[1])
print(sys.version)
for name in ('int64', 'int32-control'):
    b = (root / (name + '.pyc')).read_bytes()
    try:
        c = marshal.loads(b[16:])
        print(json.dumps(dict(case=name, bytes=len(b), sha256=hashlib.sha256(b).hexdigest(), loaded=True, consts=c.co_consts, name=c.co_name, bytecode=c.co_code.hex())))
    except Exception as e:
        print(json.dumps(dict(case=name, loaded=False, error_type=type(e).__name__, error=str(e))))
print('primitive_int64:', marshal.loads(bytes.fromhex('492a00000000000000')))

Each case was run once through the service and once through the supplementary CPython decoder. No matching 3.11 interpreter was available; no CLI or MCP adapter was run.

Expected behavior

Recognize the backward-compatible TYPE_INT64 representation and report the same integer value as TYPE_INT or TYPE_LONG, rather than classifying the selected file as malformed.

CPython labels this tag as retained for backward compatibility and dispatches it through r_long64: CPython 3.11.14 marshal reader, CPython 3.13.9 reader. REA documents inspection of supported CPython 3.7–3.13 marshal code objects and integer summaries: pyc contract.

Impact and scope

A supported-layout file using this legacy integer tag cannot be inspected at all. The failure occurs even for the value 42, far below all configured budgets. This is a bounded compatibility defect, not evidence of failures in ordinary current compiler output: CPython no longer emits the legacy tag. Real-world prevalence and other legacy integer values were not tested.

For the shown benign fixture, replacing the legacy encoding with the ordinary integer encoding preserves the decoded constant and makes inspection succeed; the original selected file still fails. General reserialization and matching-interpreter workarounds were not tested.

REA version

Source main 572d0c234089605f120dcfea340f11e38a3c3705, package metadata 6.7.0. This is unreleased source behavior: git show rea-agents-6.7.0:src/python/PythonPycParser.ts reports that the file is absent at published tag rea-agents-6.7.0 (27dc42d10249f0410904a75c0997842eb03b931c). The affected provider, service, and domain files are also absent there. No published-package reproduction is claimed.

Runtime and platform

Node v24.22.1; locked tsx 4.23.15; Linux 7.2.8-9-omarchy x86_64. CPython 3.14.8 was used only for supplementary decoding of the benign synthetic bodies; no native analysis engine was involved.

Target and artifact context

Synthetic acyclic .pyc files with CPython 3.11 magic 3495, timestamp header flags 0, no names or locals, one module code object, and one integer constant. int64.pyc: 109 bytes, SHA-256 4fb56abeb05dbe5f9e8eae49627894b3d9f774647788a1304427f60d5cb45574. int32-control.pyc: 105 bytes, SHA-256 f369b797d037ea976df3940e1bdb67fa249966a4de99a3d7c02bd0e11f12328d.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Smallest relevant service observations (temporary paths normalized):

{"case":"int64","bytes":109,"ok":false,"error":{"operation":"inspect_python_pyc","issues":[{"path":["path"],"reason":"invalid_format","message":"Selected pyc /tmp/pyc-int64-fixtures/int64.pyc: Unsupported marshal type 0x49."}],"_tag":"AnalysisInputError"}}
{"case":"int32-control","bytes":105,"ok":true,"consts":[{"kind":"int","bits":6,"value":"42"}],"coverage":{"status":"complete","objects":12,"code_objects":1,"marshal_bytes":89,"marshal_depth":2}}

Supplementary CPython 3.14.8 output:

{"case": "int64", "bytes": 109, "sha256": "4fb56abeb05dbe5f9e8eae49627894b3d9f774647788a1304427f60d5cb45574", "loaded": true, "consts": [42], "name": "<module>", "bytecode": "5300"}
{"case": "int32-control", "bytes": 105, "sha256": "f369b797d037ea976df3940e1bdb67fa249966a4de99a3d7c02bd0e11f12328d", "loaded": true, "consts": [42], "name": "<module>", "bytecode": "5300"}
primitive_int64: 42

Typed outcome or recovery expected

Successful Evidence with the integer constant 42, as in the ordinary-integer control. The legacy tag should not produce an invalid_format input issue merely because its representation differs.

Analysis or suspected cause

Source-derived: the marshal decoder's dispatch handles TYPE_INT and TYPE_LONG, but no TYPE_INT64. The default branch throws the observed malformed-input error. No repair was attempted or verified.

Additional context

Before drafting, searches covered the initial OPEN snapshot case-insensitively for python|pyc|marshal, CLOSED issues for pyc and marshal, and all-state issues for int64 and "64-bit" "marshal". No matching report was found. Related #6568 concerns historical header lengths; #6487 concerns negative code flags; neither reports this legacy scalar tag. The existing code equality findings #6593, #6597 and #6415 are independent and excluded from this report.

The research did not modify repository source, build REA, run a full suite, or execute the target. Matching CPython 3.11 execution and CLI/MCP transport behavior remain untested.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The real parser/provider/application path rejects a bounded supported-layout input containing a scalar tag that CPython retains for compatibility; the ordinary encoding of the same value succeeds. Current compiler-produced files were not shown to fail.
Workaround status: available
Workaround basis: The executed 105-byte ordinary-integer control preserves the shown decoded constant 42 and is accepted with complete coverage. This is a demonstrated workaround for the reduced benign fixture, not a tested general reserialization procedure.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6609 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The shared Python pyc service rejects a 109-byte synthetic CPython 3.11-layout file containing the backward-compatible marshal `TYPE_INT64` tag (`0x49`) for integer `42`. It returns `AnalysisInputError` with `reason: "invalid_format"` and `Unsupported marshal type 0x49.` Replacing only that integer encoding with `TYPE_INT` produces a 105-byte control that succeeds with complete coverage and constant `{"kind":"int","bits":6,"value":"42"}`. Both bodies load with the installed CPython 3.14.8 interpreter and yield `co_consts == (42,)`. This is supplementary decoding evidence, not a matching CPython 3.11 runtime test. Published CPython 3.11 and 3.13 sources explicitly retain the `TYPE_INT64` reader for backward compatibility. ## Reproduction From the source revision below with locked dependencies already present and Node v24.22.1 on PATH, save this executed probe as `/tmp/pyc-int64-probe.ts`. The temporary paths below are normalized. The probe creates two tiny local files, calls the real parser/provider/application composition, retains its responses, and closes the service. It does not execute bytecode. ```sh mkdir -p /tmp/pyc-int64-fixtures node --import ./node_modules/tsx/dist/loader.mjs /tmp/pyc-int64-probe.ts "$PWD" /tmp/pyc-int64-fixtures ``` ```typescript import { createRequire } from 'node:module'; import { pathToFileURL } from 'node:url'; import { writeFile } from 'node:fs/promises'; import { createHash } from 'node:crypto'; async function main() { const repo = process.argv[2]; const out = process.argv[3]; const { marshalCode, pythonPycHeader } = await import(pathToFileURL(`${repo}/src/python/PythonPyc.fixture.ts`).href); const { createPythonPycService } = await import(pathToFileURL(`${repo}/src/composition/pythonPyc.ts`).href); const i64 = Buffer.alloc(9); i64[0] = 0x49; i64.writeBigInt64LE(42n, 1); const int = Buffer.alloc(5); int[0] = 0x69; int.writeInt32LE(42, 1); for (const [name, constant] of [['int64', i64], ['int32-control', int]] as const) { const bytes = Buffer.concat([pythonPycHeader(3495), marshalCode('py311', { consts: [constant] })]); const file = `${out}/${name}.pyc`; await writeFile(file, bytes); const service = createPythonPycService(); try { const result = await service.inspect({ path: file }); await writeFile(`${out}/${name}-service.json`, JSON.stringify(result, null, 2)); console.log(JSON.stringify({case:name, bytes:bytes.length, sha256:createHash('sha256').update(bytes).digest('hex'), ok:result.ok, ...(result.ok ? {result:result.value.normalized_result} : {error:result.error.toJSON?.() ?? result.error})})); } finally { await service.close(); } } } main().catch(error => { console.error(error); process.exitCode = 1; }); ``` The following oracle was also executed with CPython 3.14.8 against those same generated files (paths normalized): ```sh python3 -I -S /tmp/pyc-int64-oracle.py /tmp/pyc-int64-fixtures ``` ```python import hashlib, json, marshal, pathlib, sys root = pathlib.Path(sys.argv[1]) print(sys.version) for name in ('int64', 'int32-control'): b = (root / (name + '.pyc')).read_bytes() try: c = marshal.loads(b[16:]) print(json.dumps(dict(case=name, bytes=len(b), sha256=hashlib.sha256(b).hexdigest(), loaded=True, consts=c.co_consts, name=c.co_name, bytecode=c.co_code.hex()))) except Exception as e: print(json.dumps(dict(case=name, loaded=False, error_type=type(e).__name__, error=str(e)))) print('primitive_int64:', marshal.loads(bytes.fromhex('492a00000000000000'))) ``` Each case was run once through the service and once through the supplementary CPython decoder. No matching 3.11 interpreter was available; no CLI or MCP adapter was run. ## Expected behavior Recognize the backward-compatible `TYPE_INT64` representation and report the same integer value as `TYPE_INT` or `TYPE_LONG`, rather than classifying the selected file as malformed. CPython labels this tag as retained for backward compatibility and dispatches it through `r_long64`: [CPython 3.11.14 marshal reader](https://github.com/python/cpython/blob/v3.11.14/Python/marshal.c#L1052), [CPython 3.13.9 reader](https://github.com/python/cpython/blob/v3.13.9/Python/marshal.c#L1074). REA documents inspection of supported CPython 3.7–3.13 marshal code objects and integer summaries: [pyc contract](https://github.com/morluto/rea/blob/572d0c234089605f120dcfea340f11e38a3c3705/docs/python-pyc.md#L53). ## Impact and scope A supported-layout file using this legacy integer tag cannot be inspected at all. The failure occurs even for the value `42`, far below all configured budgets. This is a bounded compatibility defect, not evidence of failures in ordinary current compiler output: CPython no longer emits the legacy tag. Real-world prevalence and other legacy integer values were not tested. For the shown benign fixture, replacing the legacy encoding with the ordinary integer encoding preserves the decoded constant and makes inspection succeed; the original selected file still fails. General reserialization and matching-interpreter workarounds were not tested. ## REA version Source main `572d0c234089605f120dcfea340f11e38a3c3705`, package metadata `6.7.0`. This is unreleased source behavior: `git show rea-agents-6.7.0:src/python/PythonPycParser.ts` reports that the file is absent at published tag `rea-agents-6.7.0` (`27dc42d10249f0410904a75c0997842eb03b931c`). The affected provider, service, and domain files are also absent there. No published-package reproduction is claimed. ## Runtime and platform Node v24.22.1; locked tsx 4.23.15; Linux 7.2.8-9-omarchy x86_64. CPython 3.14.8 was used only for supplementary decoding of the benign synthetic bodies; no native analysis engine was involved. ## Target and artifact context Synthetic acyclic `.pyc` files with CPython 3.11 magic 3495, timestamp header flags 0, no names or locals, one module code object, and one integer constant. `int64.pyc`: 109 bytes, SHA-256 `4fb56abeb05dbe5f9e8eae49627894b3d9f774647788a1304427f60d5cb45574`. `int32-control.pyc`: 105 bytes, SHA-256 `f369b797d037ea976df3940e1bdb67fa249966a4de99a3d7c02bd0e11f12328d`. ## Interface Other ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Smallest relevant service observations (temporary paths normalized): ```json {"case":"int64","bytes":109,"ok":false,"error":{"operation":"inspect_python_pyc","issues":[{"path":["path"],"reason":"invalid_format","message":"Selected pyc /tmp/pyc-int64-fixtures/int64.pyc: Unsupported marshal type 0x49."}],"_tag":"AnalysisInputError"}} {"case":"int32-control","bytes":105,"ok":true,"consts":[{"kind":"int","bits":6,"value":"42"}],"coverage":{"status":"complete","objects":12,"code_objects":1,"marshal_bytes":89,"marshal_depth":2}} ``` Supplementary CPython 3.14.8 output: ```text {"case": "int64", "bytes": 109, "sha256": "4fb56abeb05dbe5f9e8eae49627894b3d9f774647788a1304427f60d5cb45574", "loaded": true, "consts": [42], "name": "<module>", "bytecode": "5300"} {"case": "int32-control", "bytes": 105, "sha256": "f369b797d037ea976df3940e1bdb67fa249966a4de99a3d7c02bd0e11f12328d", "loaded": true, "consts": [42], "name": "<module>", "bytecode": "5300"} primitive_int64: 42 ``` ## Typed outcome or recovery expected Successful Evidence with the integer constant `42`, as in the ordinary-integer control. The legacy tag should not produce an `invalid_format` input issue merely because its representation differs. ## Analysis or suspected cause Source-derived: the [marshal decoder's dispatch](https://github.com/morluto/rea/blob/572d0c234089605f120dcfea340f11e38a3c3705/src/python/PythonPycParser.ts#L366) handles `TYPE_INT` and `TYPE_LONG`, but no `TYPE_INT64`. The [default branch](https://github.com/morluto/rea/blob/572d0c234089605f120dcfea340f11e38a3c3705/src/python/PythonPycParser.ts#L411) throws the observed malformed-input error. No repair was attempted or verified. ## Additional context Before drafting, searches covered the initial OPEN snapshot case-insensitively for `python|pyc|marshal`, CLOSED issues for `pyc` and `marshal`, and all-state issues for `int64` and `"64-bit" "marshal"`. No matching report was found. Related #6568 concerns historical header lengths; #6487 concerns negative code flags; neither reports this legacy scalar tag. The existing code equality findings #6593, #6597 and #6415 are independent and excluded from this report. The research did not modify repository source, build REA, run a full suite, or execute the target. Matching CPython 3.11 execution and CLI/MCP transport behavior remain untested. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The real parser/provider/application path rejects a bounded supported-layout input containing a scalar tag that CPython retains for compatibility; the ordinary encoding of the same value succeeds. Current compiler-produced files were not shown to fail. - Workaround status: available - Workaround basis: The executed 105-byte ordinary-integer control preserves the shown decoded constant 42 and is accepted with complete coverage. This is a demonstrated workaround for the reduced benign fixture, not a tested general reserialization procedure.

bsakaguchi closed this issue

2026-10-31 00:45:53 +00:00

bsakaguchi commented

2026-10-31 00:45:54 +00:00

Author

Owner

@ezrabrilliant commented on GitHub (Oct 23, 2026):

I'd like to take this one. The marshal reader in src/python/PythonPycParser.ts has no TYPE_INT64 (0x49) case, so it falls through to Unsupported marshal type 0x49. CPython retains the r_long64 reader for backward compatibility.

Planned fix: add the TYPE_INT64 tag and read it as an 8-byte little-endian signed integer (mirroring the existing TYPE_INT handling and smallBitLength), with regression coverage for a small value, a value whose high 32 bits are set, and a negative value.

@ezrabrilliant commented on GitHub (Oct 23, 2026): I'd like to take this one. The marshal reader in `src/python/PythonPycParser.ts` has no `TYPE_INT64` (`0x49`) case, so it falls through to `Unsupported marshal type 0x49`. CPython retains the `r_long64` reader for backward compatibility. Planned fix: add the `TYPE_INT64` tag and read it as an 8-byte little-endian signed integer (mirroring the existing `TYPE_INT` handling and `smallBitLength`), with regression coverage for a small value, a value whose high 32 bits are set, and a negative value.

bsakaguchi referenced this issue

2026-10-31 01:10:54 +00:00

[PR #3994] [CLOSED] fix: exclude credential header values from header allocation budget preflight #5980

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425