[GH-ISSUE #6490] [Bug] preserve negative code-object first line numbers #3894

New issue

Closed

opened 2026-10-31 00:45:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:00 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6490

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public inspect-python-pyc CLI exits 0 and reports firstlineno: 4294967295 for a CPython 3.12 code object whose stored first-line bytes are ffffffff. Installed CPython accepts the identical marshal payload and exposes co_firstlineno == -1, including after a marshal round trip. REA labels the result confidence: "observed" with coverage.status: "complete"; none of its four standard limitations identifies the line-number reinterpretation. The same code object with first line 1 is reported as 1 by both tools. Code flags are zero in both fixtures.

Reproduction

Use the prepared source checkout at the revision below, with its compiled runtime and repository-pinned Node.js on PATH, plus an existing CPython 3.12 interpreter. Dependency installation and runtime building are prerequisites for an unprepared checkout; neither was performed during this verification. From the checkout root, run this complete block. It writes two small synthetic pycs and isolates REA's home/config/cache/data/state/temp and DSH directories in a disposable root, then removes that root. It only loads and serializes code objects as data; no target bytecode runs.

python3.12 -I -B - "$PWD" "$(command -v node)" <<'PYTHON'
import hashlib, json, marshal, pathlib, struct, subprocess, sys, tempfile, types
assert sys.version_info[:2] == (3, 12)
repo, node = pathlib.Path(sys.argv[1]).resolve(), str(pathlib.Path(sys.argv[2]).resolve())
assert (repo / 'scripts/rea.mjs').is_file()
def word(n):
    return struct.pack('<i', n)
def blob(b):
    return b's' + word(len(b)) + b
def text(s):
    b = s.encode('ascii')
    return b'z' + bytes([len(b)]) + b
def fixture(line):
    body = b'c' + struct.pack('<5i', 0, 0, 0, 1, 0)
    body += blob(b'\x53\x00') + b')\x01N' + b')\x00' + b')\x00' + blob(b'')
    body += text('sample.py') + text('<module>') + text('<module>')
    offset = 16 + len(body)
    data = b'\xcb\x0d\x0d\x0a' + bytes(12) + body + word(line) + blob(b'') + blob(b'')
    return data, offset
with tempfile.TemporaryDirectory(prefix='pyc-lines-') as directory:
    root = pathlib.Path(directory).resolve()
    env = {'PATH': str(pathlib.Path(node).parent) + ':/usr/bin:/bin', 'NO_COLOR': '1'}
    for key in ('HOME', 'TMPDIR', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME',
                'XDG_DATA_HOME', 'XDG_STATE_HOME', 'DSH_HOME', 'DSH_AGENTS_HOME'):
        location = root / key.lower()
        location.mkdir()
        env[key] = str(location)
    files = []
    for name, line in [('control', 1), ('negative', -1)]:
        data, offset = fixture(line)
        files.append(data)
        code = marshal.loads(data[16:])
        assert isinstance(code, types.CodeType) and code.co_firstlineno == line
        assert code.co_flags == 0
        assert marshal.loads(marshal.dumps(code)).co_firstlineno == line
        path = root / (name + '.pyc')
        path.write_bytes(data)
        p = subprocess.run([node, str(repo / 'scripts/rea.mjs'), 'inspect-python-pyc',
                            str(path), '--format', 'json'], cwd=root, env=env,
                           capture_output=True, text=True, timeout=30)
        assert p.returncode == 0 and p.stderr == '', (p.returncode, p.stderr)
        evidence = json.loads(p.stdout)
        result = evidence['normalized_result']
        assert result['artifact']['sha256'] == hashlib.sha256(data).hexdigest()
        assert result['artifact']['bytes'] == len(data)
        print(json.dumps({'case': name, 'stored_hex': data[offset:offset+4].hex(),
                          'cpython_firstlineno': code.co_firstlineno,
                          'rea_firstlineno': result['code_objects'][0]['firstlineno'],
                          'exit': p.returncode, 'confidence': evidence['confidence'],
                          'coverage': result['coverage']['status']}))
    differences = [i for i, (a, b) in enumerate(zip(*files)) if a != b]
    assert differences == list(range(offset, offset + 4))
    print('differing_byte_offsets', differences)
PYTHON

The original independent probe and this inline block each reproduced the mismatch once, with successful controls: two negative cases and two control cases total. The inline block produced the exact output below.

Expected behavior

Preserve the decoded code object's signed first source line number: the negative fixture should report firstlineno: -1, and the positive control should remain 1. A normalized code-object field must not silently reinterpret an accepted signed source-line value as a large positive line number. This expectation concerns stored metadata, not whether a real source file can have a negative line number.

Impact and scope

P2: a bounded semantic scalar is wrong in successful observed Evidence. An analyst cannot recover the accepted code object's source-line value from this field without knowing about the reinterpretation. The fixtures are self-authored and differ only at the four first-line bytes; real-world prevalence is unknown. No security boundary crossing or input mutation was observed. The demonstrated external-inspection workaround is to read co_firstlineno with a matching CPython for these bounded fixtures. That requires an interpreter and does not correct REA's normalized result or replace its Python-free workflow.

REA version

Unreleased source revision 6dff37594c, using its prepared compiled runtime; package/CLI label 6.7.0. The published rea-agents-6.7.0 tag lacks this inspector/parser, so this is a source-build reproduction rather than a stable-release claim.

Runtime and platform

Node.js v24.18.0; npm 11.16.0; CPython 3.12.15 for fixture creation and independent marshal decoding only; macOS 27.0.1 arm64. No native analysis engine was needed, and REA did not invoke CPython.

Target and artifact context

Two 101-byte timestamp-based CPython 3.12 pycs, magic 3531, with zero header flags, mtime and source size; one module code object; zero code flags and argument counts; stack size 1; empty name/local/location/exception tables; one None constant; two bytecode bytes. The control stores 01000000 and the negative fixture stores ffffffff at offsets 87–90. All other bytes match.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

{"case": "control", "stored_hex": "01000000", "cpython_firstlineno": 1, "rea_firstlineno": 1, "exit": 0, "confidence": "observed", "coverage": "complete"}
{"case": "negative", "stored_hex": "ffffffff", "cpython_firstlineno": -1, "rea_firstlineno": 4294967295, "exit": 0, "confidence": "observed", "coverage": "complete"}
differing_byte_offsets [87, 88, 89, 90]

Expected source: docs/python-pyc.md:3–7,36–48 describes decoded code objects from marshal, and AGENTS.md requires preserving source metadata/values and explicit unknowns when normalization loses information. The installed matching CPython header identifies co_firstlineno as a signed int containing the first source line number; its release source is cpython/code.h. The actual load above establishes that the exact negative value is accepted. REA's internal CodeNode is described as a decoded code object, and its firstlineno is exposed as normalized metadata, not a separately documented raw word.
Failure source: the public CLI observations above, reached through scripts/rea.mjs; src/python/PythonPycParser.ts:460–481 reads the field with u32() and places it into the code-object result, while src/domain/python/pythonPyc.ts:140,160 validates firstlineno as unsigned. The prepared runtime contains the same read and schema. These are source-derived explanations of the observed scalar mismatch.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Restoration check, proposed and not yet run against a repair: repeat the exact negative and control cases. Preserve the accepted signed source-line fact -1 for the negative fixture, retain 1 for the control, and preserve artifact identity and complete bounded decoding. A malformed-flags rejection does not close this case: both fixtures have valid zero code flags and CPython accepts both.

Analysis or suspected cause

Source-derived: firstlineno is interpreted as an unsigned word before the normalized object is created, and its unsigned output schema accepts the reinterpreted positive number. This scalar is accepted metadata whose signed value needs preservation. No source change or restoration patch was tested.

Additional context

Open/closed issue and PR searches used firstlineno, co_firstlineno, line-number/Python terms, signed pyc, marshal, PythonPycParser and separate open/closed pyc queries. No matching firstlineno report was found. Related #6437 and its repair #6461 concern negative-zero float serialization; #6415 and #6424 concern code-object hashability. Both repair diffs leave firstlineno unchanged. Decode-limit recovery, Unicode previews and negative PYZ sizes also have separate outcomes. MCP, other CPython versions/layouts, naturally occurring negative-line artifacts and other platforms were not exercised.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The exact accepted negative-line fixture returns the wrong semantic integer in successful complete observed Evidence; the positive control passes. Reach is bounded to the measured synthetic input, and natural frequency is unknown.
Workaround status: available
Workaround basis: Installed matching CPython recovers -1 from the same bounded self-authored fixture without executing its bytecode, as shown inline. This is an external inspection option when that interpreter is available, not a correction of REA Evidence or a general substitute for its bounded parser.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6490 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The public `inspect-python-pyc` CLI exits 0 and reports `firstlineno: 4294967295` for a CPython 3.12 code object whose stored first-line bytes are `ffffffff`. Installed CPython accepts the identical marshal payload and exposes `co_firstlineno == -1`, including after a marshal round trip. REA labels the result `confidence: "observed"` with `coverage.status: "complete"`; none of its four standard limitations identifies the line-number reinterpretation. The same code object with first line 1 is reported as 1 by both tools. Code flags are zero in both fixtures. ## Reproduction Use the prepared source checkout at the revision below, with its compiled runtime and repository-pinned Node.js on PATH, plus an existing CPython 3.12 interpreter. Dependency installation and runtime building are prerequisites for an unprepared checkout; neither was performed during this verification. From the checkout root, run this complete block. It writes two small synthetic pycs and isolates REA's home/config/cache/data/state/temp and DSH directories in a disposable root, then removes that root. It only loads and serializes code objects as data; no target bytecode runs. ```sh python3.12 -I -B - "$PWD" "$(command -v node)" <<'PYTHON' import hashlib, json, marshal, pathlib, struct, subprocess, sys, tempfile, types assert sys.version_info[:2] == (3, 12) repo, node = pathlib.Path(sys.argv[1]).resolve(), str(pathlib.Path(sys.argv[2]).resolve()) assert (repo / 'scripts/rea.mjs').is_file() def word(n): return struct.pack('<i', n) def blob(b): return b's' + word(len(b)) + b def text(s): b = s.encode('ascii') return b'z' + bytes([len(b)]) + b def fixture(line): body = b'c' + struct.pack('<5i', 0, 0, 0, 1, 0) body += blob(b'\x53\x00') + b')\x01N' + b')\x00' + b')\x00' + blob(b'') body += text('sample.py') + text('<module>') + text('<module>') offset = 16 + len(body) data = b'\xcb\x0d\x0d\x0a' + bytes(12) + body + word(line) + blob(b'') + blob(b'') return data, offset with tempfile.TemporaryDirectory(prefix='pyc-lines-') as directory: root = pathlib.Path(directory).resolve() env = {'PATH': str(pathlib.Path(node).parent) + ':/usr/bin:/bin', 'NO_COLOR': '1'} for key in ('HOME', 'TMPDIR', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME', 'XDG_DATA_HOME', 'XDG_STATE_HOME', 'DSH_HOME', 'DSH_AGENTS_HOME'): location = root / key.lower() location.mkdir() env[key] = str(location) files = [] for name, line in [('control', 1), ('negative', -1)]: data, offset = fixture(line) files.append(data) code = marshal.loads(data[16:]) assert isinstance(code, types.CodeType) and code.co_firstlineno == line assert code.co_flags == 0 assert marshal.loads(marshal.dumps(code)).co_firstlineno == line path = root / (name + '.pyc') path.write_bytes(data) p = subprocess.run([node, str(repo / 'scripts/rea.mjs'), 'inspect-python-pyc', str(path), '--format', 'json'], cwd=root, env=env, capture_output=True, text=True, timeout=30) assert p.returncode == 0 and p.stderr == '', (p.returncode, p.stderr) evidence = json.loads(p.stdout) result = evidence['normalized_result'] assert result['artifact']['sha256'] == hashlib.sha256(data).hexdigest() assert result['artifact']['bytes'] == len(data) print(json.dumps({'case': name, 'stored_hex': data[offset:offset+4].hex(), 'cpython_firstlineno': code.co_firstlineno, 'rea_firstlineno': result['code_objects'][0]['firstlineno'], 'exit': p.returncode, 'confidence': evidence['confidence'], 'coverage': result['coverage']['status']})) differences = [i for i, (a, b) in enumerate(zip(*files)) if a != b] assert differences == list(range(offset, offset + 4)) print('differing_byte_offsets', differences) PYTHON ``` The original independent probe and this inline block each reproduced the mismatch once, with successful controls: two negative cases and two control cases total. The inline block produced the exact output below. ## Expected behavior Preserve the decoded code object's signed first source line number: the negative fixture should report `firstlineno: -1`, and the positive control should remain 1. A normalized code-object field must not silently reinterpret an accepted signed source-line value as a large positive line number. This expectation concerns stored metadata, not whether a real source file can have a negative line number. ## Impact and scope P2: a bounded semantic scalar is wrong in successful observed Evidence. An analyst cannot recover the accepted code object's source-line value from this field without knowing about the reinterpretation. The fixtures are self-authored and differ only at the four first-line bytes; real-world prevalence is unknown. No security boundary crossing or input mutation was observed. The demonstrated external-inspection workaround is to read `co_firstlineno` with a matching CPython for these bounded fixtures. That requires an interpreter and does not correct REA's normalized result or replace its Python-free workflow. ## REA version Unreleased source revision 6dff37594c0bc942c9748b34a7a028d223e88a6d, using its prepared compiled runtime; package/CLI label 6.7.0. The published rea-agents-6.7.0 tag lacks this inspector/parser, so this is a source-build reproduction rather than a stable-release claim. ## Runtime and platform Node.js v24.18.0; npm 11.16.0; CPython 3.12.15 for fixture creation and independent marshal decoding only; macOS 27.0.1 arm64. No native analysis engine was needed, and REA did not invoke CPython. ## Target and artifact context Two 101-byte timestamp-based CPython 3.12 pycs, magic 3531, with zero header flags, mtime and source size; one module code object; zero code flags and argument counts; stack size 1; empty name/local/location/exception tables; one None constant; two bytecode bytes. The control stores `01000000` and the negative fixture stores `ffffffff` at offsets 87–90. All other bytes match. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence ```text {"case": "control", "stored_hex": "01000000", "cpython_firstlineno": 1, "rea_firstlineno": 1, "exit": 0, "confidence": "observed", "coverage": "complete"} {"case": "negative", "stored_hex": "ffffffff", "cpython_firstlineno": -1, "rea_firstlineno": 4294967295, "exit": 0, "confidence": "observed", "coverage": "complete"} differing_byte_offsets [87, 88, 89, 90] ``` - Expected source: `docs/python-pyc.md:3–7,36–48` describes decoded code objects from marshal, and `AGENTS.md` requires preserving source metadata/values and explicit unknowns when normalization loses information. The installed matching CPython header identifies `co_firstlineno` as a signed int containing the first source line number; its release source is [cpython/code.h](https://github.com/python/cpython/blob/v3.12.15/Include/cpython/code.h#L144). The actual load above establishes that the exact negative value is accepted. REA's internal CodeNode is described as a decoded code object, and its firstlineno is exposed as normalized metadata, not a separately documented raw word. - Failure source: the public CLI observations above, reached through `scripts/rea.mjs`; `src/python/PythonPycParser.ts:460–481` reads the field with `u32()` and places it into the code-object result, while `src/domain/python/pythonPyc.ts:140,160` validates firstlineno as unsigned. The prepared runtime contains the same read and schema. These are source-derived explanations of the observed scalar mismatch. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Restoration check, proposed and not yet run against a repair: repeat the exact negative and control cases. Preserve the accepted signed source-line fact -1 for the negative fixture, retain 1 for the control, and preserve artifact identity and complete bounded decoding. A malformed-flags rejection does not close this case: both fixtures have valid zero code flags and CPython accepts both. ## Analysis or suspected cause Source-derived: firstlineno is interpreted as an unsigned word before the normalized object is created, and its unsigned output schema accepts the reinterpreted positive number. This scalar is accepted metadata whose signed value needs preservation. No source change or restoration patch was tested. ## Additional context Open/closed issue and PR searches used firstlineno, co_firstlineno, line-number/Python terms, signed pyc, marshal, PythonPycParser and separate open/closed pyc queries. No matching firstlineno report was found. Related [#6437](https://github.com/morluto/rea/issues/6437) and its repair [#6461](https://github.com/morluto/rea/pull/6461) concern negative-zero float serialization; [#6415](https://github.com/morluto/rea/issues/6415) and [#6424](https://github.com/morluto/rea/pull/6424) concern code-object hashability. Both repair diffs leave firstlineno unchanged. Decode-limit recovery, Unicode previews and negative PYZ sizes also have separate outcomes. MCP, other CPython versions/layouts, naturally occurring negative-line artifacts and other platforms were not exercised. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The exact accepted negative-line fixture returns the wrong semantic integer in successful complete observed Evidence; the positive control passes. Reach is bounded to the measured synthetic input, and natural frequency is unknown. - Workaround status: available - Workaround basis: Installed matching CPython recovers -1 from the same bounded self-authored fixture without executing its bytecode, as shown inline. This is an external inspection option when that interpreter is available, not a correction of REA Evidence or a general substitute for its bounded parser.

bsakaguchi closed this issue

2026-10-31 00:45:01 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:08 +00:00

[PR #4038] fix(cli): preserve command status after complete pipe reads #6009

bsakaguchi referenced this issue

2026-11-01 21:10:53 +00:00

[GH-ISSUE #3582] [Tracker] Ready to close or merge: issues and pull requests (October 2026) #1330