morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6487] [Bug] reject constructor-invalid flags before reporting complete pyc coverage #3889

New issue

Closed

opened 2026-10-31 00:44:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:44:58 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6487

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-python-pyc exits 0 with confidence: "observed", coverage.status: "complete", and code-object flags: 2147483648 for a CPython 3.12 pyc whose flags word has its sign bit set. Matching CPython rejects the same marshal body with SystemError: Objects/codeobject.c:356: bad argument to internal function. Calling code.replace(co_flags=-2147483648) independently raises ValueError: co_flags must be a positive integer. The zero-flags control and an unknown positive flag bit, 0x40000000, both load in CPython and succeed in REA. A second negative-flags encoding, 0x80000001, also produces the divergence.

Reproduction

From the source checkout with its prepared compiled runtime and locked dependencies, put the repository-pinned Node.js toolchain and an existing CPython 3.12 interpreter on PATH and run this block. No installation or build is part of the probe. It compiles an inert pass statement into a code object, serializes it, changes only the flags word, and calls the supported CLI with a clean environment and isolated home/config/cache/data/state/temp directories. Compilation, marshal decoding and code-object replacement do not execute the target bytecode. The script asserts the control outcomes and the current divergent results and retains the complete CLI responses in the chosen work directory.

WORK="$(mktemp -d)"
python3.12 -I -S - "$PWD" "$WORK" <<'PYTHON'
import hashlib, importlib.util, json, marshal, os, pathlib, platform, struct, subprocess, sys
assert sys.implementation.name == 'cpython' and sys.version_info[:2] == (3, 12)
repo, work = (pathlib.Path(p).resolve() for p in sys.argv[1:3])
work.mkdir(parents=True, exist_ok=True)
env = {'PATH': os.environ['PATH'], 'REA_LOG_LEVEL': 'silent', 'NODE_DISABLE_COMPILE_CACHE': '1'}
for key in ['HOME', 'DSH_HOME', 'DSH_AGENTS_HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME', 'TMPDIR']:
    directory = work / key.lower()
    directory.mkdir(exist_ok=True)
    env[key] = str(directory)
# Compilation constructs inert code; neither exec nor eval nor import of the target is used.
original = compile('pass\n', 'sample.py', 'exec', dont_inherit=True)
body = marshal.dumps(original)
assert body[0] & 0x7f == ord('c')
assert struct.unpack_from('<5i', body, 1) == (0, 0, 0, original.co_stacksize, 0)
header = importlib.util.MAGIC_NUMBER + bytes(12)
assert int.from_bytes(header[:2], 'little') == 3531
print(json.dumps({'runtime': sys.version, 'platform': platform.platform(), 'node': subprocess.check_output(['node','--version'],env=env,text=True).strip(), 'magic': int.from_bytes(header[:2],'little')}))
results=[]
for name, bits in [('zero',0), ('unknown-positive',0x40000000), ('sign-bit',0x80000000), ('sign-bit-plus-one',0x80000001)]:
    changed = bytearray(body)
    struct.pack_into('<I', changed, 17, bits)
    data = header + changed
    differences = [i for i,(a,b) in enumerate(zip(header + body, data)) if a != b]
    selected = work / (name + '.pyc')
    selected.write_bytes(data)
    try:
        decoded = marshal.loads(changed)
        oracle={'outcome':'loads', 'co_flags':decoded.co_flags}
    except Exception as error:
        oracle={'outcome':type(error).__name__, 'message':str(error)}
    try:
        replaced = original.replace(co_flags=struct.unpack('<i',struct.pack('<I',bits))[0])
        constructor={'outcome':'constructs','co_flags':replaced.co_flags}
    except Exception as error:
        constructor={'outcome':type(error).__name__, 'message':str(error)}
    run = subprocess.run(['node',str(repo/'scripts/rea.mjs'),'inspect-python-pyc',str(selected),'--json'],cwd=work,env=env,text=True,capture_output=True,timeout=30)
    (work/(name+'.stdout.json')).write_text(run.stdout)
    (work/(name+'.stderr')).write_text(run.stderr)
    value=json.loads(run.stdout); report=value.get('normalized_result')
    observed={'case':name,'encoded_bits':bits,'different_file_offsets':differences,'bytes':len(data),'sha256':hashlib.sha256(data).hexdigest(),'cpython':oracle,'constructor':constructor,'cli_exit':run.returncode,'confidence':value.get('confidence'),'coverage':report['coverage']['status'] if report else None,'reported_flags':report['code_objects'][0]['flags'] if report else None}
    if bits < 0x80000000:
        assert oracle['outcome']=='loads' and run.returncode==0 and observed['reported_flags']==bits
    else:
        assert oracle['outcome']=='SystemError' and constructor['outcome']=='ValueError'
        assert run.returncode==0 and observed['coverage']=='complete' and observed['reported_flags']==bits
    results.append(observed)
    print(json.dumps(observed))
(work/'results.json').write_text(json.dumps(results,indent=2))
PYTHON

The zero and sign-bit-only files each contain 109 bytes and differ at exactly one file byte, offset 36. The sign-bit-only file has SHA-256 2735855ef4b65db981c54f720b1135281889630dd80cf6f3b27d8b7fe23c25f6. This is a modified compiler-produced code object, rather than an arbitrary hand-assembled code stream. A separate negative-argument-count control on the same zero-flags body returned the existing typed malformed-input outcome, confirming that this workflow already distinguishes such input from successful complete coverage.

Expected behavior

Constructor-invalid flags in a recognized CPython code object should receive the existing malformed-input refusal with no successful normalized result. Preserve success for zero flags and the accepted unknown positive bit. The expected behavior is rejection of a malformed code object, rather than limiting flags to a list of named bits or merely reformatting the negative value in an otherwise successful report.

REA already documents rejection of signed-negative code counts, malformed code-unit lengths, and argument counts that cannot fit CPython's checked local table. Its regression test explicitly requires rejecting code CPython treats as corrupt instead of reporting complete coverage. The original merged feature PR accepted repairs for precisely those constructor-invalid cases. Those contract and repair signals support applying the existing malformed-code handling here; CPython rejection alone is not the sole basis for the expectation.

Impact and scope

P2: a bounded malformed input receives complete observed Evidence, so a caller inspecting a damaged or edited pyc cannot distinguish this constructor-invalid code object from a supported successful observation. The observed defect concerns the code-object flags word, not PEP 552 header flags. There was no target execution, input mutation, process crash, data loss, resource-exhaustion result or demonstrated security-boundary crossing. Real-world prevalence was not measured.

Measured scope is the root code object of a CPython 3.12 file on the stated host. Other interpreter versions, nested negative-flags objects, MCP and other platforms were not exercised. For this root-code layout, reading the first code object's signed flags word at marshal offset 17 detects the malformed value without decoding or executing bytecode; the reproduction performs this read before code.replace. That is a bounded workaround, not a general validator for nested code objects.

REA version

Source commit 6dff37594c; package version 6.7.0; prepared compiled CLI runtime, without a rebuild or published-package run.

Runtime and platform

Node.js v24.18.0; npm 11.16.0; CPython 3.12.15; macOS 27.0.1 build 26A434; arm64. CPython is only the fixture generator and independent comparison runtime; REA does not invoke it. No external native analysis engine was used.

Target and artifact context

Synthetic timestamp-header CPython 3.12 pycs with magic 3531, zero header flags, timestamp and source size, and a single compiler-produced module code object for pass. The mutation changes its signed flags word; the underlying bytecode and other fields remain those of the control. The sign-bit-only word is 0x80000000, interpreted by CPython as -2147483648.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed result projections from the executed independent probe:

flags 0: CPython loads 0; code.replace constructs 0; CLI exit 0; confidence observed; coverage complete; reported flags 0.
flags 0x40000000: CPython loads 1073741824; code.replace constructs 1073741824; CLI exit 0; confidence observed; coverage complete; reported flags 1073741824.
flags 0x80000000: CPython SystemError: Objects/codeobject.c:356: bad argument to internal function; code.replace ValueError: co_flags must be a positive integer; CLI exit 0; confidence observed; coverage complete; reported flags 2147483648.
flags 0x80000001: CPython SystemError: Objects/codeobject.c:356: bad argument to internal function; code.replace ValueError: co_flags must be a positive integer; CLI exit 0; confidence observed; coverage complete; reported flags 2147483649.

The separate negative-argument-count control returned exit 1, code: "invalid_request", category: "invalid_input", details.operation: "inspect_python_pyc", and an issue with path: ["path"], reason: "invalid_format", and a message ending co_argcount is negative. It returned no normalized_result.

Expected source: docs/python-pyc.md, Limits and failures; src/python/PythonPycParser.test.ts:1272, “rejects code CPython treats as corrupt instead of complete coverage”; accepted review correction and repair acknowledgement in the original feature PR. The matching existing CPython runtime rejects the exact sign-bit body and negative constructor argument as shown above.
Failure source: the real scripts/rea.mjs inspect-python-pyc results above; src/python/PythonPycParser.ts:402 reads code flags with u32(), while src/domain/python/pythonPyc.ts permits unsigned flags and src/application/python/PythonPycService.ts wraps the accepted report as observed Evidence. The prepared compiled parser contains the same unsigned read.
Evidence provenance: observed; contract and call-path interpretation are source-derived.
Local verification: reproduced
Reproduction completeness: complete

Restoration check

For the sign-bit fixtures, replace the current successful complete Evidence with the existing malformed-input result: exit 1, invalid_request, an invalid_format issue at path, and no successful normalized_result. Both zero flags and the CPython-accepted unknown positive bit must continue to return their original values with complete coverage. The negative-count control should keep its existing refusal. No restoration patch was applied or tested.

Typed outcome or recovery expected

Return the existing invalid-input/invalid-format outcome for the recognized malformed body. Unknown positive flags accepted by the matching constructor must remain inspectable; the failure is the invalid signed value, not the absence of a named flag interpretation.

Analysis or suspected cause

Source-derived: counts use a signed validation path, but code flags use the unsigned reader and unsigned output schema, allowing the sign bit to become a large positive result. No documented intentional exception for constructor-invalid flags was found in the owning guide or feature PR review history. The unsigned representation by itself does not justify successful complete coverage for this malformed code object. The report leaves implementation and version-specific handling to the maintainer.

Additional context

Open and closed issue searches covered pyc, co_flags, negative code, marshal, corrupt, flags pyc and code-object terms, with explicit open/closed pyc searches. The nearest report, code-object hashability, and its repair PR concern insertion into sets/dictionaries; the repair diff retains this flags read and would not close this fixture. Reports #6437/#6461 concern signed-zero serialization, #6438 concerns budget recovery, #6476 concerns Unicode previews, #6120/#6150 concern another PYZ reader, #2296 concerns decompilation, and closed #6262 tracks feature-PR integration. No duplicate for constructor-invalid code flags was found. This report does not claim that every CPython execution-level invariant is part of static inspection.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The supported CLI returns complete observed Evidence for a bounded recognized pyc whose root code object the matching constructor refuses. This is consequential format-validation behavior limited to malformed inputs, without observed severe harm or security impact.
Workaround status: available
Workaround basis: For the measured root-code 3.12 layout, checking the signed flags word at marshal offset 17 detects the invalid value without loading or executing the bytecode; the probe performs that exact read. This does not validate arbitrary nested code objects or replace REA's general parser.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6487 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-python-pyc` exits 0 with `confidence: "observed"`, `coverage.status: "complete"`, and code-object `flags: 2147483648` for a CPython 3.12 pyc whose flags word has its sign bit set. Matching CPython rejects the same marshal body with `SystemError: Objects/codeobject.c:356: bad argument to internal function`. Calling `code.replace(co_flags=-2147483648)` independently raises `ValueError: co_flags must be a positive integer`. The zero-flags control and an unknown positive flag bit, `0x40000000`, both load in CPython and succeed in REA. A second negative-flags encoding, `0x80000001`, also produces the divergence. ## Reproduction From the source checkout with its prepared compiled runtime and locked dependencies, put the repository-pinned Node.js toolchain and an existing CPython 3.12 interpreter on PATH and run this block. No installation or build is part of the probe. It compiles an inert `pass` statement into a code object, serializes it, changes only the flags word, and calls the supported CLI with a clean environment and isolated home/config/cache/data/state/temp directories. Compilation, marshal decoding and code-object replacement do not execute the target bytecode. The script asserts the control outcomes and the current divergent results and retains the complete CLI responses in the chosen work directory. ```sh WORK="$(mktemp -d)" python3.12 -I -S - "$PWD" "$WORK" <<'PYTHON' import hashlib, importlib.util, json, marshal, os, pathlib, platform, struct, subprocess, sys assert sys.implementation.name == 'cpython' and sys.version_info[:2] == (3, 12) repo, work = (pathlib.Path(p).resolve() for p in sys.argv[1:3]) work.mkdir(parents=True, exist_ok=True) env = {'PATH': os.environ['PATH'], 'REA_LOG_LEVEL': 'silent', 'NODE_DISABLE_COMPILE_CACHE': '1'} for key in ['HOME', 'DSH_HOME', 'DSH_AGENTS_HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME', 'TMPDIR']: directory = work / key.lower() directory.mkdir(exist_ok=True) env[key] = str(directory) # Compilation constructs inert code; neither exec nor eval nor import of the target is used. original = compile('pass\n', 'sample.py', 'exec', dont_inherit=True) body = marshal.dumps(original) assert body[0] & 0x7f == ord('c') assert struct.unpack_from('<5i', body, 1) == (0, 0, 0, original.co_stacksize, 0) header = importlib.util.MAGIC_NUMBER + bytes(12) assert int.from_bytes(header[:2], 'little') == 3531 print(json.dumps({'runtime': sys.version, 'platform': platform.platform(), 'node': subprocess.check_output(['node','--version'],env=env,text=True).strip(), 'magic': int.from_bytes(header[:2],'little')})) results=[] for name, bits in [('zero',0), ('unknown-positive',0x40000000), ('sign-bit',0x80000000), ('sign-bit-plus-one',0x80000001)]: changed = bytearray(body) struct.pack_into('<I', changed, 17, bits) data = header + changed differences = [i for i,(a,b) in enumerate(zip(header + body, data)) if a != b] selected = work / (name + '.pyc') selected.write_bytes(data) try: decoded = marshal.loads(changed) oracle={'outcome':'loads', 'co_flags':decoded.co_flags} except Exception as error: oracle={'outcome':type(error).__name__, 'message':str(error)} try: replaced = original.replace(co_flags=struct.unpack('<i',struct.pack('<I',bits))[0]) constructor={'outcome':'constructs','co_flags':replaced.co_flags} except Exception as error: constructor={'outcome':type(error).__name__, 'message':str(error)} run = subprocess.run(['node',str(repo/'scripts/rea.mjs'),'inspect-python-pyc',str(selected),'--json'],cwd=work,env=env,text=True,capture_output=True,timeout=30) (work/(name+'.stdout.json')).write_text(run.stdout) (work/(name+'.stderr')).write_text(run.stderr) value=json.loads(run.stdout); report=value.get('normalized_result') observed={'case':name,'encoded_bits':bits,'different_file_offsets':differences,'bytes':len(data),'sha256':hashlib.sha256(data).hexdigest(),'cpython':oracle,'constructor':constructor,'cli_exit':run.returncode,'confidence':value.get('confidence'),'coverage':report['coverage']['status'] if report else None,'reported_flags':report['code_objects'][0]['flags'] if report else None} if bits < 0x80000000: assert oracle['outcome']=='loads' and run.returncode==0 and observed['reported_flags']==bits else: assert oracle['outcome']=='SystemError' and constructor['outcome']=='ValueError' assert run.returncode==0 and observed['coverage']=='complete' and observed['reported_flags']==bits results.append(observed) print(json.dumps(observed)) (work/'results.json').write_text(json.dumps(results,indent=2)) PYTHON ``` The zero and sign-bit-only files each contain 109 bytes and differ at exactly one file byte, offset 36. The sign-bit-only file has SHA-256 `2735855ef4b65db981c54f720b1135281889630dd80cf6f3b27d8b7fe23c25f6`. This is a modified compiler-produced code object, rather than an arbitrary hand-assembled code stream. A separate negative-argument-count control on the same zero-flags body returned the existing typed malformed-input outcome, confirming that this workflow already distinguishes such input from successful complete coverage. ## Expected behavior Constructor-invalid flags in a recognized CPython code object should receive the existing malformed-input refusal with no successful normalized result. Preserve success for zero flags and the accepted unknown positive bit. The expected behavior is rejection of a malformed code object, rather than limiting flags to a list of named bits or merely reformatting the negative value in an otherwise successful report. REA already documents rejection of signed-negative code counts, malformed code-unit lengths, and argument counts that cannot fit CPython's checked local table. Its regression test explicitly requires rejecting code CPython treats as corrupt instead of reporting complete coverage. The original merged feature PR accepted repairs for precisely those constructor-invalid cases. Those contract and repair signals support applying the existing malformed-code handling here; CPython rejection alone is not the sole basis for the expectation. ## Impact and scope P2: a bounded malformed input receives complete observed Evidence, so a caller inspecting a damaged or edited pyc cannot distinguish this constructor-invalid code object from a supported successful observation. The observed defect concerns the code-object flags word, not PEP 552 header flags. There was no target execution, input mutation, process crash, data loss, resource-exhaustion result or demonstrated security-boundary crossing. Real-world prevalence was not measured. Measured scope is the root code object of a CPython 3.12 file on the stated host. Other interpreter versions, nested negative-flags objects, MCP and other platforms were not exercised. For this root-code layout, reading the first code object's signed flags word at marshal offset 17 detects the malformed value without decoding or executing bytecode; the reproduction performs this read before `code.replace`. That is a bounded workaround, not a general validator for nested code objects. ## REA version Source commit 6dff37594c0bc942c9748b34a7a028d223e88a6d; package version 6.7.0; prepared compiled CLI runtime, without a rebuild or published-package run. ## Runtime and platform Node.js v24.18.0; npm 11.16.0; CPython 3.12.15; macOS 27.0.1 build 26A434; arm64. CPython is only the fixture generator and independent comparison runtime; REA does not invoke it. No external native analysis engine was used. ## Target and artifact context Synthetic timestamp-header CPython 3.12 pycs with magic 3531, zero header flags, timestamp and source size, and a single compiler-produced module code object for `pass`. The mutation changes its signed flags word; the underlying bytecode and other fields remain those of the control. The sign-bit-only word is `0x80000000`, interpreted by CPython as -2147483648. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Observed result projections from the executed independent probe: ```text flags 0: CPython loads 0; code.replace constructs 0; CLI exit 0; confidence observed; coverage complete; reported flags 0. flags 0x40000000: CPython loads 1073741824; code.replace constructs 1073741824; CLI exit 0; confidence observed; coverage complete; reported flags 1073741824. flags 0x80000000: CPython SystemError: Objects/codeobject.c:356: bad argument to internal function; code.replace ValueError: co_flags must be a positive integer; CLI exit 0; confidence observed; coverage complete; reported flags 2147483648. flags 0x80000001: CPython SystemError: Objects/codeobject.c:356: bad argument to internal function; code.replace ValueError: co_flags must be a positive integer; CLI exit 0; confidence observed; coverage complete; reported flags 2147483649. ``` The separate negative-argument-count control returned exit 1, `code: "invalid_request"`, `category: "invalid_input"`, `details.operation: "inspect_python_pyc"`, and an issue with `path: ["path"]`, `reason: "invalid_format"`, and a message ending `co_argcount is negative.` It returned no `normalized_result`. - Expected source: `docs/python-pyc.md`, Limits and failures; `src/python/PythonPycParser.test.ts:1272`, “rejects code CPython treats as corrupt instead of complete coverage”; accepted review correction and repair acknowledgement in the [original feature PR](https://github.com/morluto/rea/pull/4826#discussion_r4347116343). The matching existing CPython runtime rejects the exact sign-bit body and negative constructor argument as shown above. - Failure source: the real `scripts/rea.mjs inspect-python-pyc` results above; `src/python/PythonPycParser.ts:402` reads code flags with `u32()`, while `src/domain/python/pythonPyc.ts` permits unsigned flags and `src/application/python/PythonPycService.ts` wraps the accepted report as observed Evidence. The prepared compiled parser contains the same unsigned read. - Evidence provenance: observed; contract and call-path interpretation are source-derived. - Local verification: reproduced - Reproduction completeness: complete ### Restoration check For the sign-bit fixtures, replace the current successful complete Evidence with the existing malformed-input result: exit 1, `invalid_request`, an `invalid_format` issue at `path`, and no successful `normalized_result`. Both zero flags and the CPython-accepted unknown positive bit must continue to return their original values with complete coverage. The negative-count control should keep its existing refusal. No restoration patch was applied or tested. ## Typed outcome or recovery expected Return the existing invalid-input/invalid-format outcome for the recognized malformed body. Unknown positive flags accepted by the matching constructor must remain inspectable; the failure is the invalid signed value, not the absence of a named flag interpretation. ## Analysis or suspected cause Source-derived: counts use a signed validation path, but code flags use the unsigned reader and unsigned output schema, allowing the sign bit to become a large positive result. No documented intentional exception for constructor-invalid flags was found in the owning guide or feature PR review history. The unsigned representation by itself does not justify successful complete coverage for this malformed code object. The report leaves implementation and version-specific handling to the maintainer. ## Additional context Open and closed issue searches covered pyc, co_flags, negative code, marshal, corrupt, flags pyc and code-object terms, with explicit open/closed pyc searches. The nearest report, [code-object hashability](https://github.com/morluto/rea/issues/6415), and its [repair PR](https://github.com/morluto/rea/pull/6424) concern insertion into sets/dictionaries; the repair diff retains this flags read and would not close this fixture. Reports #6437/#6461 concern signed-zero serialization, #6438 concerns budget recovery, #6476 concerns Unicode previews, #6120/#6150 concern another PYZ reader, #2296 concerns decompilation, and closed #6262 tracks feature-PR integration. No duplicate for constructor-invalid code flags was found. This report does not claim that every CPython execution-level invariant is part of static inspection. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The supported CLI returns complete observed Evidence for a bounded recognized pyc whose root code object the matching constructor refuses. This is consequential format-validation behavior limited to malformed inputs, without observed severe harm or security impact. - Workaround status: available - Workaround basis: For the measured root-code 3.12 layout, checking the signed flags word at marshal offset 17 detects the invalid value without loading or executing the bytecode; the probe performs that exact read. This does not validate arbitrary nested code objects or replace REA's general parser.

bsakaguchi closed this issue

2026-10-31 00:44:59 +00:00

bsakaguchi referenced this issue

2026-10-31 00:47:35 +00:00

[GH-ISSUE #6854] [Bug] Obligation ledger verifies a changed owner using proof for the previous implementation #4193

bsakaguchi referenced this issue

2026-10-31 01:12:30 +00:00

[PR #4337] [CLOSED] fix(domain): fail closed on conflicting reconstruction proofs and evaluate required claim surfaces #6181