morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6120] [Bug] Negative PYZ marshal sizes abort sibling inventory as a resource limit #3753

New issue

Closed

opened 2026-10-31 00:43:40 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:43:40 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6120

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A tiny PyInstaller executable containing one healthy data.txt and a PYZ directory with a negative marshal string length, list count or tuple count fails the entire public inspect-artifact call with code: "truncated" and details.reason: "limit". No inventory is delivered. The message recommends a smaller artifact.

Changing only the PYZ table to a truncated string or unsupported opcode instead succeeds, preserves data.txt, and leaves that PYZ opaque. A valid empty table also succeeds. These were actual built CLI calls with generated bytes, not direct parser calls.

Reproduction

From the built source checkout with locked dependencies, pipe this script into node --import tsx --input-type=module. It generates six synthetic ELF/CArchive fixtures through the repository fixture writer, executes the public CLI sequentially, checks observed outputs, and removes disposable files. No recovered code is executed.

import assert from 'node:assert/strict';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { spawnSync } from 'node:child_process';
const {cArchiveExecutable,pyzArchive,PYTHON_39_MAGIC} = await import(new URL('./src/artifacts/pyinstaller/CArchive.fixture.ts', `file://${process.cwd()}/`));
const root=mkdtempSync(join(tmpdir(),'rea-marshal-repro-'));
const env={...process.env,REA_LOG_LEVEL:'silent'};
for(const [key,name] of [['HOME','home'],['TMPDIR','temp'],['XDG_CONFIG_HOME','config'],['XDG_DATA_HOME','data'],['XDG_CACHE_HOME','cache'],['XDG_STATE_HOME','state']]){const path=join(root,name);mkdirSync(path);env[key]=path;}
try {
  for(const [label,hex] of [['negative-string','73ffffffff'],['negative-list','5bffffffff'],['negative-tuple','28ffffffff'],['truncated-string','7301000000'],['unsupported','30'],['valid','5b00000000']]){
    const path=join(root,label);
    writeFileSync(path,cArchiveExecutable({layout:'2.1',pythonVersion:309,members:[
      {name:'data.txt',typeCode:'x',payload:Buffer.from('healthy'),compress:false},
      {name:'PYZ.pyz',typeCode:'z',payload:pyzArchive({magic:PYTHON_39_MAGIC,modules:[],toc:Buffer.from(hex,'hex')}),compress:false},
    ]}));
    const result=spawnSync(process.execPath,['scripts/rea.mjs','inspect-artifact',path,'--json'],{env,encoding:'utf8'});
    const output=JSON.parse(result.stdout);
    assert.equal(result.status,label.startsWith('negative')?1:0,result.stderr);assert.equal(result.stderr,'');
    if(result.status){assert.equal(output.code,'truncated');console.log(JSON.stringify({label,exit:result.status,code:output.code,details:output.details}));}
    else {
      const inventory=output.normalized_result.substeps.find(s=>s.operation==='inventory_artifact').evidence.normalized_result;
      const healthy=inventory.occurrences.find(o=>o.logical_path==='data.txt');
      const pyz=inventory.occurrences.find(o=>o.logical_path==='PYZ.pyz');
      assert(healthy);assert.equal(pyz.format_metadata.pyz_status,label==='valid'?'decoded':'opaque');
      console.log(JSON.stringify({label,exit:result.status,healthySiblingPresent:!!healthy,pyzStatus:pyz.format_metadata.pyz_status}));
    }
  }
}finally{rmSync(root,{recursive:true,force:true});}

Expected behavior

A negative length/count is malformed structure, so preserve the parser's format distinction and use the existing opaque-PYZ recovery. Keep healthy sibling inventory and mark unknown module coverage. Positive values exceeding real resource bounds should still fail safely as limits.

The failure type separates format from limit. Existing PYZ recovery leaves malformed marshal directories opaque. The documented behavior distinguishes resource budgets from opaque unsupported directories.

Impact and scope

P2: one damaged PYZ table prevents the caller from retrieving otherwise healthy sibling inventory and gives size-oriented recovery for malformed bytes. The three negative-size cases deterministically fail; the two malformed/unsupported controls preserve sibling inventory. Normal valid inputs are not claimed to fail, no input is modified, and no code execution or security-boundary failure is claimed.

Observed scope is stored PYZ images with supported Python magic in a synthetic CArchive. Real-world prevalence, extraction, MCP delivery and other operating systems were not tested. A separate archive reader or repairing/repacking the damaged member may be possible, but no practical alternate recovery was demonstrated.

REA version

rea-agents 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e; existing built CLI.

Runtime and platform

macOS 27.0.1 arm64; repository-pinned Node.js v24.18.0; locked tsx for fixture generation. No Python interpreter or external analysis engine ran.

Target and artifact context

Synthetic ELF with appended PyInstaller 2.1 CArchive, Python version field 309, healthy type-x data.txt containing healthy, and one uncompressed type-z PYZ with magic 610d0d0a. Table bytes are varied independently.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Executed six-case observations:

{"label":"negative-string","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal string exceeds the supported length"}}
{"label":"negative-list","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal sequence exceeds the supported count"}}
{"label":"negative-tuple","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal sequence exceeds the supported count"}}
{"label":"truncated-string","exit":0,"healthySiblingPresent":true,"pyzStatus":"opaque"}
{"label":"unsupported","exit":0,"healthySiblingPresent":true,"pyzStatus":"opaque"}
{"label":"valid","exit":0,"healthySiblingPresent":true,"pyzStatus":"decoded"}

All stderr streams are empty. The negative cases also return this message and identical remediation action:

Artifact is too large to process safely. Narrow the requested path or use a smaller artifact.

Typed outcome or recovery expected

Treat impossible signed sizes as malformed PYZ directory values and retain an opaque member plus unknown child coverage. Do not convert these into an artifact-wide size refusal. This does not request relaxing any upper resource limit or treating unknown module contents as absent.

Analysis or suspected cause

Source-derived: readSequence combines negative counts with counts above MAX_SEQUENCE; readBytes combines negative lengths with lengths above MAX_STRING_BYTES. Both return limit. PyzArchive only recovers format as opaque; the reader throws propagated errors. Public projection consequently maps the wrong reason to truncated. The producing parser classification, rather than generic size wording alone, explains lost sibling inventory.

Additional context

The affected parser and reader are unchanged in fetched upstream main aa51e4343963fdd0b97ec2ebdc3c41579fd9cb38. Upstream was source-inspected, not executed. #5836 concerns a genuine 128-file private-storage capacity allowance losing its typed boundary and recovery; correcting that presentation does not restore opaque recovery for malformed signed sizes. #6067 concerns a changed-file admission mapping to io. Original archive PR #2433 and standalone pyc-inspector PR #4826 do not address this parser classification. No restoration patch was tested.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6120 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A tiny PyInstaller executable containing one healthy `data.txt` and a PYZ directory with a negative marshal string length, list count or tuple count fails the entire public `inspect-artifact` call with `code: "truncated"` and `details.reason: "limit"`. No inventory is delivered. The message recommends a smaller artifact. Changing only the PYZ table to a truncated string or unsupported opcode instead succeeds, preserves `data.txt`, and leaves that PYZ opaque. A valid empty table also succeeds. These were actual built CLI calls with generated bytes, not direct parser calls. ### Reproduction From the built source checkout with locked dependencies, pipe this script into `node --import tsx --input-type=module`. It generates six synthetic ELF/CArchive fixtures through the repository fixture writer, executes the public CLI sequentially, checks observed outputs, and removes disposable files. No recovered code is executed. ```javascript import assert from 'node:assert/strict'; import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { spawnSync } from 'node:child_process'; const {cArchiveExecutable,pyzArchive,PYTHON_39_MAGIC} = await import(new URL('./src/artifacts/pyinstaller/CArchive.fixture.ts', `file://${process.cwd()}/`)); const root=mkdtempSync(join(tmpdir(),'rea-marshal-repro-')); const env={...process.env,REA_LOG_LEVEL:'silent'}; for(const [key,name] of [['HOME','home'],['TMPDIR','temp'],['XDG_CONFIG_HOME','config'],['XDG_DATA_HOME','data'],['XDG_CACHE_HOME','cache'],['XDG_STATE_HOME','state']]){const path=join(root,name);mkdirSync(path);env[key]=path;} try { for(const [label,hex] of [['negative-string','73ffffffff'],['negative-list','5bffffffff'],['negative-tuple','28ffffffff'],['truncated-string','7301000000'],['unsupported','30'],['valid','5b00000000']]){ const path=join(root,label); writeFileSync(path,cArchiveExecutable({layout:'2.1',pythonVersion:309,members:[ {name:'data.txt',typeCode:'x',payload:Buffer.from('healthy'),compress:false}, {name:'PYZ.pyz',typeCode:'z',payload:pyzArchive({magic:PYTHON_39_MAGIC,modules:[],toc:Buffer.from(hex,'hex')}),compress:false}, ]})); const result=spawnSync(process.execPath,['scripts/rea.mjs','inspect-artifact',path,'--json'],{env,encoding:'utf8'}); const output=JSON.parse(result.stdout); assert.equal(result.status,label.startsWith('negative')?1:0,result.stderr);assert.equal(result.stderr,''); if(result.status){assert.equal(output.code,'truncated');console.log(JSON.stringify({label,exit:result.status,code:output.code,details:output.details}));} else { const inventory=output.normalized_result.substeps.find(s=>s.operation==='inventory_artifact').evidence.normalized_result; const healthy=inventory.occurrences.find(o=>o.logical_path==='data.txt'); const pyz=inventory.occurrences.find(o=>o.logical_path==='PYZ.pyz'); assert(healthy);assert.equal(pyz.format_metadata.pyz_status,label==='valid'?'decoded':'opaque'); console.log(JSON.stringify({label,exit:result.status,healthySiblingPresent:!!healthy,pyzStatus:pyz.format_metadata.pyz_status})); } } }finally{rmSync(root,{recursive:true,force:true});} ``` ### Expected behavior A negative length/count is malformed structure, so preserve the parser's `format` distinction and use the existing opaque-PYZ recovery. Keep healthy sibling inventory and mark unknown module coverage. Positive values exceeding real resource bounds should still fail safely as limits. The [failure type](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/pyinstaller/PyInstallerFailure.ts#L1-L10) separates format from limit. [Existing PYZ recovery](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/pyinstaller/PyzArchive.ts#L132-L150) leaves malformed marshal directories opaque. The [documented behavior](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/python-packaged-applications.md#L80-L103) distinguishes resource budgets from opaque unsupported directories. ### Impact and scope P2: one damaged PYZ table prevents the caller from retrieving otherwise healthy sibling inventory and gives size-oriented recovery for malformed bytes. The three negative-size cases deterministically fail; the two malformed/unsupported controls preserve sibling inventory. Normal valid inputs are not claimed to fail, no input is modified, and no code execution or security-boundary failure is claimed. Observed scope is stored PYZ images with supported Python magic in a synthetic CArchive. Real-world prevalence, extraction, MCP delivery and other operating systems were not tested. A separate archive reader or repairing/repacking the damaged member may be possible, but no practical alternate recovery was demonstrated. ### REA version `rea-agents` 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`; existing built CLI. ### Runtime and platform macOS 27.0.1 arm64; repository-pinned Node.js v24.18.0; locked tsx for fixture generation. No Python interpreter or external analysis engine ran. ### Target and artifact context Synthetic ELF with appended PyInstaller 2.1 CArchive, Python version field 309, healthy type-x data.txt containing healthy, and one uncompressed type-z PYZ with magic 610d0d0a. Table bytes are varied independently. ### Interface CLI ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence Executed six-case observations: ```json {"label":"negative-string","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal string exceeds the supported length"}} {"label":"negative-list","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal sequence exceeds the supported count"}} {"label":"negative-tuple","exit":1,"code":"truncated","details":{"operation":"inspect_artifact","reason":"limit","truncated":true,"detail":"PYZ marshal sequence exceeds the supported count"}} {"label":"truncated-string","exit":0,"healthySiblingPresent":true,"pyzStatus":"opaque"} {"label":"unsupported","exit":0,"healthySiblingPresent":true,"pyzStatus":"opaque"} {"label":"valid","exit":0,"healthySiblingPresent":true,"pyzStatus":"decoded"} ``` All stderr streams are empty. The negative cases also return this message and identical remediation action: ```text Artifact is too large to process safely. Narrow the requested path or use a smaller artifact. ``` ### Typed outcome or recovery expected Treat impossible signed sizes as malformed PYZ directory values and retain an opaque member plus unknown child coverage. Do not convert these into an artifact-wide size refusal. This does not request relaxing any upper resource limit or treating unknown module contents as absent. ### Analysis or suspected cause Source-derived: [readSequence](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/pyinstaller/MarshalSubset.ts#L100-L112) combines negative counts with counts above MAX_SEQUENCE; [readBytes](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/pyinstaller/MarshalSubset.ts#L235-L248) combines negative lengths with lengths above MAX_STRING_BYTES. Both return `limit`. [PyzArchive](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/pyinstaller/PyzArchive.ts#L132-L141) only recovers `format` as opaque; the [reader](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/PyInstallerArtifactReader.ts#L455-L461) throws propagated errors. [Public projection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisErrorProjection.ts#L196-L203) consequently maps the wrong reason to `truncated`. The producing parser classification, rather than generic size wording alone, explains lost sibling inventory. ### Additional context The affected parser and reader are unchanged in fetched upstream main `aa51e4343963fdd0b97ec2ebdc3c41579fd9cb38`. Upstream was source-inspected, not executed. #5836 concerns a genuine 128-file private-storage capacity allowance losing its typed boundary and recovery; correcting that presentation does not restore opaque recovery for malformed signed sizes. #6067 concerns a changed-file admission mapping to io. Original archive PR #2433 and standalone pyc-inspector PR #4826 do not address this parser classification. No restoration patch was tested.

bsakaguchi closed this issue

2026-10-31 00:43:40 +00:00

bsakaguchi commented

2026-10-31 00:43:41 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

确认，这跟 #6123 同类：把格式损坏误归因为资源限制，还建议用户缩小 artifact，方向完全错。更麻烦的是负 size 这一个成员能让整次 inventory 中止、连 healthy 的 data.txt 都交付不了，而其他损坏形式（截断字符串/不支持 opcode）反而能 opaque 化并保留 sibling——说明读 PYZ 时对"负长度"的处理没有走既有 recovery 路径。建议 #6150 修完带上两条验收：负 marshal size 归 input/format 缺陷而非 resource limit；负 size 的 PYZ 成员与其他损坏形式一样 opaque 化，不中止整个 inventory。

@XinwangTao commented on GitHub (Oct 22, 2026): 确认，这跟 #6123 同类：把格式损坏误归因为资源限制，还建议用户缩小 artifact，方向完全错。更麻烦的是负 size 这一个成员能让整次 inventory 中止、连 healthy 的 data.txt 都交付不了，而其他损坏形式（截断字符串/不支持 opcode）反而能 opaque 化并保留 sibling——说明读 PYZ 时对"负长度"的处理没有走既有 recovery 路径。建议 #6150 修完带上两条验收：负 marshal size 归 input/format 缺陷而非 resource limit；负 size 的 PYZ 成员与其他损坏形式一样 opaque 化，不中止整个 inventory。

bsakaguchi referenced this issue

2026-10-31 01:10:07 +00:00

[PR #3753] feat(jvm): export Vineflower sources with explicit dependencies #5876

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425