morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6510] [Bug] original DOS header padding triggers a false PE refusal #3906

New issue

Closed

opened 2026-10-31 00:45:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:45:07 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6510

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

A valid original DOS MZ layout with a 64-byte header and two relocation records starting at 0x1C is rejected by public inspect before provider selection when unused header bytes 0x3C–0x3F contain ABCD. REA returns target_unavailable with reason invalid or truncated PE header. The otherwise identical zero-padding control resolves as dos-mz / x86 and reaches provider availability handling. No target or native analysis engine was executed.

Reproduction

Run the following from a source checkout with its compiled runtime available and the pinned Node.js on PATH. The script uses a clean subprocess environment, disposable home/configuration directories, and deliberately absent provider installations. It prints only the admission result and artifact digest, then removes its fixtures. The fixture pair and malformed-header controls below were run locally; the original supplied variant with entry 0 and a ret byte reproduces the same distinction.

python3 - <<'PY'
from pathlib import Path
import hashlib, json, os, shutil, struct, subprocess, tempfile

repo = Path.cwd()
node = shutil.which('node')
assert node and (repo / 'dist/cli-runtime/cli.js').is_file()
work = Path(tempfile.mkdtemp(prefix='mz-padding-'))
home = work / 'home'
home.mkdir()
env = {
    'PATH': os.environ['PATH'], 'HOME': str(home),
    'XDG_CONFIG_HOME': str(home), 'XDG_CACHE_HOME': str(home),
    'XDG_DATA_HOME': str(home), 'TMPDIR': str(work),
    'DSH_HOME': str(home / 'dsh'), 'DSH_AGENTS_HOME': str(home / 'agents'),
    'GHIDRA_INSTALL_DIR': str(work / 'missing-ghidra'),
    'HOPPER_LAUNCHER_PATH': str(work / 'missing-hopper'),
    'REA_LOG_LEVEL': 'silent',
}

def mz(padding):
    b = bytearray(128)
    struct.pack_into('<14H', b, 0,
        0x5A4D, 128, 1, 2, 4, 64, 0xFFFF, 0, 64, 0, 8, 0, 28, 0)
    struct.pack_into('<HHHH', b, 28, 0, 0, 2, 0)
    struct.pack_into('<I', b, 60, padding)
    b[72:77] = bytes.fromhex('b8004ccd21')
    return b

cases = {'padding-nonzero': mz(0x44434241), 'padding-zero': mz(0)}
for name, signature, pointer in [
    ('truncated-pe', b'PE\0\0', 120),
    ('invalid-windows-signature', b'XX\0\0', 96),
    ('ne', b'NE', 96), ('le', b'LE', 96), ('lx', b'LX', 96),
    ('truncated-ne', b'NE', 126),
    ('truncated-le', b'LE', 126), ('truncated-lx', b'LX', 126),
]:
    b = mz(pointer)
    struct.pack_into('<H', b, 6, 0)
    struct.pack_into('<H', b, 24, 64)
    b[pointer:pointer + len(signature)] = signature
    cases[name] = b

try:
    for name, data in cases.items():
        path = work / (name + '.exe')
        path.write_bytes(data)
        run = subprocess.run([
            node, str(repo / 'scripts/rea.mjs'), 'inspect', str(path),
            '--provider', 'ghidra', '--format', 'json',
        ], cwd=repo, env=env, text=True, capture_output=True, timeout=45)
        result = json.loads(run.stdout)
        details = result.get('details', {})
        formats = sorted({r.get('diagnostics', {}).get('target_format')
            for r in details.get('rejections', [])
            if r.get('diagnostics', {}).get('target_format')})
        print(json.dumps({'fixture': name, 'exit': run.returncode,
            'code': result.get('code'), 'reason': details.get('reason'),
            'target_formats': formats,
            'sha256': hashlib.sha256(data).hexdigest()}))
finally:
    shutil.rmtree(work)
PY

Both padding commands exit 1: the nonzero case fails target admission, while the zero control reaches the deliberately unavailable providers. The discriminator is target_unavailable versus provider_unavailable, with dos-mz in the latter's diagnostics. The malformed Windows controls must continue to return target_unavailable, including after this DOS case is restored.

Expected behavior

Admit the unchanged original DOS fixture as dos-mz / x86, reaching provider selection. Its page counts declare 128 image bytes; the four-paragraph header occupies [0,64), its relocation table occupies [28,36), and its initialized module occupies [64,128). The two relocation destinations are module words at offsets 0 and 2, and entry 0:8 lies within the module. Header bytes [36,64) are padding in this original layout, so changing the last four padding bytes must not manufacture a Windows declaration.

Preserve the separate contract for actual Windows declarations: confirmed PE remains PE, NE/LE/LX remain unsupported, and damaged or truncated declared Windows headers remain rejected. Absence of a Windows signature alone is not a sufficient restoration rule.

Impact and scope

The observed consequence is a deterministic false target-admission refusal for this valid DOS layout, before a native provider can handle the artifact. Real-world frequency and successful Ghidra import or decompilation were not measured. The four-byte diagnostic edit preserves initialized-module and relocation semantics but changes the source artifact's digest; it does not restore admission of the original artifact. No complete workaround preserving original identity was verified.

REA version

Source revision b395190e4273b2542a599aa2cdce61107f3a7ebc; the dispatcher reports 6.7.0.

Runtime and platform

Node.js v24.18.0; npm 11.16.0; macOS 27.0.1 build 26A434; arm64. Provider installations were intentionally absent. The discovery probe reported no Java runtime. No Ghidra session or DOS execution was run.

Target and artifact context

Synthetic 128-byte DOS MZ pair with a 64-byte header, two relocation records at 0x1C, a 64-byte initialized module, CS:IP 0:8 and SS:SP 0:64. The initialized module contains two data words followed by an exit sequence; it was read only. The files differ solely at offsets 60–63, with no Windows header at the value encoded there.

Interface

CLI

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Expected source: docs/ghidra-dos.md:26-32 describes DOS structural admission, separation of DOS fields from Windows declarations, and malformed-Windows rejection. Microsoft's original DOS loader, especially the original header fields and exec_find_res / exec_do_reloc, reads the initialized module at the paragraph-sized header boundary and reads relocation records separately. The original loader source supports the original layout's padding semantics; file independently recognizes both images as MS-DOS executables.
Failure source: Executed public inspect and filesystem target resolution for the inline fixtures. src/domain/dosMz.ts:53-69 misses padding ownership when a relocation table ends before byte 60, then treats every nonzero dword at byte 60 as a Windows declaration for a header of at least 64 bytes. src/application/BinaryTargetResolver.ts:560-585 follows it into readPeMetadata; its fixed-header read at the out-of-file offset returns the reported error. Independent evaluation of the original source helper and compiled helper agreed on every case.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete
Restoration check: The unchanged nonzero-padding fixture must resolve as dos-mz / x86 and reach the same provider-selection stage as its zero-padding control. Keep the malformed Windows controls rejected, preserve existing confirmed-PE and NE/LE/LX behavior, and retain DOS page-count, relocation and entry-point validation. Existing src/domain/dosMz.test.ts:271-281 specifically requires damaged Windows declarations to remain rejected; restoring this DOS case must preserve that guarantee.

Observed output, restricted to fields printed by the reproduction:

padding-nonzero: exit=1, code=target_unavailable, reason=invalid or truncated PE header
padding-zero: exit=1, code=provider_unavailable, target_formats=[dos-mz]
truncated-pe, invalid-windows-signature: exit=1, code=target_unavailable, reason=invalid or truncated PE header
ne, le, lx: exit=1, code=target_unavailable, reason=unsupported NE/LE/LX executable in MZ image (respectively)
truncated-ne, truncated-le, truncated-lx: exit=1, code=target_unavailable, reason=invalid or truncated PE header

The nonzero-padding digest is ea6747b60d3922712358da7838529185975aa15a82bb36321a1ebc86768b5f6f; the zero-padding digest is 708b7dc9c79834c86289a17e29a8f2af25a3ec56ebf58263bc1e31939fa6afdc. The source helper reports declared, offset 1145258561, signature unobserved for the nonzero fixture; filesystem resolution of the zero fixture succeeds with dos-mz / x86. The focused existing source suites src/domain/dosMz.test.ts and src/domain/binaryTarget.test.ts passed all 67 tests with one Vitest worker; that establishes existing safeguards, not a tested fix.

Additional context

Searched both open and closed issues using MZ, e_lfanew, DOS header, header padding, invalid or truncated PE header, and relocation, and compared likely matches and linked repairs. #2172 restores real PE recognition after short DOS stubs; its closing repair preserves the full-header branch implicated here. #5168 and the current diff and comments of PR #5216 concern signature-confirmed PE at offsets below 64. The PR changes minimum offsets and consumers but leaves the original-DOS false declaration intact; this conclusion comes from its source diff, not a runtime test of the PR. The baseline reproduction has a distinct restoration predicate, and neither repair addresses it. Related high-bit signature and unsupported-format wording reports concern different contracts.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A valid original DOS layout receives a false pre-provider format refusal. The bounded fixtures directly establish admission failure and incorrect format interpretation; severe real-world consequences or an otherwise working provider workflow were not observed.
Workaround status: unverified
Workaround basis: Zeroing the padding is a verified admission bypass for a different artifact, with a different digest. It preserves the initialized module but not the original artifact's identity; no complete workaround for unchanged-artifact DOS admission was verified.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6510 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Native analysis or decompilation ## Observed behavior A valid original DOS MZ layout with a 64-byte header and two relocation records starting at 0x1C is rejected by public `inspect` before provider selection when unused header bytes 0x3C–0x3F contain `ABCD`. REA returns `target_unavailable` with reason `invalid or truncated PE header`. The otherwise identical zero-padding control resolves as `dos-mz` / `x86` and reaches provider availability handling. No target or native analysis engine was executed. ## Reproduction Run the following from a source checkout with its compiled runtime available and the pinned Node.js on PATH. The script uses a clean subprocess environment, disposable home/configuration directories, and deliberately absent provider installations. It prints only the admission result and artifact digest, then removes its fixtures. The fixture pair and malformed-header controls below were run locally; the original supplied variant with entry 0 and a `ret` byte reproduces the same distinction. ```sh python3 - <<'PY' from pathlib import Path import hashlib, json, os, shutil, struct, subprocess, tempfile repo = Path.cwd() node = shutil.which('node') assert node and (repo / 'dist/cli-runtime/cli.js').is_file() work = Path(tempfile.mkdtemp(prefix='mz-padding-')) home = work / 'home' home.mkdir() env = { 'PATH': os.environ['PATH'], 'HOME': str(home), 'XDG_CONFIG_HOME': str(home), 'XDG_CACHE_HOME': str(home), 'XDG_DATA_HOME': str(home), 'TMPDIR': str(work), 'DSH_HOME': str(home / 'dsh'), 'DSH_AGENTS_HOME': str(home / 'agents'), 'GHIDRA_INSTALL_DIR': str(work / 'missing-ghidra'), 'HOPPER_LAUNCHER_PATH': str(work / 'missing-hopper'), 'REA_LOG_LEVEL': 'silent', } def mz(padding): b = bytearray(128) struct.pack_into('<14H', b, 0, 0x5A4D, 128, 1, 2, 4, 64, 0xFFFF, 0, 64, 0, 8, 0, 28, 0) struct.pack_into('<HHHH', b, 28, 0, 0, 2, 0) struct.pack_into('<I', b, 60, padding) b[72:77] = bytes.fromhex('b8004ccd21') return b cases = {'padding-nonzero': mz(0x44434241), 'padding-zero': mz(0)} for name, signature, pointer in [ ('truncated-pe', b'PE\0\0', 120), ('invalid-windows-signature', b'XX\0\0', 96), ('ne', b'NE', 96), ('le', b'LE', 96), ('lx', b'LX', 96), ('truncated-ne', b'NE', 126), ('truncated-le', b'LE', 126), ('truncated-lx', b'LX', 126), ]: b = mz(pointer) struct.pack_into('<H', b, 6, 0) struct.pack_into('<H', b, 24, 64) b[pointer:pointer + len(signature)] = signature cases[name] = b try: for name, data in cases.items(): path = work / (name + '.exe') path.write_bytes(data) run = subprocess.run([ node, str(repo / 'scripts/rea.mjs'), 'inspect', str(path), '--provider', 'ghidra', '--format', 'json', ], cwd=repo, env=env, text=True, capture_output=True, timeout=45) result = json.loads(run.stdout) details = result.get('details', {}) formats = sorted({r.get('diagnostics', {}).get('target_format') for r in details.get('rejections', []) if r.get('diagnostics', {}).get('target_format')}) print(json.dumps({'fixture': name, 'exit': run.returncode, 'code': result.get('code'), 'reason': details.get('reason'), 'target_formats': formats, 'sha256': hashlib.sha256(data).hexdigest()})) finally: shutil.rmtree(work) PY ``` Both padding commands exit 1: the nonzero case fails target admission, while the zero control reaches the deliberately unavailable providers. The discriminator is `target_unavailable` versus `provider_unavailable`, with `dos-mz` in the latter's diagnostics. The malformed Windows controls must continue to return `target_unavailable`, including after this DOS case is restored. ## Expected behavior Admit the unchanged original DOS fixture as `dos-mz` / `x86`, reaching provider selection. Its page counts declare 128 image bytes; the four-paragraph header occupies [0,64), its relocation table occupies [28,36), and its initialized module occupies [64,128). The two relocation destinations are module words at offsets 0 and 2, and entry 0:8 lies within the module. Header bytes [36,64) are padding in this original layout, so changing the last four padding bytes must not manufacture a Windows declaration. Preserve the separate contract for actual Windows declarations: confirmed PE remains PE, NE/LE/LX remain unsupported, and damaged or truncated declared Windows headers remain rejected. Absence of a Windows signature alone is not a sufficient restoration rule. ## Impact and scope The observed consequence is a deterministic false target-admission refusal for this valid DOS layout, before a native provider can handle the artifact. Real-world frequency and successful Ghidra import or decompilation were not measured. The four-byte diagnostic edit preserves initialized-module and relocation semantics but changes the source artifact's digest; it does not restore admission of the original artifact. No complete workaround preserving original identity was verified. ## REA version Source revision `b395190e4273b2542a599aa2cdce61107f3a7ebc`; the dispatcher reports 6.7.0. ## Runtime and platform Node.js v24.18.0; npm 11.16.0; macOS 27.0.1 build 26A434; arm64. Provider installations were intentionally absent. The discovery probe reported no Java runtime. No Ghidra session or DOS execution was run. ## Target and artifact context Synthetic 128-byte DOS MZ pair with a 64-byte header, two relocation records at 0x1C, a 64-byte initialized module, CS:IP 0:8 and SS:SP 0:64. The initialized module contains two data words followed by an exit sequence; it was read only. The files differ solely at offsets 60–63, with no Windows header at the value encoded there. ## Interface CLI ## Earliest failure stage Request parsing or validation ## Relevant output or evidence - Expected source: `docs/ghidra-dos.md:26-32` describes DOS structural admission, separation of DOS fields from Windows declarations, and malformed-Windows rejection. Microsoft's [original DOS loader](https://raw.githubusercontent.com/microsoft/MS-DOS/2d04cacc5322951f187bb17e017c12920ac8ebe2/v2.0/source/EXEC.ASM), especially the original header fields and `exec_find_res` / `exec_do_reloc`, reads the initialized module at the paragraph-sized header boundary and reads relocation records separately. The original loader source supports the original layout's padding semantics; `file` independently recognizes both images as MS-DOS executables. - Failure source: Executed public `inspect` and filesystem target resolution for the inline fixtures. `src/domain/dosMz.ts:53-69` misses padding ownership when a relocation table ends before byte 60, then treats every nonzero dword at byte 60 as a Windows declaration for a header of at least 64 bytes. `src/application/BinaryTargetResolver.ts:560-585` follows it into `readPeMetadata`; its fixed-header read at the out-of-file offset returns the reported error. Independent evaluation of the original source helper and compiled helper agreed on every case. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete - Restoration check: The unchanged nonzero-padding fixture must resolve as `dos-mz` / `x86` and reach the same provider-selection stage as its zero-padding control. Keep the malformed Windows controls rejected, preserve existing confirmed-PE and NE/LE/LX behavior, and retain DOS page-count, relocation and entry-point validation. Existing `src/domain/dosMz.test.ts:271-281` specifically requires damaged Windows declarations to remain rejected; restoring this DOS case must preserve that guarantee. Observed output, restricted to fields printed by the reproduction: ```text padding-nonzero: exit=1, code=target_unavailable, reason=invalid or truncated PE header padding-zero: exit=1, code=provider_unavailable, target_formats=[dos-mz] truncated-pe, invalid-windows-signature: exit=1, code=target_unavailable, reason=invalid or truncated PE header ne, le, lx: exit=1, code=target_unavailable, reason=unsupported NE/LE/LX executable in MZ image (respectively) truncated-ne, truncated-le, truncated-lx: exit=1, code=target_unavailable, reason=invalid or truncated PE header ``` The nonzero-padding digest is `ea6747b60d3922712358da7838529185975aa15a82bb36321a1ebc86768b5f6f`; the zero-padding digest is `708b7dc9c79834c86289a17e29a8f2af25a3ec56ebf58263bc1e31939fa6afdc`. The source helper reports `declared`, offset `1145258561`, signature `unobserved` for the nonzero fixture; filesystem resolution of the zero fixture succeeds with `dos-mz` / `x86`. The focused existing source suites `src/domain/dosMz.test.ts` and `src/domain/binaryTarget.test.ts` passed all 67 tests with one Vitest worker; that establishes existing safeguards, not a tested fix. ## Additional context Searched both open and closed issues using `MZ`, `e_lfanew`, `DOS header`, `header padding`, `invalid or truncated PE header`, and `relocation`, and compared likely matches and linked repairs. [#2172](https://github.com/morluto/rea/issues/2172) restores real PE recognition after short DOS stubs; its closing repair preserves the full-header branch implicated here. [#5168](https://github.com/morluto/rea/issues/5168) and the current diff and comments of [PR #5216](https://github.com/morluto/rea/pull/5216) concern signature-confirmed PE at offsets below 64. The PR changes minimum offsets and consumers but leaves the original-DOS false declaration intact; this conclusion comes from its source diff, not a runtime test of the PR. The baseline reproduction has a distinct restoration predicate, and neither repair addresses it. Related high-bit signature and unsupported-format wording reports concern different contracts. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A valid original DOS layout receives a false pre-provider format refusal. The bounded fixtures directly establish admission failure and incorrect format interpretation; severe real-world consequences or an otherwise working provider workflow were not observed. - Workaround status: unverified - Workaround basis: Zeroing the padding is a verified admission bypass for a different artifact, with a different digest. It preserves the initialized module but not the original artifact's identity; no complete workaround for unchanged-artifact DOS admission was verified.

bsakaguchi closed this issue

2026-10-31 00:45:07 +00:00

bsakaguchi commented

2026-10-31 00:45:08 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

At d87e73f, the original false confirmed-PE diagnosis has improved, but the requested admission result has not been restored. The parser now preserves a nonzero unsigned pointer as a candidate; filesystem admission still refuses that candidate while retaining independently validated DOS facts.

I ran a dependency-free probe of the current header helper on the issue's byte-identical fixture pair, not the CLI or a provider. Both validate as DOS with a 64-byte header, two relocations at 28 and a 64-byte module. ABCD at 60 yields candidate, offset 1145258561, signature unobserved; zero padding yields absent. No target was executed.

This now needs an explicit compatibility decision: the current filesystem regression requires refusal for this layout, and the documented admission policy deliberately leaves nonzero unsigned pointers unresolved even when DOS structure validates. Thus this is no longer simply an omitted regression test or the old PE-error path.

To close as restored, the unchanged nonzero-padding fixture must reach provider selection as dos-mz/x86, with a justified rule distinguishing original DOS padding from Windows-header evidence. Preserve confirmed PE, NE/LE/LX handling, malformed/truncated Windows refusal and DOS structural validation. Alternatively, explicitly decide that this ambiguous original-DOS case remains unsupported and reconcile that decision with the requested compatibility. Better diagnostics alone do not establish restored admission.

@coygeek commented on GitHub (Oct 24, 2026): At `d87e73f`, the original false confirmed-PE diagnosis has improved, but the requested admission result has not been restored. [The parser now preserves a nonzero unsigned pointer as a candidate](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/dosMz.ts#L72-L107); [filesystem admission still refuses that candidate](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/BinaryTargetResolver.ts#L842-L859) while retaining independently validated DOS facts. I ran a dependency-free probe of the current header helper on the issue's byte-identical fixture pair, not the CLI or a provider. Both validate as DOS with a 64-byte header, two relocations at 28 and a 64-byte module. `ABCD` at 60 yields `candidate`, offset `1145258561`, signature `unobserved`; zero padding yields `absent`. No target was executed. This now needs an explicit compatibility decision: the [current filesystem regression](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/filesystem/binaryTarget.test.ts#L1222-L1257) requires refusal for this layout, and the [documented admission policy](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/docs/ghidra-dos.md#L26-L38) deliberately leaves nonzero unsigned pointers unresolved even when DOS structure validates. Thus this is no longer simply an omitted regression test or the old PE-error path. To close as restored, the unchanged nonzero-padding fixture must reach provider selection as `dos-mz`/`x86`, with a justified rule distinguishing original DOS padding from Windows-header evidence. Preserve confirmed PE, NE/LE/LX handling, malformed/truncated Windows refusal and DOS structural validation. Alternatively, explicitly decide that this ambiguous original-DOS case remains unsupported and reconcile that decision with the requested compatibility. Better diagnostics alone do not establish restored admission.

bsakaguchi referenced this issue

2026-10-31 01:14:19 +00:00

[PR #4803] [MERGED] fix(setup): recognize shared CLI launcher aliases #6417

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3906

No description provided.

Rows
Columns