morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7333] [Bug] PE imphash withholding reason says pefile omits a long name that it includes in its hash #4362

New issue

Closed

opened 2026-10-31 00:49:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:13 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7333

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

For a standard import named by 513 ASCII A bytes, inspect-pe-imports retains the complete name and correctly returns imphash.md5: null, but its reason says pefile treats the name as invalid and omits it. The pinned pefile producer instead retains the first 512 bytes as a named import and includes them in its hash. This report concerns the explanation; the documented hash-withholding behavior should remain.

Reproduction

Run from a built source checkout using the runtime below. The Python oracle must already be available as pefile==2024.8.26 on Python's module search path. No Windows host, target execution, SDK or disassembler is needed.

The following generator encodes the same independent 2048-byte PE32 import-table fixtures used in verification. It makes an aligned, read-only data section containing one descriptor, matching terminated ILT/IAT arrays, k.dll, and a terminated hint/name entry:

work_dir=$(mktemp -d)
mkdir "$work_dir/home"
python3 - "$work_dir" <<'PY'
from pathlib import Path
import struct
import sys

def fixture(name):
    data = bytearray(0x800)
    def put(fmt, offset, *values):
        struct.pack_into('<' + fmt, data, offset, *values)
    data[:2] = b'MZ'
    put('I', 0x3c, 0x80)
    data[0x80:0x84] = b'PE\0\0'
    put('HHIIIHH', 0x84, 0x14c, 1, 0, 0, 0, 224, 0x2102)
    opt = 0x98
    put('H', opt, 0x10b)
    put('I', opt + 8, 0x600)
    put('I', opt + 28, 0x10000000)
    put('II', opt + 32, 0x1000, 0x200)
    put('HH', opt + 40, 6, 0)
    put('HH', opt + 48, 6, 0)
    put('II', opt + 56, 0x2000, 0x200)
    put('H', opt + 68, 3)
    put('IIII', opt + 72, 0x100000, 0x1000, 0x100000, 0x1000)
    put('I', opt + 92, 16)
    put('II', opt + 104, 0x1000, 40)
    section = opt + 224
    data[section:section + 6] = b'.rdata'
    put('IIII', section + 8, 0x600, 0x1000, 0x600, 0x200)
    put('I', section + 36, 0x40000040)
    put('IIIII', 0x200, 0x1040, 0, 0, 0x1060, 0x1050)
    put('II', 0x240, 0x1080, 0)
    put('II', 0x250, 0x1080, 0)
    data[0x260:0x266] = b'k.dll\0'
    put('H', 0x280, 0)
    data[0x282:0x282 + len(name) + 1] = name + b'\0'
    return bytes(data)

root = Path(sys.argv[1])
for label, name in [('short', b'ExitProcess'), ('name-511', b'A' * 511),
                    ('name-512', b'A' * 512), ('name-513', b'A' * 513),
                    ('invalid-character', b'Has Space')]:
    (root / (label + '.dll')).write_bytes(fixture(name))
PY

for case in short name-511 name-512 name-513 invalid-character; do
  HOME="$work_dir/home" XDG_CONFIG_HOME="$work_dir/home/.config" \
    XDG_CACHE_HOME="$work_dir/home/.cache" \
    node scripts/rea.mjs inspect-pe-imports "$work_dir/$case.dll" --json
done

PYTHONDONTWRITEBYTECODE=1 python3 - "$work_dir" <<'PY'
from pathlib import Path
import pefile
import sys
assert pefile.__version__ == '2024.8.26', pefile.__version__
for path in sorted(Path(sys.argv[1]).glob('*.dll')):
    pe = pefile.PE(str(path))
    names = [f.name for d in getattr(pe, 'DIRECTORY_ENTRY_IMPORT', [])
             for f in d.imports]
    print(path.name, [len(n) for n in names], repr(pe.get_imphash()))
    pe.close()
PY

I generated each fixture independently and ran one public CLI call and one oracle parse per fixture. All five CLI calls exited 0 with empty stderr, complete coverage, one native import and no issues. The fixtures remained unchanged. Has Space is an intentionally invalid-character control, not a claim about a valid Windows symbol.

Expected behavior

For the 513-byte name, retain the full native name, complete coverage and null imphash, with an explanation consistent with pefile's bounded-name read and inclusion of its 512-byte prefix. docs/pe-imports.md:52–56 already distinguishes names pefile considers invalid from names it truncates. Preserve that withholding contract and normal short-name hashing. At exactly 512 bytes, pefile retains the entire name; the result does not establish that the current conservative withholding threshold should change.

Impact and scope

P3: the explanation falsely describes how the pinned producer handles long names. This can mislead someone comparing REA and pefile, but no wrong non-null REA hash, lost native import, crash or consequential behavior failure was observed. The accurate documentation and an independent producer check provide a practical way to interpret the result. Observed scope is the two length-boundary cases above; the first affected release and prevalence in real artifacts are NOT_ENOUGH_INFO.

REA version

6.8.0, source revision dbe2c97af3; existing compiled build supplied for that revision.

Runtime and platform

Node.js v24.22.1; Python 3.15.0; macOS 27.0.1 arm64; pefile 2024.8.26. The oracle module's SHA-256 matches the pinned identity documented in the repository: 0d0eb68f0f169182613dc64b3ab50b20855508c1c1d2faafe8fccf23edb6a345.

Target and artifact context

Synthetic 2048-byte PE32 DLL metadata fixtures. ASCII names and the entire hint/name entry fit inside one file-backed section; descriptor and thunk arrays are terminated. Files were parsed only, never executed. Windows loader or symbol-binding acceptance was not tested. pefile emits sparse-file and zero-entrypoint warnings for these synthetic files but retains the long-name import.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Public CLI normalized_result.imphash for the 513-byte case:

{
  "algorithm": "pefile",
  "md5": null,
  "reason": "Standard import descriptor 0 has a function name with a length or characters pefile treats as invalid and omits."
}

The same result has coverage.status: "complete", coverage.imports: 1, issues: [], and the full 513-byte name in imports[0].functions[0].name.

Function name	pefile named rows / name bytes	pefile hash	REA hash
`ExitProcess`	1 / 11	`4af0cb8f5c6a7923beaa51d01993185b`	Same
511 `A` bytes	1 / 511	`897726129e4219bdf7853234ba4eeb69`	Same
512 `A` bytes	1 / 512	`03252a41c9b51dd793fbdaa719d82e42`	Null
513 `A` bytes	1 / 512	`03252a41c9b51dd793fbdaa719d82e42`	Null
`Has Space`	0 / absent	Empty string	Null

For the 513-byte case, pefile's hash equals MD5 of k. plus 512 lowercase a bytes. Hashing REA's complete 513-byte name would instead give 38aaa73bb4998b4774d99fc40d59a51b, supporting the current decision to withhold. Unlike the long-name cases, the invalid-character control actually loses its row in pefile.

Typed outcome or recovery expected

Keep the existing report shape, full native import name and null imphash. Correct the explanation of the producer behavior without emitting a hash of the truncated prefix as though it represented the complete native name.

Analysis or suspected cause

Source-derived: src/native/pe/pefileImphash.ts:108–111 groups null names, names at least 512 bytes long and invalid characters under one invalid-and-omitted explanation. In pinned pefile, parse_imports() reads MAX_IMPORT_NAME_LENGTH bytes through get_string_at_rva(), validates the retained letters and appends the import. get_imphash() then includes it. The shared REA service emits the reason unchanged after validating artifact identity, coverage and counts. No fix was applied or tested.

Additional context

Expected source: docs/pe-imports.md:45–56 distinguishes truncation from invalid-name handling and documents hash withholding. The pinned producer identity is recorded under Independent oracle and regressions in docs/verification/pe-imports-2026-10-24.md.
Failure source: observed public CLI output and oracle results above; producing branch at src/native/pe/pefileImphash.ts:108–111.
Evidence provenance: observed; source traces explain the observed difference.
Local verification: reproduced through the public compiled CLI and pinned pefile, using independently generated fixtures and isolated CLI home directories. No installation, build, full suite or target execution ran. MCP was not invoked.
Reproduction completeness: complete for this CLI explanation defect. The inline generator encodes the verified fixture bytes. First affected release and Windows loading behavior remain unestablished and are not needed to compare these parser results.
Restoration check: repeat the 513-byte case and confirm an accurate producer explanation, the unchanged full native name, complete coverage and null hash. Preserve the normal hashes for short and 511-byte names, existing withholding at the boundary, and correct treatment of genuinely invalid names. Post-fix restoration has not been run.
Related history: closed PR #6509 added import inspection; open feature #4539 requests broader import triage. Neither reports this explanation defect. The saved complete history and final live issue and pull-request searches contained no exact duplicate.

Triage assessment

Impact level: P3
Assessment status: supported
Impact basis: The public reason incorrectly describes the pinned producer's long-name handling; native inventory and conservative hash withholding remain intact, with no observed consequential behavior change.
Workaround status: available
Workaround basis: The current PE-import documentation accurately distinguishes truncation from invalid names; the pinned producer independently confirms that the long-name row remains and participates in its hash.

Originally created by @coygeek on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7333 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Native analysis or decompilation ## Observed behavior For a standard import named by 513 ASCII `A` bytes, `inspect-pe-imports` retains the complete name and correctly returns `imphash.md5: null`, but its reason says pefile treats the name as invalid and omits it. The pinned pefile producer instead retains the first 512 bytes as a named import and includes them in its hash. This report concerns the explanation; the documented hash-withholding behavior should remain. ## Reproduction Run from a built source checkout using the runtime below. The Python oracle must already be available as `pefile==2024.8.26` on Python's module search path. No Windows host, target execution, SDK or disassembler is needed. The following generator encodes the same independent 2048-byte PE32 import-table fixtures used in verification. It makes an aligned, read-only data section containing one descriptor, matching terminated ILT/IAT arrays, `k.dll`, and a terminated hint/name entry: ```sh work_dir=$(mktemp -d) mkdir "$work_dir/home" python3 - "$work_dir" <<'PY' from pathlib import Path import struct import sys def fixture(name): data = bytearray(0x800) def put(fmt, offset, *values): struct.pack_into('<' + fmt, data, offset, *values) data[:2] = b'MZ' put('I', 0x3c, 0x80) data[0x80:0x84] = b'PE\0\0' put('HHIIIHH', 0x84, 0x14c, 1, 0, 0, 0, 224, 0x2102) opt = 0x98 put('H', opt, 0x10b) put('I', opt + 8, 0x600) put('I', opt + 28, 0x10000000) put('II', opt + 32, 0x1000, 0x200) put('HH', opt + 40, 6, 0) put('HH', opt + 48, 6, 0) put('II', opt + 56, 0x2000, 0x200) put('H', opt + 68, 3) put('IIII', opt + 72, 0x100000, 0x1000, 0x100000, 0x1000) put('I', opt + 92, 16) put('II', opt + 104, 0x1000, 40) section = opt + 224 data[section:section + 6] = b'.rdata' put('IIII', section + 8, 0x600, 0x1000, 0x600, 0x200) put('I', section + 36, 0x40000040) put('IIIII', 0x200, 0x1040, 0, 0, 0x1060, 0x1050) put('II', 0x240, 0x1080, 0) put('II', 0x250, 0x1080, 0) data[0x260:0x266] = b'k.dll\0' put('H', 0x280, 0) data[0x282:0x282 + len(name) + 1] = name + b'\0' return bytes(data) root = Path(sys.argv[1]) for label, name in [('short', b'ExitProcess'), ('name-511', b'A' * 511), ('name-512', b'A' * 512), ('name-513', b'A' * 513), ('invalid-character', b'Has Space')]: (root / (label + '.dll')).write_bytes(fixture(name)) PY for case in short name-511 name-512 name-513 invalid-character; do HOME="$work_dir/home" XDG_CONFIG_HOME="$work_dir/home/.config" \ XDG_CACHE_HOME="$work_dir/home/.cache" \ node scripts/rea.mjs inspect-pe-imports "$work_dir/$case.dll" --json done PYTHONDONTWRITEBYTECODE=1 python3 - "$work_dir" <<'PY' from pathlib import Path import pefile import sys assert pefile.__version__ == '2024.8.26', pefile.__version__ for path in sorted(Path(sys.argv[1]).glob('*.dll')): pe = pefile.PE(str(path)) names = [f.name for d in getattr(pe, 'DIRECTORY_ENTRY_IMPORT', []) for f in d.imports] print(path.name, [len(n) for n in names], repr(pe.get_imphash())) pe.close() PY ``` I generated each fixture independently and ran one public CLI call and one oracle parse per fixture. All five CLI calls exited 0 with empty stderr, complete coverage, one native import and no issues. The fixtures remained unchanged. `Has Space` is an intentionally invalid-character control, not a claim about a valid Windows symbol. ## Expected behavior For the 513-byte name, retain the full native name, complete coverage and null imphash, with an explanation consistent with pefile's bounded-name read and inclusion of its 512-byte prefix. `docs/pe-imports.md:52–56` already distinguishes names pefile considers invalid from names it truncates. Preserve that withholding contract and normal short-name hashing. At exactly 512 bytes, pefile retains the entire name; the result does not establish that the current conservative withholding threshold should change. ## Impact and scope P3: the explanation falsely describes how the pinned producer handles long names. This can mislead someone comparing REA and pefile, but no wrong non-null REA hash, lost native import, crash or consequential behavior failure was observed. The accurate documentation and an independent producer check provide a practical way to interpret the result. Observed scope is the two length-boundary cases above; the first affected release and prevalence in real artifacts are NOT_ENOUGH_INFO. ## REA version 6.8.0, source revision dbe2c97af34361b11b9504e37611dd47715b52cb; existing compiled build supplied for that revision. ## Runtime and platform Node.js v24.22.1; Python 3.15.0; macOS 27.0.1 arm64; pefile 2024.8.26. The oracle module's SHA-256 matches the pinned identity documented in the repository: `0d0eb68f0f169182613dc64b3ab50b20855508c1c1d2faafe8fccf23edb6a345`. ## Target and artifact context Synthetic 2048-byte PE32 DLL metadata fixtures. ASCII names and the entire hint/name entry fit inside one file-backed section; descriptor and thunk arrays are terminated. Files were parsed only, never executed. Windows loader or symbol-binding acceptance was not tested. pefile emits sparse-file and zero-entrypoint warnings for these synthetic files but retains the long-name import. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Public CLI `normalized_result.imphash` for the 513-byte case: ```json { "algorithm": "pefile", "md5": null, "reason": "Standard import descriptor 0 has a function name with a length or characters pefile treats as invalid and omits." } ``` The same result has `coverage.status: "complete"`, `coverage.imports: 1`, `issues: []`, and the full 513-byte name in `imports[0].functions[0].name`. | Function name | pefile named rows / name bytes | pefile hash | REA hash | | --- | --- | --- | --- | | `ExitProcess` | 1 / 11 | `4af0cb8f5c6a7923beaa51d01993185b` | Same | | 511 `A` bytes | 1 / 511 | `897726129e4219bdf7853234ba4eeb69` | Same | | 512 `A` bytes | 1 / 512 | `03252a41c9b51dd793fbdaa719d82e42` | Null | | 513 `A` bytes | 1 / 512 | `03252a41c9b51dd793fbdaa719d82e42` | Null | | `Has Space` | 0 / absent | Empty string | Null | For the 513-byte case, pefile's hash equals MD5 of `k.` plus 512 lowercase `a` bytes. Hashing REA's complete 513-byte name would instead give `38aaa73bb4998b4774d99fc40d59a51b`, supporting the current decision to withhold. Unlike the long-name cases, the invalid-character control actually loses its row in pefile. ## Typed outcome or recovery expected Keep the existing report shape, full native import name and null imphash. Correct the explanation of the producer behavior without emitting a hash of the truncated prefix as though it represented the complete native name. ## Analysis or suspected cause Source-derived: `src/native/pe/pefileImphash.ts:108–111` groups null names, names at least 512 bytes long and invalid characters under one invalid-and-omitted explanation. In pinned pefile, `parse_imports()` reads `MAX_IMPORT_NAME_LENGTH` bytes through `get_string_at_rva()`, validates the retained letters and appends the import. `get_imphash()` then includes it. The shared REA service emits the reason unchanged after validating artifact identity, coverage and counts. No fix was applied or tested. ## Additional context - Expected source: `docs/pe-imports.md:45–56` distinguishes truncation from invalid-name handling and documents hash withholding. The pinned producer identity is recorded under Independent oracle and regressions in `docs/verification/pe-imports-2026-10-24.md`. - Failure source: observed public CLI output and oracle results above; producing branch at `src/native/pe/pefileImphash.ts:108–111`. - Evidence provenance: observed; source traces explain the observed difference. - Local verification: reproduced through the public compiled CLI and pinned pefile, using independently generated fixtures and isolated CLI home directories. No installation, build, full suite or target execution ran. MCP was not invoked. - Reproduction completeness: complete for this CLI explanation defect. The inline generator encodes the verified fixture bytes. First affected release and Windows loading behavior remain unestablished and are not needed to compare these parser results. - Restoration check: repeat the 513-byte case and confirm an accurate producer explanation, the unchanged full native name, complete coverage and null hash. Preserve the normal hashes for short and 511-byte names, existing withholding at the boundary, and correct treatment of genuinely invalid names. Post-fix restoration has not been run. - Related history: closed [PR #6509](https://github.com/morluto/rea/pull/6509) added import inspection; open [feature #4539](https://github.com/morluto/rea/issues/4539) requests broader import triage. Neither reports this explanation defect. The saved complete history and final live issue and pull-request searches contained no exact duplicate. ## Triage assessment - Impact level: P3 - Assessment status: supported - Impact basis: The public reason incorrectly describes the pinned producer's long-name handling; native inventory and conservative hash withholding remain intact, with no observed consequential behavior change. - Workaround status: available - Workaround basis: The current PE-import documentation accurately distinguishes truncation from invalid names; the pinned producer independently confirms that the long-name row remains and participates in its hash.

bsakaguchi closed this issue

2026-10-31 00:49:13 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:39 +00:00

[PR #4362] feat(unreal): inspect Unreal pak footers and indexes #6204

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425