[GH-ISSUE #6880] [Bug] HIP bitcode payloads are labeled malformed instead of unsupported #4216

New issue

Closed

opened 2026-10-31 00:47:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:48 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6880

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

inspect-gpu-code-objects --analysis kernels recognizes a Clang HIP bundle containing valid AMDGPU LLVM bitcode, retains its target and payload digest, but reports the payload as inspection: "malformed" with not an ELF image. LLVM successfully decodes the unchanged bitcode. The equivalent ELF payload compiled from that bitcode is inspected successfully.

The requested correction is the typed classification of an unsupported payload. This report does not ask REA to extract kernels from LLVM bitcode.

Reproduction

Use the stated built checkout, Node.js 24.22.1, Python 3, and official LLVM 23.1.4 tools on PATH. Save the following as /tmp/hip-bitcode-repro.py and run python3 /tmp/hip-bitcode-repro.py from the checkout root. This final script was executed with all assertions passing before this draft was written.

The fixture is a trivial local kernel encoded by llvm-as. The real clang-offload-bundler --type=bc binary writer creates a one-entry intermediate hip-amdgcn-amd-amdhsa--gfx1100 bundle. The script checks the bundle's payload range against the exact input bytes. llvm-dis checks the bitcode, while llc produces the ELF control. No target code executes.

from pathlib import Path
import hashlib,json,os,subprocess,tempfile
repo=Path.cwd()
r=Path(tempfile.mkdtemp(prefix='hip-bitcode-',dir=os.environ.get('PROBE_PARENT','/tmp')))
def run(args):
 return subprocess.run(args,capture_output=True,text=True,timeout=40)
ir=r/'kernel.ll';ir.write_text('target triple = "amdgcn-amd-amdhsa"\ndefine amdgpu_kernel void @sample() { ret void }\n')
for args in [['llvm-as',str(ir),'-o',str(r/'kernel.bc')],['llvm-dis',str(r/'kernel.bc'),'-o',str(r/'decoded.ll')],['llc','-mcpu=gfx1100','-filetype=obj',str(r/'kernel.bc'),'-o',str(r/'kernel.o')]]:
 p=run(args);assert p.returncode==0,(args,p.stderr)
for name in ['bc','o']:
 p=run(['clang-offload-bundler','--type=bc','--input='+str(r/f'kernel.{name}'),'--targets=hip-amdgcn-amd-amdhsa--gfx1100','--output='+str(r/f'bundle-{name}.bin')]);assert p.returncode==0,p.stderr
 p=run(['clang-offload-bundler','--type=bc','--input='+str(r/f'bundle-{name}.bin'),'--list']);assert p.returncode==0 and p.stdout.strip()=='hip-amdgcn-amd-amdhsa--gfx1100'
for name in ['bc','o']:
 data=(r/f'bundle-{name}.bin').read_bytes()
 assert data[:24]==b'__CLANG_OFFLOAD_BUNDLE__' and int.from_bytes(data[24:32],'little')==1
 offset=int.from_bytes(data[32:40],'little'); size=int.from_bytes(data[40:48],'little')
 assert data[offset:offset+size]==(r/f'kernel.{name}').read_bytes()
summary=[]
for name,file,analysis in [('bitcode-inventory','bundle-bc.bin','inventory'),('bitcode-kernels','bundle-bc.bin','kernels'),('elf-kernels','bundle-o.bin','kernels'),('raw-bitcode','kernel.bc','kernels')]:
 source=r/file;digest=hashlib.sha256(source.read_bytes()).hexdigest()
 p=run([os.environ.get('NODE_EXE','node'),str(repo/'scripts/rea.mjs'),'inspect-gpu-code-objects',str(source),'--analysis',analysis,'--detail','complete','--format','json'])
 (r/(name+'.json')).write_text(p.stdout);(r/(name+'.stderr')).write_text(p.stderr)
 x=json.loads(p.stdout);n=x.get('normalized_result',{})
 assert hashlib.sha256(source.read_bytes()).hexdigest()==digest
 item={'case':name,'exit':p.returncode,'code':x.get('code'),'objects':[{k:o[k] for k in ['kind','target','inspection','limitations']} for o in n.get('objects',[])],'kernels':[k['name'] for k in n.get('kernels',[])],'coverage':n.get('coverage')}
 summary.append(item)
assert summary[0]['objects'][0]['inspection']=='inventory' and summary[0]['coverage']['payloads']=='not-requested'
assert summary[1]['exit']==0 and summary[1]['objects'][0]['inspection']=='malformed'
assert summary[1]['objects'][0]['limitations']==['not an ELF image']
assert summary[2]['objects'][0]['inspection']=='complete' and summary[2]['kernels']==['sample']
assert summary[3]['code']=='unsupported_target'
(r/'summary.json').write_text(json.dumps(summary,indent=2))
print(json.dumps(summary,indent=2));print('evidence:',r)

Expected behavior

Keep the recognized bundle, gfx1100, payload locations and digest. Mark the bitcode entry inspection: "unsupported", explain that bitcode metadata is outside current kernel coverage, and keep payload coverage partial. Corrupted recognized ELF objects should still be marked malformed.

REA's binary failure model explicitly distinguishes corruption from containers outside parser coverage. Its boundary rule separates recognition from provider admission. LLVM 23.1.4's released bundler documentation describes hip for intermediate toolchain bundles; this fixture uses that kind rather than claiming a runtime-loadable hipv4 ELF image.

Impact and scope

The observed consequence is a false machine-readable malformed classification for one valid, manually bundled intermediate artifact. No data corruption, lost payload bytes, false complete coverage or runtime-loading claim is established. Other producer pipelines and real-world prevalence were not measured.

A verified fallback is --analysis inventory, which retains the bundle and target without the false malformed judgment, followed by direct llvm-dis for static IR inspection. This does not supply bitcode kernel analysis in REA. The ELF control demonstrates the currently supported metadata path but changes artifact representation.

REA version

Package 6.7.0; exact tested main source/build commit d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f. No installed or published REA package was used as the code under test.

Runtime and platform

Node.js v24.22.1; Linux 7.2.8-9-omarchy x86_64; locked repository dependencies; official LLVM 23.1.4 Linux x64 release tools. The release archive SHA-256 matched GitHub's published asset digest. Earlier exploratory checks with LLVM 23.1.3 showed the same classification; the final script used 23.1.4.

Target and artifact context

A self-authored amdgpu_kernel void @sample() with an immediate return, encoded as LLVM bitcode and as an ELF object. Both are placed in the same generic binary bundle format with the documented intermediate HIP target ID. The script checks that each embedded payload exactly equals its selected compiler output and that REA leaves every inspected file unchanged.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The final script observed:

bitcode-inventory: exit 0, target gfx1100, inspection inventory,
                  coverage.payloads not-requested
bitcode-kernels:   exit 0, target gfx1100, inspection malformed,
                  limitations ["not an ELF image"], coverage.payloads partial
elf-kernels:       exit 0, target gfx1100, inspection complete,
                  kernels ["sample"], coverage.payloads complete
raw-bitcode:       exit 1, code unsupported_target

All three LLVM production/decoding commands succeeded, as did bundle creation/listing and exact payload comparisons. Each complete CLI JSON response and stderr is saved by the script along with summary.json. The raw-bitcode refusal is a control for unsupported format recognition, not an expectation that an embedded-object status use an operation-level error code.

Typed outcome or recovery expected

Object inspection: "unsupported" with an explicit bitcode coverage limitation and retained identity. The surrounding recognized HIP bundle can still return successful partial Evidence. Kernel extraction remains outside this report.

Analysis or suspected cause

Source-derived: Clang bundle admission assigns kind amdgpu from the bundle ID. Payload dispatch sends that kind directly into ELF parsing. The ELF reader's missing-magic failure then becomes the malformed object state. The target declaration identifies the toolchain target without establishing an ELF payload representation.

Additional context

#3677 requests LLVM bitcode/LTO analysis. The correction here can preserve that unsupported capability while avoiding a false corruption label. Searches for bitcode, HIP plus malformed, and the exact not an ELF image diagnostic found no matching classification report before drafting. A fresh duplicate search is performed before submission.

CLI inspection and direct LLVM tools were exercised; MCP, published-package behavior, a full HIP compiler driver build, and GPU execution were not. AI-assisted investigation. All bug reproduction and controls finished before drafting; no post-draft reproduction is claimed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The real public CLI returns an incorrect machine-readable malformed state for an unchanged valid bitcode payload; the declared target and payload bytes are correctly retained. Broader workflow consequences are unverified.
Workaround status: available
Workaround basis: The executed inventory-mode control retains the bundle/target without a malformed judgment, and llvm-dis successfully exposes the same unchanged bitcode as static IR. This fallback does not add bitcode kernel extraction to REA.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6880 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Native analysis or decompilation ## Observed behavior `inspect-gpu-code-objects --analysis kernels` recognizes a Clang HIP bundle containing valid AMDGPU LLVM bitcode, retains its target and payload digest, but reports the payload as `inspection: "malformed"` with `not an ELF image`. LLVM successfully decodes the unchanged bitcode. The equivalent ELF payload compiled from that bitcode is inspected successfully. The requested correction is the typed classification of an unsupported payload. This report does not ask REA to extract kernels from LLVM bitcode. ## Reproduction Use the stated built checkout, Node.js 24.22.1, Python 3, and official LLVM 23.1.4 tools on PATH. Save the following as `/tmp/hip-bitcode-repro.py` and run `python3 /tmp/hip-bitcode-repro.py` from the checkout root. This final script was executed with all assertions passing before this draft was written. The fixture is a trivial local kernel encoded by `llvm-as`. The real `clang-offload-bundler --type=bc` binary writer creates a one-entry intermediate `hip-amdgcn-amd-amdhsa--gfx1100` bundle. The script checks the bundle's payload range against the exact input bytes. `llvm-dis` checks the bitcode, while `llc` produces the ELF control. No target code executes. ```python from pathlib import Path import hashlib,json,os,subprocess,tempfile repo=Path.cwd() r=Path(tempfile.mkdtemp(prefix='hip-bitcode-',dir=os.environ.get('PROBE_PARENT','/tmp'))) def run(args): return subprocess.run(args,capture_output=True,text=True,timeout=40) ir=r/'kernel.ll';ir.write_text('target triple = "amdgcn-amd-amdhsa"\ndefine amdgpu_kernel void @sample() { ret void }\n') for args in [['llvm-as',str(ir),'-o',str(r/'kernel.bc')],['llvm-dis',str(r/'kernel.bc'),'-o',str(r/'decoded.ll')],['llc','-mcpu=gfx1100','-filetype=obj',str(r/'kernel.bc'),'-o',str(r/'kernel.o')]]: p=run(args);assert p.returncode==0,(args,p.stderr) for name in ['bc','o']: p=run(['clang-offload-bundler','--type=bc','--input='+str(r/f'kernel.{name}'),'--targets=hip-amdgcn-amd-amdhsa--gfx1100','--output='+str(r/f'bundle-{name}.bin')]);assert p.returncode==0,p.stderr p=run(['clang-offload-bundler','--type=bc','--input='+str(r/f'bundle-{name}.bin'),'--list']);assert p.returncode==0 and p.stdout.strip()=='hip-amdgcn-amd-amdhsa--gfx1100' for name in ['bc','o']: data=(r/f'bundle-{name}.bin').read_bytes() assert data[:24]==b'__CLANG_OFFLOAD_BUNDLE__' and int.from_bytes(data[24:32],'little')==1 offset=int.from_bytes(data[32:40],'little'); size=int.from_bytes(data[40:48],'little') assert data[offset:offset+size]==(r/f'kernel.{name}').read_bytes() summary=[] for name,file,analysis in [('bitcode-inventory','bundle-bc.bin','inventory'),('bitcode-kernels','bundle-bc.bin','kernels'),('elf-kernels','bundle-o.bin','kernels'),('raw-bitcode','kernel.bc','kernels')]: source=r/file;digest=hashlib.sha256(source.read_bytes()).hexdigest() p=run([os.environ.get('NODE_EXE','node'),str(repo/'scripts/rea.mjs'),'inspect-gpu-code-objects',str(source),'--analysis',analysis,'--detail','complete','--format','json']) (r/(name+'.json')).write_text(p.stdout);(r/(name+'.stderr')).write_text(p.stderr) x=json.loads(p.stdout);n=x.get('normalized_result',{}) assert hashlib.sha256(source.read_bytes()).hexdigest()==digest item={'case':name,'exit':p.returncode,'code':x.get('code'),'objects':[{k:o[k] for k in ['kind','target','inspection','limitations']} for o in n.get('objects',[])],'kernels':[k['name'] for k in n.get('kernels',[])],'coverage':n.get('coverage')} summary.append(item) assert summary[0]['objects'][0]['inspection']=='inventory' and summary[0]['coverage']['payloads']=='not-requested' assert summary[1]['exit']==0 and summary[1]['objects'][0]['inspection']=='malformed' assert summary[1]['objects'][0]['limitations']==['not an ELF image'] assert summary[2]['objects'][0]['inspection']=='complete' and summary[2]['kernels']==['sample'] assert summary[3]['code']=='unsupported_target' (r/'summary.json').write_text(json.dumps(summary,indent=2)) print(json.dumps(summary,indent=2));print('evidence:',r) ``` ## Expected behavior Keep the recognized bundle, `gfx1100`, payload locations and digest. Mark the bitcode entry `inspection: "unsupported"`, explain that bitcode metadata is outside current kernel coverage, and keep payload coverage partial. Corrupted recognized ELF objects should still be marked malformed. REA's [binary failure model](https://github.com/morluto/rea/blob/d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f/src/native/metadata/BinaryContainer.ts#L1-L9) explicitly distinguishes corruption from containers outside parser coverage. Its [boundary rule](https://github.com/morluto/rea/blob/d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f/docs/tool-design.md#L87-L90) separates recognition from provider admission. [LLVM 23.1.4's released bundler documentation](https://github.com/llvm/llvm-project/blob/llvmorg-23.1.4/clang/docs/ClangOffloadBundler.rst#L237-L245) describes `hip` for intermediate toolchain bundles; this fixture uses that kind rather than claiming a runtime-loadable `hipv4` ELF image. ## Impact and scope The observed consequence is a false machine-readable malformed classification for one valid, manually bundled intermediate artifact. No data corruption, lost payload bytes, false complete coverage or runtime-loading claim is established. Other producer pipelines and real-world prevalence were not measured. A verified fallback is `--analysis inventory`, which retains the bundle and target without the false malformed judgment, followed by direct `llvm-dis` for static IR inspection. This does not supply bitcode kernel analysis in REA. The ELF control demonstrates the currently supported metadata path but changes artifact representation. ## REA version Package 6.7.0; exact tested main source/build commit `d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f`. No installed or published REA package was used as the code under test. ## Runtime and platform Node.js v24.22.1; Linux 7.2.8-9-omarchy x86_64; locked repository dependencies; official LLVM 23.1.4 Linux x64 release tools. The release archive SHA-256 matched GitHub's published asset digest. Earlier exploratory checks with LLVM 23.1.3 showed the same classification; the final script used 23.1.4. ## Target and artifact context A self-authored `amdgpu_kernel void @sample()` with an immediate return, encoded as LLVM bitcode and as an ELF object. Both are placed in the same generic binary bundle format with the documented intermediate HIP target ID. The script checks that each embedded payload exactly equals its selected compiler output and that REA leaves every inspected file unchanged. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence The final script observed: ```text bitcode-inventory: exit 0, target gfx1100, inspection inventory, coverage.payloads not-requested bitcode-kernels: exit 0, target gfx1100, inspection malformed, limitations ["not an ELF image"], coverage.payloads partial elf-kernels: exit 0, target gfx1100, inspection complete, kernels ["sample"], coverage.payloads complete raw-bitcode: exit 1, code unsupported_target ``` All three LLVM production/decoding commands succeeded, as did bundle creation/listing and exact payload comparisons. Each complete CLI JSON response and stderr is saved by the script along with `summary.json`. The raw-bitcode refusal is a control for unsupported format recognition, not an expectation that an embedded-object status use an operation-level error code. ## Typed outcome or recovery expected Object `inspection: "unsupported"` with an explicit bitcode coverage limitation and retained identity. The surrounding recognized HIP bundle can still return successful partial Evidence. Kernel extraction remains outside this report. ## Analysis or suspected cause Source-derived: [Clang bundle admission](https://github.com/morluto/rea/blob/d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f/src/gpu/ClangOffloadBundle.ts#L63-L74) assigns kind `amdgpu` from the bundle ID. [Payload dispatch](https://github.com/morluto/rea/blob/d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f/src/gpu/GpuInventory.ts#L396-L397) sends that kind directly into ELF parsing. The ELF reader's missing-magic failure then becomes [the malformed object state](https://github.com/morluto/rea/blob/d78892eb4ba41cf8746e5410b3ddf6e7f0ba524f/src/gpu/GpuInventory.ts#L281-L294). The target declaration identifies the toolchain target without establishing an ELF payload representation. ## Additional context [#3677](https://github.com/morluto/rea/issues/3677) requests LLVM bitcode/LTO analysis. The correction here can preserve that unsupported capability while avoiding a false corruption label. Searches for bitcode, HIP plus malformed, and the exact `not an ELF image` diagnostic found no matching classification report before drafting. A fresh duplicate search is performed before submission. CLI inspection and direct LLVM tools were exercised; MCP, published-package behavior, a full HIP compiler driver build, and GPU execution were not. AI-assisted investigation. All bug reproduction and controls finished before drafting; no post-draft reproduction is claimed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The real public CLI returns an incorrect machine-readable malformed state for an unchanged valid bitcode payload; the declared target and payload bytes are correctly retained. Broader workflow consequences are unverified. - Workaround status: available - Workaround basis: The executed inventory-mode control retains the bundle/target without a malformed judgment, and llvm-dis successfully exposes the same unchanged bitcode as static IR. This fallback does not add bitcode kernel extraction to REA.

bsakaguchi closed this issue

2026-10-31 00:47:48 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:05 +00:00

[PR #4216] [MERGED] fix(sqlite): honor selected-table reply budgets #6128