[PR #427] Improve error when cuobjdump is missing or fails #440

New issue

Open

opened 2026-10-26 21:17:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:17:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/427
Author: @monperrus
Created: 9/6/2026
Status: 🔄 Open

Base: main ← Head: fix/cuobjdump-missing-diagnostic

📝 Commits (1)

b75f054 jit: actionable error when cuobjdump is missing or fails

📊 Changes

1 file changed (+19 additions, -2 deletions)

View changed files

📝 csrc/jit/kernel_runtime.hpp (+19 -2)

📄 Description

Problem

KernelRuntime finds a cubin's kernel symbol by shelling out to
$CUDA_HOME/bin/cuobjdump -symbols kernel.cubin and then asserting
DG_HOST_ASSERT(exit_code == 0).

cuobjdump is part of the CUDA toolkit but is packaged separately from
nvcc — e.g. conda's cuda-cuobjdump (or cuda-command-line-tools).
A JIT-only CUDA install that has nvcc but not cuobjdump compiles kernels
fine and then dies here with:

Assertion error (.../csrc/jit/kernel_runtime.hpp:NN): exit_code == 0

which gives the user nothing to go on. call_external_command already
captures the command's combined stdout/stderr (it appends 2>&1), but that
output — which contains the actual cuobjdump: not found / error text — is
discarded on the failure path.

This is easy to hit from the kernels/Hugging Face flow and from minimal
CUDA containers.

Change

Diagnostics only, no behavior change on the success path:

Check cuobjdump exists up front and, if not, report its expected path and
how to fix it (install cuda-cuobjdump / cuda-command-line-tools, or point
CUDA_HOME at a full toolkit).
On a non-zero exit, surface the command, exit code, and the captured output
instead of the bare exit_code == 0 assertion.

Reproduce

A CUDA env with nvcc but no cuobjdump (e.g.
conda create -c nvidia cuda-nvcc without cuda-cuobjdump) — any JIT kernel
load reaches this path.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/427 **Author:** [@monperrus](https://github.com/monperrus) **Created:** 9/6/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/cuobjdump-missing-diagnostic` --- ### 📝 Commits (1) - [`b75f054`](https://github.com/deepseek-ai/DeepGEMM/commit/b75f0542eff590a90d166753c678d02a58008398) jit: actionable error when cuobjdump is missing or fails ### 📊 Changes **1 file changed** (+19 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `csrc/jit/kernel_runtime.hpp` (+19 -2) </details> ### 📄 Description ### Problem `KernelRuntime` finds a cubin's kernel symbol by shelling out to `$CUDA_HOME/bin/cuobjdump -symbols kernel.cubin` and then asserting `DG_HOST_ASSERT(exit_code == 0)`. `cuobjdump` is part of the CUDA toolkit but is packaged **separately from `nvcc`** — e.g. conda's `cuda-cuobjdump` (or `cuda-command-line-tools`). A JIT-only CUDA install that has `nvcc` but not `cuobjdump` compiles kernels fine and then dies here with: ``` Assertion error (.../csrc/jit/kernel_runtime.hpp:NN): exit_code == 0 ``` which gives the user nothing to go on. `call_external_command` already captures the command's combined stdout/stderr (it appends `2>&1`), but that output — which contains the actual `cuobjdump: not found` / error text — is discarded on the failure path. This is easy to hit from the `kernels`/Hugging Face flow and from minimal CUDA containers. ### Change Diagnostics only, no behavior change on the success path: - Check `cuobjdump` exists up front and, if not, report its expected path and how to fix it (install `cuda-cuobjdump` / `cuda-command-line-tools`, or point `CUDA_HOME` at a full toolkit). - On a non-zero exit, surface the command, exit code, and the captured output instead of the bare `exit_code == 0` assertion. ### Reproduce A CUDA env with `nvcc` but no `cuobjdump` (e.g. `conda create -c nvidia cuda-nvcc` without `cuda-cuobjdump`) — any JIT kernel load reaches this path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 21:17:48 +00:00

bsakaguchi referenced this issue

2026-10-26 21:18:00 +00:00

[PR #440] Add a fused SM90 FP8 MegaMoE: one kernel, a lag-ordered schedule and a ring activation pool #449

bsakaguchi referenced this issue

2026-10-26 21:18:18 +00:00

[PR #473] Add optimized fused FP8 x MXFP4 MegaMoE for SM90 #474