deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[GH-ISSUE #156] RuntimeError: Failed: CUDA runtime error csrc/jit/kernel_runtime.hpp:108 '98' #89

New issue
Closed
opened 2026-10-26 21:12:13 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:12:13 +00:00
Owner
Copy link

Originally created by @krishung5 on GitHub (Aug 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/156

Seeing an CUDA runtime error coming from deep_gemm.py when running vLLM WideEP multinode with DeepSeek R1 using commits after https://github.com/deepseek-ai/DeepGEMM/pull/112.

Same issue reported here. It occurs during model loading:

2025-08-01T08:01:22.575940Z ERROR core.run_engine_core: EngineCore failed to start.
Traceback (most recent call last):
  File "/opt/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core
    engine_core = DPEngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/v1/engine/core.py", line 881, in __init__
    super().__init__(vllm_config, local_client, handshake_address,
  File "/opt/vllm/vllm/v1/engine/core.py", line 441, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/opt/vllm/vllm/v1/engine/core.py", line 77, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/opt/vllm/vllm/executor/uniproc_executor.py", line 49, in _init_executor
    self.collective_rpc("load_model")
  File "/opt/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/utils/__init__.py", line 2985, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/v1/worker/gpu_worker.py", line 201, in load_model
    self.model_runner.load_model(eep_scale_up=eep_scale_up)
  File "/opt/vllm/vllm/v1/worker/gpu_model_runner.py", line 1876, in load_model
    self.model = model_loader.load_model(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model
    process_weights_after_loading(model, model_config, target_device)
  File "/opt/vllm/vllm/model_executor/model_loader/utils.py", line 126, in process_weights_after_loading
    module.process_weights_after_loading(model_config.dtype)
  File "/opt/vllm/vllm/attention/layer.py", line 310, in process_weights_after_loading
    self.impl.process_weights_after_loading(act_dtype)
  File "/opt/vllm/vllm/v1/attention/backends/mla/common.py", line 994, in process_weights_after_loading
    kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/v1/attention/backends/mla/common.py", line 983, in get_and_maybe_dequant_weights
    dequant_weights = layer.quant_method.apply(layer,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/model_executor/layers/quantization/fp8.py", line 451, in apply
    return torch.ops.vllm.apply_w8a8_block_fp8_linear(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 147, in apply_w8a8_block_fp8_linear
    output = torch.ops.vllm.w8a8_block_fp8_matmul_deepgemm(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dynamo/venv/lib/python3.12/site-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm/vllm/model_executor/layers/quantization/deepgemm.py", line 58, in w8a8_block_fp8_matmul_deepgemm
    fp8_gemm_nt((A, As), (B, Bs), C)
  File "/opt/vllm/vllm/utils/deep_gemm.py", line 92, in fp8_gemm_nt
    return _fp8_gemm_nt_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed: CUDA runtime error csrc/jit/kernel_runtime.hpp:108 '98'

To repro, build the container using the dockerfile provided in the comment, and run the vllm WideEP multinode example:

# node 1
VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \
    vllm serve  deepseek-ai/DeepSeek-R1 \
    --tensor-parallel-size 1 --enable-expert-parallel --data-parallel-size 16 --data-parallel-size-local 8 --data-parallel-address <node1 ip address> \
    --data-parallel-rpc-port 13345 --api-server-count=8 --gpu-memory-utilization 0.95 --max-model-len 10240 --enforce-eager

# node 2
VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \
    vllm serve  deepseek-ai/DeepSeek-R1 \
    --tensor-parallel-size 1 --enable-expert-parallel --data-parallel-size 16 --data-parallel-size-local 8 --data-parallel-address <node1 ip address> \
    --data-parallel-rpc-port 13345 --data-parallel-start-rank 8 --gpu-memory-utilization 0.95 --max-model-len 10240 --enforce-eager --headless
Originally created by @krishung5 on GitHub (Aug 7, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/156 Seeing an CUDA runtime error coming from deep_gemm.py when running vLLM WideEP multinode with DeepSeek R1 using commits after https://github.com/deepseek-ai/DeepGEMM/pull/112. Same issue reported [here](https://github.com/vllm-project/vllm/issues/19653#issuecomment-3136177524). It occurs during model loading: ``` 2025-08-01T08:01:22.575940Z ERROR core.run_engine_core: EngineCore failed to start. Traceback (most recent call last): File "/opt/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core engine_core = DPEngineCoreProc(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/v1/engine/core.py", line 881, in __init__ super().__init__(vllm_config, local_client, handshake_address, File "/opt/vllm/vllm/v1/engine/core.py", line 441, in __init__ super().__init__(vllm_config, executor_class, log_stats, File "/opt/vllm/vllm/v1/engine/core.py", line 77, in __init__ self.model_executor = executor_class(vllm_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/executor/executor_base.py", line 53, in __init__ self._init_executor() File "/opt/vllm/vllm/executor/uniproc_executor.py", line 49, in _init_executor self.collective_rpc("load_model") File "/opt/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc answer = run_method(self.driver_worker, method, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/utils/__init__.py", line 2985, in run_method return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/v1/worker/gpu_worker.py", line 201, in load_model self.model_runner.load_model(eep_scale_up=eep_scale_up) File "/opt/vllm/vllm/v1/worker/gpu_model_runner.py", line 1876, in load_model self.model = model_loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model process_weights_after_loading(model, model_config, target_device) File "/opt/vllm/vllm/model_executor/model_loader/utils.py", line 126, in process_weights_after_loading module.process_weights_after_loading(model_config.dtype) File "/opt/vllm/vllm/attention/layer.py", line 310, in process_weights_after_loading self.impl.process_weights_after_loading(act_dtype) File "/opt/vllm/vllm/v1/attention/backends/mla/common.py", line 994, in process_weights_after_loading kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/v1/attention/backends/mla/common.py", line 983, in get_and_maybe_dequant_weights dequant_weights = layer.quant_method.apply(layer, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/model_executor/layers/quantization/fp8.py", line 451, in apply return torch.ops.vllm.apply_w8a8_block_fp8_linear( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/dynamo/venv/lib/python3.12/site-packages/torch/_ops.py", line 1158, in __call__ return self._op(*args, **(kwargs or {})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 147, in apply_w8a8_block_fp8_linear output = torch.ops.vllm.w8a8_block_fp8_matmul_deepgemm( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/dynamo/venv/lib/python3.12/site-packages/torch/_ops.py", line 1158, in __call__ return self._op(*args, **(kwargs or {})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/vllm/vllm/model_executor/layers/quantization/deepgemm.py", line 58, in w8a8_block_fp8_matmul_deepgemm fp8_gemm_nt((A, As), (B, Bs), C) File "/opt/vllm/vllm/utils/deep_gemm.py", line 92, in fp8_gemm_nt return _fp8_gemm_nt_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Failed: CUDA runtime error csrc/jit/kernel_runtime.hpp:108 '98' ``` To repro, build the container using the dockerfile provided in the [comment](https://github.com/vllm-project/vllm/issues/19653#issuecomment-3136177524), and run the [vllm WideEP multinode example](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html#multi-node-deployment): ``` # node 1 VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \ vllm serve deepseek-ai/DeepSeek-R1 \ --tensor-parallel-size 1 --enable-expert-parallel --data-parallel-size 16 --data-parallel-size-local 8 --data-parallel-address <node1 ip address> \ --data-parallel-rpc-port 13345 --api-server-count=8 --gpu-memory-utilization 0.95 --max-model-len 10240 --enforce-eager # node 2 VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \ vllm serve deepseek-ai/DeepSeek-R1 \ --tensor-parallel-size 1 --enable-expert-parallel --data-parallel-size 16 --data-parallel-size-local 8 --data-parallel-address <node1 ip address> \ --data-parallel-rpc-port 13345 --data-parallel-start-rank 8 --gpu-memory-utilization 0.95 --max-model-len 10240 --enforce-eager --headless ```
bsakaguchi commented 2026-10-26 21:12:15 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Aug 14, 2025):

98 means cudaErrorInvalidDeviceFunction: The requested device function does not exist or is not compiled for the proper device architecture. You may use DG_JIT_DEBUG=1 to see whether the compiler arch flag (e.g., 90a for Hopper series) is suitable for your own device.

<!-- gh-comment-id:3187562667 --> @LyricZhao commented on GitHub (Aug 14, 2025): `98` means `cudaErrorInvalidDeviceFunction`: The requested device function does not exist or is not compiled for the proper device architecture. You may use `DG_JIT_DEBUG=1` to see whether the compiler arch flag (e.g., `90a` for Hopper series) is suitable for your own device.
bsakaguchi commented 2026-10-26 21:12:15 +00:00
Author
Owner
Copy link

@crischeng commented on GitHub (Aug 22, 2025):

I encountered the same problem on H20-96G
driver version: 565.57.01
nvcc version: cuda12.9

<!-- gh-comment-id:3213640949 --> @crischeng commented on GitHub (Aug 22, 2025): I encountered the same problem on H20-96G driver version: 565.57.01 nvcc version: cuda12.9
bsakaguchi commented 2026-10-26 21:12:16 +00:00
Author
Owner
Copy link

@krishung5 commented on GitHub (Aug 25, 2025):

Using the commit f85ec64 worked from my side. Closing this issue.

<!-- gh-comment-id:3221009390 --> @krishung5 commented on GitHub (Aug 25, 2025): Using the commit f85ec64 worked from my side. Closing this issue.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#89
No description provided.