[GH-ISSUE #84] Error on (M,K,N) of (32, 8192, 8192) #46

New issue

Closed

opened 2026-10-26 21:11:11 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:11:11 +00:00

Owner

Originally created by @veritas9872 on GitHub (Apr 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/84

Hello!

Thank you for this awesome work.

While I was trying to obtain throughput measurements for some shapes, I found that the gemm_fp8_fp8_bf16_nt function produced an error when running on the shape (32 x 8192) x (8192 x 8192), I recieved this error.

Traceback (most recent call last):
  File "/opt/DeepGEMM/tests/test_core.py", line 161, in <module>
    test_gemm()
  File "/opt/DeepGEMM/tests/test_core.py", line 83, in test_gemm
    t = bench_kineto(test_func, 'fp8_gemm', suppress_kineto_output=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/DeepGEMM/deep_gemm/utils.py", line 127, in bench_kineto
    profiler.step()
  File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 825, in step
    self._transit_action(prev_action, self.current_action)
  File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 859, in _transit_action
    action()
  File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 241, in stop_trace
    self.profiler.__exit__(None, None, None)
  File "/usr/local/lib/python3.12/dist-packages/torch/autograd/profiler.py", line 370, in __exit__
    device_module.synchronize()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 999, in synchronize
    return torch._C._cuda_synchronize()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

However, for all other shapes, including (64 x 8192) x (8192 x 8192) and (128 x 8192) x (8192 x 8192), there was no such error.

Could this be an edge case?

Many thanks in advance!

Originally created by @veritas9872 on GitHub (Apr 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/84 Hello! Thank you for this awesome work. While I was trying to obtain throughput measurements for some shapes, I found that the `gemm_fp8_fp8_bf16_nt ` function produced an error when running on the shape (32 x 8192) x (8192 x 8192), I recieved this error. ``` Traceback (most recent call last): File "/opt/DeepGEMM/tests/test_core.py", line 161, in <module> test_gemm() File "/opt/DeepGEMM/tests/test_core.py", line 83, in test_gemm t = bench_kineto(test_func, 'fp8_gemm', suppress_kineto_output=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/DeepGEMM/deep_gemm/utils.py", line 127, in bench_kineto profiler.step() File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 825, in step self._transit_action(prev_action, self.current_action) File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 859, in _transit_action action() File "/usr/local/lib/python3.12/dist-packages/torch/profiler/profiler.py", line 241, in stop_trace self.profiler.__exit__(None, None, None) File "/usr/local/lib/python3.12/dist-packages/torch/autograd/profiler.py", line 370, in __exit__ device_module.synchronize() File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 999, in synchronize return torch._C._cuda_synchronize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` However, for all other shapes, including (64 x 8192) x (8192 x 8192) and (128 x 8192) x (8192 x 8192), there was no such error. Could this be an edge case? Many thanks in advance!

bsakaguchi closed this issue

2026-10-26 21:11:11 +00:00

bsakaguchi commented

2026-10-26 21:11:12 +00:00

Author

Owner

@LyricZhao commented on GitHub (Apr 14, 2025):

I modify the tests (the latest main) into:

def test_gemm() -> None:
    print('Testing GEMM:')
    for m in (32, ):
        for k, n in [(8192, 8192)]:
            x_fp8, y_fp8, out, ref_out = construct(m, k, n)
            deep_gemm.gemm_fp8_fp8_bf16_nt(x_fp8, y_fp8, out)
            diff = calc_diff(out, ref_out)
            assert diff < 0.001, f'{m=}, {k=}, {n=}, {diff:.5f}'

And there is no such error, could you please pull the main branch and retry? If there is still a bug, please reply in this issue. Thanks!

@LyricZhao commented on GitHub (Apr 14, 2025): I modify the tests (the latest main) into: ```python def test_gemm() -> None: print('Testing GEMM:') for m in (32, ): for k, n in [(8192, 8192)]: x_fp8, y_fp8, out, ref_out = construct(m, k, n) deep_gemm.gemm_fp8_fp8_bf16_nt(x_fp8, y_fp8, out) diff = calc_diff(out, ref_out) assert diff < 0.001, f'{m=}, {k=}, {n=}, {diff:.5f}' ``` And there is no such error, could you please pull the main branch and retry? If there is still a bug, please reply in this issue. Thanks!

bsakaguchi referenced this issue

2026-10-26 21:16:54 +00:00

[PR #323] feat: MegaMOE adaptation for SM90 #367