[GH-ISSUE #9] AssertionError: Errors of the kernel fp8_gemm in the profiling table #8

New issue

Open

opened 2026-10-26 21:10:13 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:10:13 +00:00

Owner

Originally created by @vlluvia on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/9

Python 3.12.3
CUDA 12.6
pytorch 2.6
CUTLASS 3.8

Originally created by @vlluvia on GitHub (Feb 26, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/9 Python 3.12.3 CUDA 12.6 pytorch 2.6 CUTLASS 3.8

bsakaguchi referenced this issue

2026-10-26 21:10:13 +00:00

[GH-ISSUE #8] Group Gemm Baseline #9

bsakaguchi commented

2026-10-26 21:10:14 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 26, 2025):

This means the bench_kineto function does not detect any kernel running.

For all bench_kineto function call, you can set suppress_kineto_output=False, which may print more error information and share to us.

@LyricZhao commented on GitHub (Feb 26, 2025): This means the `bench_kineto` function does not detect any kernel running. For all `bench_kineto` function call, you can set `suppress_kineto_output=False`, which may print more error information and share to us.

bsakaguchi commented

2026-10-26 21:10:14 +00:00

Author

Owner

@vlluvia commented on GitHub (Feb 26, 2025):

Library path:

['/root/DeepGEMM/deep_gemm']

Testing GEMM:
WARNING:2025-02-26 09:43:38 68350:68350 init.cpp:178] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_NOT_INITIALIZED (15)
WARNING:2025-02-26 09:43:38 68350:68350 init.cpp:179] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2025-02-26 09:43:38 68350:68350 init.cpp:181] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti
Traceback (most recent call last):
File "/root/DeepGEMM/tests/test_core.py", line 156, in
test_gemm()
File "/root/DeepGEMM/tests/test_core.py", line 80, in test_gemm
t = bench_kineto(test_func, 'fp8_gemm', suppress_kineto_output=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/DeepGEMM/deep_gemm/utils.py", line 119, in bench_kineto
assert sum([name in line for line in prof_lines]) == 1, f'Errors of the kernel {name} in the profiling table'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Errors of the kernel fp8_gemm in the profiling table

@vlluvia commented on GitHub (Feb 26, 2025): Library path: > ['/root/DeepGEMM/deep_gemm'] Testing GEMM: WARNING:2025-02-26 09:43:38 68350:68350 init.cpp:178] function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_NOT_INITIALIZED (15) WARNING:2025-02-26 09:43:38 68350:68350 init.cpp:179] CUPTI initialization failed - CUDA profiler activities will be missing INFO:2025-02-26 09:43:38 68350:68350 init.cpp:181] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti Traceback (most recent call last): File "/root/DeepGEMM/tests/test_core.py", line 156, in <module> test_gemm() File "/root/DeepGEMM/tests/test_core.py", line 80, in test_gemm t = bench_kineto(test_func, 'fp8_gemm', suppress_kineto_output=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/DeepGEMM/deep_gemm/utils.py", line 119, in bench_kineto assert sum([name in line for line in prof_lines]) == 1, f'Errors of the kernel {name} in the profiling table' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Errors of the kernel fp8_gemm in the profiling table

bsakaguchi commented

2026-10-26 21:10:15 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 26, 2025):

Seems you don't have sufficient privilege for CUPTI profiling, which is important for microsecond-level accurate timing.

Try to follow the PyTorch Kineto profile warning information to solve the issue. Thanks!

@LyricZhao commented on GitHub (Feb 26, 2025): Seems you don't have sufficient privilege for CUPTI profiling, which is important for microsecond-level accurate timing. Try to follow the PyTorch Kineto profile warning information to solve the issue. Thanks!

bsakaguchi referenced this issue

2026-10-26 21:18:19 +00:00

[PR #470] [CLOSED] Sync/upstream dev e1f418c2 #477