[GH-ISSUE #46] how to use nsight compute to get the kernel performance？ #30

New issue

Closed

opened 2026-10-26 21:10:52 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:10:52 +00:00

Owner

Originally created by @youzjuer on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/46

i use command ncu --set full -o deepgemm python tests/test_core.py
but report error:
==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM
Traceback (most recent call last):
File "/mnt/workspace/xxx/DeepGEMM-main/tests/test_core.py", line 167, in
test_gemm()
File "/mnt/workspace/xxx/DeepGEMM-main/tests/test_core.py", line 80, in test_gemm
deep_gemm.gemm_fp8_fp8_bf16_nt(x_fp8, y_fp8, out)
File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit_kernels/gemm.py", line 162, in gemm_fp8_fp8_bf16_nt
runtime = jit_tuner.compile_and_tune(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit_kernels/tuner.py", line 40, in compile_and_tune
kernels.append((build(name, arg_defs, code), tuned_keys))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit/compiler.py", line 110, in build
enable_sass_opt = get_nvcc_compiler()[1] <= '12.8' and int(os.getenv('DG_DISABLE_FFMA_INTERLEAVE', 0)) == 0
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit/compiler.py", line 57, in get_nvcc_compiler
version = match.group(1)
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
==PROF== Disconnected from process 25948
==ERROR== The application returned an error code (1).

Originally created by @youzjuer on GitHub (Mar 7, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/46 i use command ncu --set full -o deepgemm python tests/test_core.py but report error: ==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM Traceback (most recent call last): File "/mnt/workspace/xxx/DeepGEMM-main/tests/test_core.py", line 167, in <module> test_gemm() File "/mnt/workspace/xxx/DeepGEMM-main/tests/test_core.py", line 80, in test_gemm deep_gemm.gemm_fp8_fp8_bf16_nt(x_fp8, y_fp8, out) File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit_kernels/gemm.py", line 162, in gemm_fp8_fp8_bf16_nt runtime = jit_tuner.compile_and_tune( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit_kernels/tuner.py", line 40, in compile_and_tune kernels.append((build(name, arg_defs, code), tuned_keys)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit/compiler.py", line 110, in build enable_sass_opt = get_nvcc_compiler()[1] <= '12.8' and int(os.getenv('DG_DISABLE_FFMA_INTERLEAVE', 0)) == 0 ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/deep_gemm/jit/compiler.py", line 57, in get_nvcc_compiler version = match.group(1) ^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'group' ==PROF== Disconnected from process 25948 ==ERROR== The application returned an error code (1).

bsakaguchi closed this issue

2026-10-26 21:10:52 +00:00

bsakaguchi commented

2026-10-26 21:10:54 +00:00

Author

Owner

@soundOfDestiny commented on GitHub (Mar 7, 2025):

DeepGEMM cannot find the version of your NVCC.
You can print os.popen(f'{path} --version').read() in code before version = match.group(1), or simply run 'nvcc --version' to debug.

@soundOfDestiny commented on GitHub (Mar 7, 2025): DeepGEMM cannot find the version of your NVCC. You can print `os.popen(f'{path} --version').read()` in code before `version = match.group(1)`, or simply run 'nvcc --version' to debug.

bsakaguchi commented

2026-10-26 21:10:54 +00:00

Author

Owner

@sazczmh commented on GitHub (Apr 9, 2025):

The error message itself contains the solution.

==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM

To allow access for any user, create a file with the .conf extension containing options nvidia NVreg_RestrictProfilingToAdminUsers=0 in /etc/modprobe.d.

@sazczmh commented on GitHub (Apr 9, 2025): The error message itself contains the solution. > ==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM <img width="1307" alt="Image" src="https://github.com/user-attachments/assets/a410943c-368a-4d36-a849-e20f9b2c61aa" /> To allow access for any user, create a file with the .conf extension containing options **nvidia NVreg_RestrictProfilingToAdminUsers=0** in **/etc/modprobe.d**.

bsakaguchi referenced this issue

2026-10-26 21:14:57 +00:00

[PR #30] [CLOSED] Specify uint8_t as enum class size #214