deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[GH-ISSUE #137] Bug: ImportError: /home/tiger/.local/lib/python3.12/site-packages/deep_gemm_cpp.cpython-312-x86_64-linux-gnu.so: undefined symbol: cudaLibraryGetKernel, version libcudart.so.12 #77

New issue
Closed
opened 2026-10-26 21:11:58 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 21:11:58 +00:00
Owner
Copy link

Originally created by @imxKyrie on GitHub (Jul 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/137

I have issues using the latest repository, whether development or installation, running

python tests/test_core.py

Traceback (most recent call last):
  File "/opt/tiger/h20/third-party/DeepGEMM/tests/test_core.py", line 6, in <module>
    import deep_gemm
  File "/opt/tiger/h20/third-party/DeepGEMM/deep_gemm/__init__.py", line 16, in <module>
    import deep_gemm_cpp
ImportError: /opt/tiger/h20/third-party/DeepGEMM/deep_gemm_cpp.cpython-312-x86_64-linux-gnu.so: undefined symbol: cudaLibraryGetKernel, version libcudart.so.12

But before that, I would not encounter this problem using the old repository, that only has hopper architecture code. Cuda version is 12.9. How should I solve this problem?

Originally created by @imxKyrie on GitHub (Jul 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/137 I have issues using the latest repository, whether development or installation, running ``` python tests/test_core.py ``` ``` Traceback (most recent call last): File "/opt/tiger/h20/third-party/DeepGEMM/tests/test_core.py", line 6, in <module> import deep_gemm File "/opt/tiger/h20/third-party/DeepGEMM/deep_gemm/__init__.py", line 16, in <module> import deep_gemm_cpp ImportError: /opt/tiger/h20/third-party/DeepGEMM/deep_gemm_cpp.cpython-312-x86_64-linux-gnu.so: undefined symbol: cudaLibraryGetKernel, version libcudart.so.12 ``` But before that, I would not encounter this problem using the old repository, that only has hopper architecture code. Cuda version is 12.9. How should I solve this problem?
bsakaguchi commented 2026-10-26 21:12:00 +00:00
Author
Owner
Copy link

@lucifer1004 commented on GitHub (Jul 24, 2025):

Your driver version might be too old. The cuda-compat package should help.

<!-- gh-comment-id:3113085629 --> @lucifer1004 commented on GitHub (Jul 24, 2025): Your driver version might be too old. The `cuda-compat` package should help.
bsakaguchi commented 2026-10-26 21:12:00 +00:00
Author
Owner
Copy link

@imxKyrie commented on GitHub (Jul 24, 2025):

@lucifer1004 I tried to install cuda-compat, it prompts that it is already the latest version cuda-compat-12-9 is already the newest version (575.51.03-0ubuntu1). And driver version is 535.161.08. But it will still report an error

<!-- gh-comment-id:3113372954 --> @imxKyrie commented on GitHub (Jul 24, 2025): @lucifer1004 I tried to install cuda-compat, it prompts that it is already the latest version `cuda-compat-12-9 is already the newest version (575.51.03-0ubuntu1)`. And driver version is 535.161.08. But it will still report an error
bsakaguchi commented 2026-10-26 21:12:01 +00:00
Author
Owner
Copy link

@Chtholly-Boss commented on GitHub (Jul 26, 2025):

Hi, I have encountered the same problem.
It seems that the problem is when loading deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so, the system load libcudart.so in /DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12, you can run strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart" to check this.
DeepGEMM requires CUDA12.8 or higher, but torch use 12.6 for now.
You can export the LD_PRELOAD variable to load your system cuda library like export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12.

<!-- gh-comment-id:3121839769 --> @Chtholly-Boss commented on GitHub (Jul 26, 2025): Hi, I have encountered the same problem. It seems that the problem is when loading `deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so`, the system load `libcudart.so` in `/DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12`, you can run `strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart"` to check this. DeepGEMM requires CUDA12.8 or higher, but torch use 12.6 for now. You can export the `LD_PRELOAD` variable to load your system cuda library like `export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12`.
bsakaguchi commented 2026-10-26 21:12:01 +00:00
Author
Owner
Copy link

@Yi-sir commented on GitHub (Aug 14, 2025):

Hi, I have encountered the same problem.你好，我也遇到了同样的问题。 It seems that the problem is when loading deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so, the system load libcudart.so in /DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12, you can run strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart" to check this.问题似乎出现在加载 deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so 时，系统在 /DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12 中加载了 libcudart.so ，你可以运行 strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart" 来检查这一点。 DeepGEMM requires CUDA12.8 or higher, but torch use 12.6 for now.DeepGEMM 需要 CUDA12.8 或更高版本，但当前 torch 使用的是 12.6 版本。 You can export the LD_PRELOAD variable to load your system cuda library like export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12.你可以导出 LD_PRELOAD 变量来加载系统 cuda 库，例如 export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12 。

It works, thank you!

<!-- gh-comment-id:3187879552 --> @Yi-sir commented on GitHub (Aug 14, 2025): > Hi, I have encountered the same problem.你好，我也遇到了同样的问题。 It seems that the problem is when loading `deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so`, the system load `libcudart.so` in `/DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12`, you can run `strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart"` to check this.问题似乎出现在加载 `deep_gemm_cpp.cpython-310-x86_64-linux-gnu.so` 时，系统在 `/DeepGEMM/.venv/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12` 中加载了 `libcudart.so` ，你可以运行 `strace -e openat,open python3 /root/DeepGEMM/tests/test_layout.py 2>&1 | grep "cudart"` 来检查这一点。 DeepGEMM requires CUDA12.8 or higher, but torch use 12.6 for now.DeepGEMM 需要 CUDA12.8 或更高版本，但当前 torch 使用的是 12.6 版本。 You can export the `LD_PRELOAD` variable to load your system cuda library like `export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12`.你可以导出 `LD_PRELOAD` 变量来加载系统 cuda 库，例如 `export LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12` 。 It works, thank you!
bsakaguchi commented 2026-10-26 21:12:01 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Aug 28, 2025):

Fixed in github.com/deepseek-ai/DeepGEMM@ea9c5d9270. Now we are using driver API by default.

<!-- gh-comment-id:3231054326 --> @LyricZhao commented on GitHub (Aug 28, 2025): Fixed in https://github.com/deepseek-ai/DeepGEMM/commit/ea9c5d9270226c5dd7a577c212e9ea385f6ef048. Now we are using driver API by default.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#77
No description provided.