deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #224] named symbol not found on B200 #173

New issue
Closed
opened 2026-10-26 09:55:47 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:55:47 +00:00
Owner
Copy link

Originally created by @yewentao256 on GitHub (Jun 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/224

When installing on B200m, seems that it is still sm_90 structure

running bdist_egg
running egg_info
writing deep_ep.egg-info/PKG-INFO
writing dependency_links to deep_ep.egg-info/dependency_links.txt
writing top-level names to deep_ep.egg-info/top_level.txt
adding license file 'LICENSE'
writing manifest file 'deep_ep.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-cpython-312/deep_ep
copying deep_ep/__init__.py -> build/lib.linux-x86_64-cpython-312/deep_ep
copying deep_ep/buffer.py -> build/lib.linux-x86_64-cpython-312/deep_ep
copying deep_ep/utils.py -> build/lib.linux-x86_64-cpython-312/deep_ep
running build_ext
/home/wentao/.wentao_env/lib/python3.12/site-packages/torch/utils/cpp_extension.py:490: UserWarning: There are no x86_64-linux-gnu-g++ version bounds defined for CUDA version 12.8
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'deep_ep_cpp' extension
creating /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc
creating /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels
Emitting ninja build file /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
...
[7/7] /usr/local/cuda/bin/nvcc /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/deep_ep.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode_ll.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/intranode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/layout.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/runtime.o -o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/dlink.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -dlink -L/opt/deepep/nvshmem/lib -lnvshmem -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1016" -DTORCH_EXTENSION_NAME=deep_ep_cpp -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_90,code=sm_90
x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/deep_ep.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode_ll.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/intranode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/layout.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/runtime.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/dlink.o -L/opt/deepep/nvshmem/lib -L/home/wentao/.wentao_env/lib/python3.12/site-packages/torch/lib -L/usr/local/cuda/lib64 -L/usr/lib/x86_64-linux-gnu -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-cpython-312/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so -l:libnvshmem.a -l:nvshmem_bootstrap_uid.so -Wl,-rpath,/opt/deepep/nvshmem/lib
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-cpython-312/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/deep_ep
copying build/lib.linux-x86_64-cpython-312/deep_ep/__init__.py -> build/bdist.linux-x86_64/egg/deep_ep
copying build/lib.linux-x86_64-cpython-312/deep_ep/buffer.py -> build/bdist.linux-x86_64/egg/deep_ep
copying build/lib.linux-x86_64-cpython-312/deep_ep/utils.py -> build/bdist.linux-x86_64/egg/deep_ep
byte-compiling build/bdist.linux-x86_64/egg/deep_ep/__init__.py to __init__.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/deep_ep/buffer.py to buffer.cpython-312.pyc
byte-compiling build/bdist.linux-x86_64/egg/deep_ep/utils.py to utils.cpython-312.pyc
creating stub loader for deep_ep_cpp.cpython-312-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/deep_ep_cpp.py to deep_ep_cpp.cpython-312.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying deep_ep.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying deep_ep.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying deep_ep.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying deep_ep.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.deep_ep_cpp.cpython-312: module references __file__
creating 'dist/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg
creating /home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg
Extracting deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg to /home/wentao/.wentao_env/lib/python3.12/site-packages
Adding deep-ep 1.1.0+9d4f7ef to easy-install.pth file

Installed /home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg
Processing dependencies for deep-ep==1.1.0+9d4f7ef
Finished processing dependencies for deep-ep==1.1.0+9d4f7ef

After the install, and meet this issue when try to run pytest test_deepep_moe.py in vllm

E       -- Process 0 terminated with the following error:
E       Traceback (most recent call last):
E         File "/home/wentao/.wentao_env/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
E           fn(i, *args)
E         File "/home/wentao/vllm-source/tests/kernels/moe/deepep_utils.py", line 62, in _worker_parallel_launch
E           worker(
E         File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 349, in _deep_ep_moe
E           deepep_combined = deep_ep_moe_impl(
E                             ^^^^^^^^^^^^^^^^^
E         File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 255, in deep_ep_moe_impl
E           process_chunk(chunk_start,
E         File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 224, in process_chunk
E           out = mk.forward(hidden_states=rank_tokens_chunk,
E                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E         File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 381, in forward
E           _expert_topk_weights) = self.prepare_finalize.prepare(
E                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E         File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", line 175, in prepare
E           expert_topk_weights) = self._do_dispatch(
E                                  ^^^^^^^^^^^^^^^^^^
E         File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", line 73, in _do_dispatch
E           is_token_in_rank, event) = self.buffer.get_dispatch_layout(
E                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E         File "/home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 250, in get_dispatch_layout
E           self.runtime.get_dispatch_layout(topk_idx, num_experts, getattr(previous_event, 'event', None),
E       RuntimeError: Failed: CUDA error /home/wentao/DeepEP/csrc/kernels/layout.cu:129 'named symbol not found'

I saw some issues before, seems not a problem of myself. Could you give us some general guidance to figure it out?

Originally created by @yewentao256 on GitHub (Jun 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/224 When installing on B200m, seems that it is still sm_90 structure ```bash running bdist_egg running egg_info writing deep_ep.egg-info/PKG-INFO writing dependency_links to deep_ep.egg-info/dependency_links.txt writing top-level names to deep_ep.egg-info/top_level.txt adding license file 'LICENSE' writing manifest file 'deep_ep.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build/lib.linux-x86_64-cpython-312/deep_ep copying deep_ep/__init__.py -> build/lib.linux-x86_64-cpython-312/deep_ep copying deep_ep/buffer.py -> build/lib.linux-x86_64-cpython-312/deep_ep copying deep_ep/utils.py -> build/lib.linux-x86_64-cpython-312/deep_ep running build_ext /home/wentao/.wentao_env/lib/python3.12/site-packages/torch/utils/cpp_extension.py:490: UserWarning: There are no x86_64-linux-gnu-g++ version bounds defined for CUDA version 12.8 warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}') building 'deep_ep_cpp' extension creating /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc creating /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels Emitting ninja build file /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ... [7/7] /usr/local/cuda/bin/nvcc /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/deep_ep.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode_ll.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/intranode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/layout.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/runtime.o -o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/dlink.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -dlink -L/opt/deepep/nvshmem/lib -lnvshmem -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1016" -DTORCH_EXTENSION_NAME=deep_ep_cpp -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_90,code=sm_90 x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/deep_ep.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/internode_ll.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/intranode.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/layout.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/kernels/runtime.o /home/wentao/DeepEP/build/temp.linux-x86_64-cpython-312/csrc/dlink.o -L/opt/deepep/nvshmem/lib -L/home/wentao/.wentao_env/lib/python3.12/site-packages/torch/lib -L/usr/local/cuda/lib64 -L/usr/lib/x86_64-linux-gnu -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-cpython-312/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so -l:libnvshmem.a -l:nvshmem_bootstrap_uid.so -Wl,-rpath,/opt/deepep/nvshmem/lib creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-cpython-312/deep_ep_cpp.cpython-312-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/deep_ep copying build/lib.linux-x86_64-cpython-312/deep_ep/__init__.py -> build/bdist.linux-x86_64/egg/deep_ep copying build/lib.linux-x86_64-cpython-312/deep_ep/buffer.py -> build/bdist.linux-x86_64/egg/deep_ep copying build/lib.linux-x86_64-cpython-312/deep_ep/utils.py -> build/bdist.linux-x86_64/egg/deep_ep byte-compiling build/bdist.linux-x86_64/egg/deep_ep/__init__.py to __init__.cpython-312.pyc byte-compiling build/bdist.linux-x86_64/egg/deep_ep/buffer.py to buffer.cpython-312.pyc byte-compiling build/bdist.linux-x86_64/egg/deep_ep/utils.py to utils.cpython-312.pyc creating stub loader for deep_ep_cpp.cpython-312-x86_64-linux-gnu.so byte-compiling build/bdist.linux-x86_64/egg/deep_ep_cpp.py to deep_ep_cpp.cpython-312.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying deep_ep.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying deep_ep.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying deep_ep.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying deep_ep.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... __pycache__.deep_ep_cpp.cpython-312: module references __file__ creating 'dist/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg creating /home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg Extracting deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg to /home/wentao/.wentao_env/lib/python3.12/site-packages Adding deep-ep 1.1.0+9d4f7ef to easy-install.pth file Installed /home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg Processing dependencies for deep-ep==1.1.0+9d4f7ef Finished processing dependencies for deep-ep==1.1.0+9d4f7ef ``` After the install, and meet this issue when try to run `pytest test_deepep_moe.py` in vllm ```bash E -- Process 0 terminated with the following error: E Traceback (most recent call last): E File "/home/wentao/.wentao_env/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap E fn(i, *args) E File "/home/wentao/vllm-source/tests/kernels/moe/deepep_utils.py", line 62, in _worker_parallel_launch E worker( E File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 349, in _deep_ep_moe E deepep_combined = deep_ep_moe_impl( E ^^^^^^^^^^^^^^^^^ E File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 255, in deep_ep_moe_impl E process_chunk(chunk_start, E File "/home/wentao/vllm-source/tests/kernels/moe/test_deepep_moe.py", line 224, in process_chunk E out = mk.forward(hidden_states=rank_tokens_chunk, E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 381, in forward E _expert_topk_weights) = self.prepare_finalize.prepare( E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", line 175, in prepare E expert_topk_weights) = self._do_dispatch( E ^^^^^^^^^^^^^^^^^^ E File "/home/wentao/vllm-source/vllm/model_executor/layers/fused_moe/deepep_ht_prepare_finalize.py", line 73, in _do_dispatch E is_token_in_rank, event) = self.buffer.get_dispatch_layout( E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/wentao/.wentao_env/lib/python3.12/site-packages/deep_ep-1.1.0+9d4f7ef-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 250, in get_dispatch_layout E self.runtime.get_dispatch_layout(topk_idx, num_experts, getattr(previous_event, 'event', None), E RuntimeError: Failed: CUDA error /home/wentao/DeepEP/csrc/kernels/layout.cu:129 'named symbol not found' ``` I saw some issues before, seems not a problem of myself. Could you give us some general guidance to figure it out?
bsakaguchi commented 2026-10-26 09:55:50 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 18, 2025):

https://github.com/deepseek-ai/DeepEP/issues/104
https://github.com/deepseek-ai/DeepEP/issues/87

<!-- gh-comment-id:2985737612 --> @yewentao256 commented on GitHub (Jun 18, 2025): https://github.com/deepseek-ai/DeepEP/issues/104 https://github.com/deepseek-ai/DeepEP/issues/87
bsakaguchi commented 2026-10-26 09:55:51 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 18, 2025):

Fixed, TORCH_CUDA_ARCH_LIST=10.0 NVSHMEM_DIR=/opt/deepep/nvshmem/ python setup.py install
Simply add TORCH_CUDA_ARCH_LIST=10.0 to the arch list

<!-- gh-comment-id:2985783610 --> @yewentao256 commented on GitHub (Jun 18, 2025): Fixed, `TORCH_CUDA_ARCH_LIST=10.0 NVSHMEM_DIR=/opt/deepep/nvshmem/ python setup.py install` Simply add TORCH_CUDA_ARCH_LIST=10.0 to the arch list
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#173
No description provided.