deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #87] "RuntimeError: Failed: CUDA error /opt/DeepEP/csrc/kernels/internode.cu:133 ' named symbol not found" in container when running test_intranode.py #81

New issue
Closed
opened 2026-10-26 09:53:56 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:53:56 +00:00
Owner
Copy link

Originally created by @yuwenjingsei on GitHub (Mar 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/87

Hi all,

I have installed gdrcopy and nvshmem in container (with gdrcopy_copybw, nvshmem-info -a verified), but when I run

python3 tests/test_intranode.py

I encounter the following errors:

[config] num_tokens=4096, hidden=7168, num_topk=8
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79673 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79674 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79675 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79676 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79678 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79679 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79680 via signal SIGTERM
Traceback (most recent call last):
  File "/opt/DeepEP/tests/test_intranode.py", line 225, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 281, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 237, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 188, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 75, in _wrap
    fn(i, *args)
  File "/opt/DeepEP/tests/test_intranode.py", line 213, in test_loop
    test_main(i, local_rank, num_ranks, rank, buffer, group)
  File "/opt/DeepEP/tests/test_intranode.py", line 58, in test_main
    buffer.get_dispatch_layout(topk_idx, num_experts)
  File "/usr/local/lib/python3.10/dist-packages/deep_ep-1.0.0+c4b8ffc-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 227, in get_dispatch_layout
    self.runtime.get_dispatch_layout(topk_idx, num_experts, getattr(previous_event, 'event', None),
RuntimeError: Failed: CUDA error /opt/DeepEP/csrc/kernels/internode.cu:133 'named symbol not found'

Any hints on how to solve the problem?

Originally created by @yuwenjingsei on GitHub (Mar 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/87 Hi all, I have installed gdrcopy and nvshmem in container (with gdrcopy_copybw, nvshmem-info -a verified), but when I run `python3 tests/test_intranode.py` I encounter the following errors: ``` [config] num_tokens=4096, hidden=7168, num_topk=8 terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found' W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79673 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79674 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79675 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79676 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79678 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79679 via signal SIGTERM W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79680 via signal SIGTERM Traceback (most recent call last): File "/opt/DeepEP/tests/test_intranode.py", line 225, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 281, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 237, in start_processes while not context.join(): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 188, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 75, in _wrap fn(i, *args) File "/opt/DeepEP/tests/test_intranode.py", line 213, in test_loop test_main(i, local_rank, num_ranks, rank, buffer, group) File "/opt/DeepEP/tests/test_intranode.py", line 58, in test_main buffer.get_dispatch_layout(topk_idx, num_experts) File "/usr/local/lib/python3.10/dist-packages/deep_ep-1.0.0+c4b8ffc-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 227, in get_dispatch_layout self.runtime.get_dispatch_layout(topk_idx, num_experts, getattr(previous_event, 'event', None), RuntimeError: Failed: CUDA error /opt/DeepEP/csrc/kernels/internode.cu:133 'named symbol not found' ``` Any hints on how to solve the problem?
bsakaguchi commented 2026-10-26 09:53:57 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 25, 2025):

It seems something wrong with the compilation/link, runtime can not find the kernel symbol. We never run into such errors, I suggest you check your compilation toolchains/compilation flags/link flags, and you may reproduce this even for any kernel but not DeepEP. Or just try to Google "named symbol not found" :)

<!-- gh-comment-id:2749816211 --> @LyricZhao commented on GitHub (Mar 25, 2025): It seems something wrong with the compilation/link, runtime can not find the kernel symbol. We never run into such errors, I suggest you check your compilation toolchains/compilation flags/link flags, and you may reproduce this even for any kernel but not DeepEP. Or just try to Google "named symbol not found" :)
bsakaguchi commented 2026-10-26 09:53:57 +00:00
Author
Owner
Copy link

@yuwenjingsei commented on GitHub (Mar 25, 2025):

turns out modifying cuda arch value solved this problem.
I was running tests on A100 and saw in setup.py line 12, "only support Hopper arch ...",
and in line 13, 9.0 is the cuda arch value for H series NV gpu (H100/H800/H200).

 12     # TODO: currently, we only support Hopper architecture, we may add Ampere support later
 13     os.environ['TORCH_CUDA_ARCH_LIST'] = '9.0'

after replace 9.0 with 8.0, which represents A series NV gpu arch, intranode.py test run successfully,
although not so sure whether "only support Hopper arch", "may add Ampere support later" will effect the performance on A gpus

<!-- gh-comment-id:2750351663 --> @yuwenjingsei commented on GitHub (Mar 25, 2025): turns out modifying cuda arch value solved this problem. I was running tests on A100 and saw in setup.py line 12, "only support Hopper arch ...", and in line 13, 9.0 is the cuda arch value for H series NV gpu (H100/H800/H200). ``` 12 # TODO: currently, we only support Hopper architecture, we may add Ampere support later 13 os.environ['TORCH_CUDA_ARCH_LIST'] = '9.0' ``` after replace 9.0 with 8.0, which represents A series NV gpu arch, intranode.py test run successfully, although not so sure whether "only support Hopper arch", "may add Ampere support later" will effect the performance on A gpus
bsakaguchi commented 2026-10-26 09:53:58 +00:00
Author
Owner
Copy link

@yewentao256 commented on GitHub (Jun 19, 2025):

https://github.com/deepseek-ai/DeepEP/issues/224#event-18220765845
Fixed and can take a look, hopefully it is helpful

<!-- gh-comment-id:2988072813 --> @yewentao256 commented on GitHub (Jun 19, 2025): https://github.com/deepseek-ai/DeepEP/issues/224#event-18220765845 Fixed and can take a look, hopefully it is helpful
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#81
No description provided.