deepseek-ai/DeepSeek-MoE
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-MoE.git synced 2026-11-05 00:57:43 +00:00
Code Issues 24 Projects Packages Wiki Activity

[GH-ISSUE #47] CUDA error: unspecified launch failure #42

New issue
Closed
opened 2026-10-26 21:02:00 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:02:00 +00:00
Owner
Copy link

Originally created by @SeekPoint on GitHub (Apr 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/47

[rank1]:[E410 01:50:33.889423318 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ef4bb3161b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7ef4bb2bfa76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7ef4bb404918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7ef4687f1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7ef4687fe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x7ef468800557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7ef4688016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7ef4bb4815c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7ef4bc494ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7ef4bc526850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank3]:[E410 01:50:33.892242694 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x71764e6bf1b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x71764e668a76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x71764e7ad918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7175fbbf1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7175fbbfe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x7175fbc00557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7175fbc016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x71764e82a5c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x71764f894ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x71764f926850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank2]:[E410 01:50:33.892851226 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x72a4ec1041b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x72a4ec0ada76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x72a4ec1f2918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x72a4995f1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x72a4995fe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x72a499600557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x72a4996016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x72a4ec26f5c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x72a4ed294ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x72a4ed326850 in /lib/x86_64-linux-gnu/libc.so.6)

Originally created by @SeekPoint on GitHub (Apr 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-MoE/issues/47 [rank1]:[E410 01:50:33.889423318 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ef4bb3161b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7ef4bb2bfa76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7ef4bb404918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7ef4687f1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7ef4687fe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x7ef468800557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7ef4688016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #7: <unknown function> + 0x145c0 (0x7ef4bb4815c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #8: <unknown function> + 0x94ac3 (0x7ef4bc494ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #9: <unknown function> + 0x126850 (0x7ef4bc526850 in /lib/x86_64-linux-gnu/libc.so.6) [rank3]:[E410 01:50:33.892242694 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x71764e6bf1b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x71764e668a76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x71764e7ad918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7175fbbf1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7175fbbfe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x7175fbc00557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7175fbc016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #7: <unknown function> + 0x145c0 (0x71764e82a5c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #8: <unknown function> + 0x94ac3 (0x71764f894ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #9: <unknown function> + 0x126850 (0x71764f926850 in /lib/x86_64-linux-gnu/libc.so.6) [rank2]:[E410 01:50:33.892851226 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x72a4ec1041b6 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x72a4ec0ada76 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x72a4ec1f2918 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libc10_cuda.so) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x72a4995f1556 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x72a4995fe8c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x617 (0x72a499600557 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x72a4996016ed in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #7: <unknown function> + 0x145c0 (0x72a4ec26f5c0 in /home/amd00/anaconda3/envs/DeepSeek-MoE/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #8: <unknown function> + 0x94ac3 (0x72a4ed294ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #9: <unknown function> + 0x126850 (0x72a4ed326850 in /lib/x86_64-linux-gnu/libc.so.6)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

  
WIP
No labels
pull-request
WIP
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-MoE#42
No description provided.