mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #53] Error Accept failed No space left on device when test 2 * 8 H800 test_internode #49
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#49
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Huixxi on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/53
proxy.cc:1587 NCCL WARN [Service thread] Accept failed No space left on device
misc/shmutils.cc:87 NCCL WARN Error: failed to extend /dev/shm/nccl-nyj24T to 10486116 bytes, error: No space left on device (28)
terminate called after throwing an instance of 'terminate called after throwing an instance of 'terminate called after throwing an instance of 'EPExceptionEPExceptionEPException'
'
'
what(): what(): Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered' what():
Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /home/logs/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'
@Huixxi commented on GitHub (Mar 7, 2025):
sudo mount -o remount,size=8G /dev/shm # 调整为更大值如16G