deep_ep.cpp:629 'num_tokens_per_rdma_rank.has_value()' #111

New issue

Closed

opened 2026-10-26 09:54:28 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:54:28 +00:00

Owner

Originally created by @yushuai1 on GitHub (Apr 16, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/125

2*8 gpu h200
启动命令
NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=0 python tests/test_internode.py
NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=1 python tests/test_internode.py

Traceback (most recent call last):
File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 222, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 5 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 210, in test_loop
test_main(i, local_rank, num_ranks, rank, buffer, group)
File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 93, in test_main
recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/yushuai/DeepEP/deep_ep/buffer.py", line 292, in dispatch
return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/yushuai/DeepEP/deep_ep/buffer.py", line 400, in internode_dispatch
recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed: Assertion error /mnt/yushuai/DeepEP/csrc/deep_ep.cpp:629 'num_tokens_per_rdma_rank.has_value()'

这是NCCL兼容性造成的吗？

python：12
torch：2.6
cuda：12.6
nccl：nvidia-nccl-cu12 2.21.5

Originally created by @yushuai1 on GitHub (Apr 16, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/125 2*8 gpu h200 启动命令 NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=0 python tests/test_internode.py NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=1 python tests/test_internode.py Traceback (most recent call last): File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 222, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 5 terminated with the following error: Traceback (most recent call last): File "/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 210, in test_loop test_main(i, local_rank, num_ranks, rank, buffer, group) File "/mnt/yushuai/DeepEP/tests/test_intranode.py", line 93, in test_main recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/yushuai/DeepEP/deep_ep/buffer.py", line 292, in dispatch return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/yushuai/DeepEP/deep_ep/buffer.py", line 400, in internode_dispatch recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Failed: Assertion error /mnt/yushuai/DeepEP/csrc/deep_ep.cpp:629 'num_tokens_per_rdma_rank.has_value()' 这是NCCL兼容性造成的吗？ python：12 torch：2.6 cuda：12.6 nccl：nvidia-nccl-cu12 2.21.5

bsakaguchi closed this issue

2026-10-26 09:54:28 +00:00

bsakaguchi commented

2026-10-26 09:54:29 +00:00

Author

Owner

@yushuai1 commented on GitHub (Apr 18, 2025):

环境没装好

@yushuai1 commented on GitHub (Apr 18, 2025): 环境没装好

bsakaguchi commented

2026-10-26 09:54:30 +00:00

Author