deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #477] cannot run internode test on 2 Node setup , with each having Two GPU - num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode' #307

New issue
Open
opened 2026-10-26 09:58:21 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:58:21 +00:00
Owner
Copy link

Originally created by @alokprasad on GitHub (Nov 3, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/477

I have below setup and what to run test_internode.py
Image

but getting below error

Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'

NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2
NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=1 python test_internode.py --num-processes=2

(venv) host@host-AS-5126GS-TNRT:~/alok/deepep/DeepEP$ NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO cudaDriverVersion 13000
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO cudaDriverVersion 13000
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO NCCL version 2.27.5+cuda12.9
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Initialized NET plugin IB
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO DMA-BUF is available on GPU device 1
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init START
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0>
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Initialized NET plugin IB
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO DMA-BUF is available on GPU device 0
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init START
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Bootstrap timings total 0.670492 (create 0.000033, send 0.000136, recv 0.502401, ring 0.156162, delay 0.000000)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Bootstrap timings total 0.535270 (create 0.000034, send 0.000133, recv 0.000200, ring 0.018268, delay 0.000000)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO comm 0x235bc8f0 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO comm 0x45ff2410 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 00/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 01/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
host-AS-5126GS-TNRT:160540:160796 [1] NCCL INFO [Proxy Service] Device 1 CPU core 207
host-AS-5126GS-TNRT:160540:160797 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 214
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
host-AS-5126GS-TNRT:160539:160799 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97
host-AS-5126GS-TNRT:160539:160798 [0] NCCL INFO [Proxy Service] Device 0 CPU core 95
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO CC Off, workFifoBytes 1048576
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init COMPLETE
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init COMPLETE
host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 4 total 0.90 (kernels 0.12, alloc 0.10, bootstrap 0.67, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 4 total 0.80 (kernels 0.15, alloc 0.10, bootstrap 0.54, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO DMA-BUF is available on GPU device 1
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Assigned NET plugin IB to comm
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Using network IB
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO DMA-BUF is available on GPU device 0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1- Init START
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0- Init START
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO comm 0x238d0a60 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO comm 0x46853990 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 00/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 01/02 : 0 1 2 3
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO P2P Chunksize set to 131072
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
host-AS-5126GS-TNRT:160540:160806 [1] NCCL INFO [Proxy Service] Device 1 CPU core 203
host-AS-5126GS-TNRT:160540:160807 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 81
host-AS-5126GS-TNRT:160539:160808 [0] NCCL INFO [Proxy Service] Device 0 CPU core 90
host-AS-5126GS-TNRT:160539:160809 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO CC Off, workFifoBytes 1048576
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1 - Init COMPLETE
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0 - Init COMPLETE
host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Init timings - ncclCommSplit: rank 0 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02)
host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Init timings - ncclCommSplit: rank 1 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02)
[rank0]:[W1103 14:04:39.231865515 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
host-AS-5126GS-TNRT:160540:160811 [1] NCCL INFO comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160539:160813 [0] NCCL INFO comm 0x46853990 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160540:160843 [1] NCCL INFO comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE
host-AS-5126GS-TNRT:160539:160845 [0] NCCL INFO comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE
W1103 14:04:41.835000 160472 venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:174] Terminating process 160539 via signal SIGTERM
Traceback (most recent call last):
  File "/home/host/alok/deepep/DeepEP/test_internode.py", line 375, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
  File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 364, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 320, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 220, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 95, in _wrap
    fn(i, *args)
  File "/home/host/alok/deepep/DeepEP/test_internode.py", line 307, in test_loop
    buffer = deep_ep.Buffer(group,
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/host/alok/deepep/DeepEP/deep_ep/buffer.py", line 90, in __init__
    self.runtime = deep_ep_cpp.Buffer(self.rank, self.group_size, num_nvl_bytes, num_rdma_bytes, low_latency_mode, explicitly_destroy,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed: Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'
Originally created by @alokprasad on GitHub (Nov 3, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/477 I have below setup and what to run test_internode.py <img width="876" height="408" alt="Image" src="https://github.com/user-attachments/assets/06e60a9c-f421-43c3-89f3-6eb6db613130" /> but getting below error **Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode'** NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2 NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=1 python test_internode.py --num-processes=2 ``` (venv) host@host-AS-5126GS-TNRT:~/alok/deepep/DeepEP$ NCCL_DEBUG=INFO MASTER_ADDR=100.100.100.1 WORLD_SIZE=2 RANK=0 python test_internode.py --num-processes=2 host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0> host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO cudaDriverVersion 13000 host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO NCCL version 2.27.5+cuda12.9 host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO cudaDriverVersion 13000 host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Bootstrap: Using enp81s0f0np0:10.28.38.82<0> host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO NCCL version 2.27.5+cuda12.9 host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1 host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0> host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Initialized NET plugin IB host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Assigned NET plugin IB to comm host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Using network IB host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO DMA-BUF is available on GPU device 1 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init START host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO NET/IB : Using [0]rocep113s0:1/RoCE [1]mlx5_0:1/RoCE [RO]; OOB enp81s0f0np0:10.28.38.82<0> host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Initialized NET plugin IB host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Assigned NET plugin IB to comm host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Using network IB host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO DMA-BUF is available on GPU device 0 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init START host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028> host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Bootstrap timings total 0.670492 (create 0.000033, send 0.000136, recv 0.502401, ring 0.156162, delay 0.000000) host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Bootstrap timings total 0.535270 (create 0.000034, send 0.000133, recv 0.000200, ring 0.018268, delay 0.000000) host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO comm 0x235bc8f0 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO P2P Chunksize set to 131072 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO comm 0x45ff2410 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 00/02 : 0 1 2 3 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Channel 01/02 : 0 1 2 3 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO P2P Chunksize set to 131072 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. host-AS-5126GS-TNRT:160540:160796 [1] NCCL INFO [Proxy Service] Device 1 CPU core 207 host-AS-5126GS-TNRT:160540:160797 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 214 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 host-AS-5126GS-TNRT:160539:160799 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97 host-AS-5126GS-TNRT:160539:160798 [0] NCCL INFO [Proxy Service] Device 0 CPU core 95 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO CC Off, workFifoBytes 1048576 host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO ncclCommInitRankConfig comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 commId 0xfb702ae9b623d36f - Init COMPLETE host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO ncclCommInitRankConfig comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 commId 0xfb702ae9b623d36f - Init COMPLETE host-AS-5126GS-TNRT:160540:160786 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 4 total 0.90 (kernels 0.12, alloc 0.10, bootstrap 0.67, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00) host-AS-5126GS-TNRT:160539:160787 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 4 total 0.80 (kernels 0.15, alloc 0.10, bootstrap 0.54, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.00) host-AS-5126GS-TNRT:160540:160540 [1] NCCL INFO Comm config Blocking set to 1 host-AS-5126GS-TNRT:160539:160539 [0] NCCL INFO Comm config Blocking set to 1 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Assigned NET plugin IB to comm host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Using network IB host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO DMA-BUF is available on GPU device 1 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Assigned NET plugin IB to comm host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Using network IB host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO DMA-BUF is available on GPU device 0 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1- Init START host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0- Init START host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Setting affinity for GPU 1 to 64-127,192-238,240-255 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Setting affinity for GPU 0 to 64-127,192-238,240-255 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO comm 0x238d0a60 rank 1 nRanks 4 nNodes 2 localRanks 2 localRank 1 MNNVL 0 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO P2P Chunksize set to 131072 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO comm 0x46853990 rank 0 nRanks 4 nNodes 2 localRanks 2 localRank 0 MNNVL 0 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 00/02 : 0 1 2 3 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Channel 01/02 : 0 1 2 3 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Trees [0] 1/2/-1->0->-1 [1] 1/-1/-1->0->2 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO P2P Chunksize set to 131072 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 host-AS-5126GS-TNRT:160540:160806 [1] NCCL INFO [Proxy Service] Device 1 CPU core 203 host-AS-5126GS-TNRT:160540:160807 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 81 host-AS-5126GS-TNRT:160539:160808 [0] NCCL INFO [Proxy Service] Device 0 CPU core 90 host-AS-5126GS-TNRT:160539:160809 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 97 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO CC Off, workFifoBytes 1048576 host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO ncclCommSplit comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId f1000 parent 0x235bc8f0 splitCount 1 color 2003953581 key 1 - Init COMPLETE host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO ncclCommSplit comm 0x46853990 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId e1000 parent 0x45ff2410 splitCount 1 color 2003953581 key 0 - Init COMPLETE host-AS-5126GS-TNRT:160539:160804 [0] NCCL INFO Init timings - ncclCommSplit: rank 0 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02) host-AS-5126GS-TNRT:160540:160802 [1] NCCL INFO Init timings - ncclCommSplit: rank 1 nranks 4 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.01, topo 0.01, graphs 0.00, connections 0.00, rest 0.02) [rank0]:[W1103 14:04:39.231865515 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) host-AS-5126GS-TNRT:160540:160811 [1] NCCL INFO comm 0x238d0a60 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE host-AS-5126GS-TNRT:160539:160813 [0] NCCL INFO comm 0x46853990 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE host-AS-5126GS-TNRT:160540:160843 [1] NCCL INFO comm 0x235bc8f0 rank 1 nranks 4 cudaDev 1 busId f1000 - Abort COMPLETE host-AS-5126GS-TNRT:160539:160845 [0] NCCL INFO comm 0x45ff2410 rank 0 nranks 4 cudaDev 0 busId e1000 - Abort COMPLETE W1103 14:04:41.835000 160472 venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:174] Terminating process 160539 via signal SIGTERM Traceback (most recent call last): File "/home/host/alok/deepep/DeepEP/test_internode.py", line 375, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes) File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 364, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 320, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 220, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 1 terminated with the following error: Traceback (most recent call last): File "/home/host/alok/deepep/DeepEP/venv/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 95, in _wrap fn(i, *args) File "/home/host/alok/deepep/DeepEP/test_internode.py", line 307, in test_loop buffer = deep_ep.Buffer(group, ^^^^^^^^^^^^^^^^^^^^^ File "/home/host/alok/deepep/DeepEP/deep_ep/buffer.py", line 90, in __init__ self.runtime = deep_ep_cpp.Buffer(self.rank, self.group_size, num_nvl_bytes, num_rdma_bytes, low_latency_mode, explicitly_destroy, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Failed: Assertion error /home/host/alok/deepep/DeepEP/csrc/deep_ep.cpp:48 'num_ranks > NUM_MAX_NVL_PEERS or low_latency_mode' ```
bsakaguchi commented 2026-10-26 09:58:22 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Nov 6, 2025):

Normal kernels currently do not support system configurations other than 1 node with 8 GPUs.

<!-- gh-comment-id:3494939626 --> @sphish commented on GitHub (Nov 6, 2025): Normal kernels currently do not support system configurations other than 1 node with 8 GPUs.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#307
No description provided.