deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #133] error: nvshmem API called before nvshmem_init #115

New issue

Open

opened 2026-10-26 09:54:34 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-26 09:54:34 +00:00

Owner

Originally created by @guodongrui-gdr on GitHub (Apr 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/133

error when run test_low_latency.py:
Allocating buffer size: 2115.111296 MB ...
/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887117 via signal SIGTERM
W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887118 via signal SIGTERM
W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887120 via signal SIGTERM
W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887121 via signal SIGTERM
Traceback (most recent call last):
File "/root/DeepEP-main/tests/test_low_latency.py", line 172, in
torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 5 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/root/DeepEP-main/tests/test_low_latency.py", line 156, in test_loop
buffer = deep_ep.Buffer(group, num_rdma_bytes=num_rdma_bytes, low_latency_mode=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/deep_ep-1.0.0+8a0ca8e-py3.11-linux-x86_64.egg/deep_ep/buffer.py", line 59, in init
dist.all_gather_object(device_ids, local_device_id, group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object
all_gather(object_size_list, local_size, group=group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather
work = group.allgather([tensor_list], [tensor])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 1 'invalid argument'

Originally created by @guodongrui-gdr on GitHub (Apr 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/133 error when run test_low_latency.py: Allocating buffer size: 2115.111296 MB ... /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887117 via signal SIGTERM W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887118 via signal SIGTERM W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887120 via signal SIGTERM W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887121 via signal SIGTERM Traceback (most recent call last): File "/root/DeepEP-main/tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 5 terminated with the following error: Traceback (most recent call last): File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/root/DeepEP-main/tests/test_low_latency.py", line 156, in test_loop buffer = deep_ep.Buffer(group, num_rdma_bytes=num_rdma_bytes, low_latency_mode=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/deep_ep-1.0.0+8a0ca8e-py3.11-linux-x86_64.egg/deep_ep/buffer.py", line 59, in __init__ dist.all_gather_object(device_ids, local_device_id, group) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object all_gather(object_size_list, local_size, group=group) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather work = group.allgather([tensor_list], [tensor]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5 ncclUnhandledCudaError: Call to CUDA function failed. Last error: Cuda failure 1 'invalid argument'

bsakaguchi commented

2026-10-26 09:54:35 +00:00

Author

Owner

@sphish commented on GitHub (Apr 25, 2025):

https://github.com/deepseek-ai/DeepEP/issues/121 Is this the same issue?

@sphish commented on GitHub (Apr 25, 2025): https://github.com/deepseek-ai/DeepEP/issues/121 Is this the same issue?

bsakaguchi commented

2026-10-26 09:54:35 +00:00

Author

Owner

@guodongrui-gdr commented on GitHub (Apr 25, 2025):

#121 Is this the same issue?

not same, i only have one node

@guodongrui-gdr commented on GitHub (Apr 25, 2025): > [#121](https://github.com/deepseek-ai/DeepEP/issues/121) Is this the same issue? not same, i only have one node

bsakaguchi commented

2026-10-26 09:54:35 +00:00

Author

Owner

@alpha-baby commented on GitHub (Apr 25, 2025):

#121 Is this the same issue?

not same, i only have one node

i have the traceback for nvshmem init. maybe this can help you you to add log for debugging.

0# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(print_trace(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x20) [0x7f1baec05240]
1# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_setup_connections(nvshmemi_state_dec*)+0x7d) [0x7f1baec0532d]
2# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_common_init(nvshmemi_state_dec*)+0x7e8) [0x7f1baeab09f8]
3# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_try_common_init(nvshmemi_state_dec*)+0x9) [0x7f1baeab20c9]
4# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemid_hostlib_init_attr+0x218) [0x7f1baeab26b8]
5# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_init_thread(int, int*, unsigned int, nvshmemx_init_attr_v1*, nvshmemi_version_t)+0x2a) [0x7f1baeba2c8a]
6# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(deep_ep::internode::init(std::vector<unsigned char, std::allocator<unsigned char> > const&, int, int, bool)+0xe6) [0x7f1baeaa40e6]

@alpha-baby commented on GitHub (Apr 25, 2025): > > [#121](https://github.com/deepseek-ai/DeepEP/issues/121) Is this the same issue? > > not same, i only have one node i have the traceback for nvshmem init. maybe this can help you you to add log for debugging. ```log 0# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(print_trace(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x20) [0x7f1baec05240] 1# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_setup_connections(nvshmemi_state_dec*)+0x7d) [0x7f1baec0532d] 2# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_common_init(nvshmemi_state_dec*)+0x7e8) [0x7f1baeab09f8] 3# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_try_common_init(nvshmemi_state_dec*)+0x9) [0x7f1baeab20c9] 4# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemid_hostlib_init_attr+0x218) [0x7f1baeab26b8] 5# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(nvshmemi_init_thread(int, int*, unsigned int, nvshmemx_init_attr_v1*, nvshmemi_version_t)+0x2a) [0x7f1baeba2c8a] 6# /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so(deep_ep::internode::init(std::vector<unsigned char, std::allocator<unsigned char> > const&, int, int, bool)+0xe6) [0x7f1baeaa40e6] ```

bsakaguchi commented

2026-10-26 09:54:35 +00:00

Author

Owner

@sphish commented on GitHub (Apr 25, 2025):

I haven't encountered this issue before, but I suspect it might be related to the CUDA driver version. Is there a way to pinpoint which CUDA call is triggering the "invalid argument" error?

@sphish commented on GitHub (Apr 25, 2025): I haven't encountered this issue before, but I suspect it might be related to the CUDA driver version. Is there a way to pinpoint which CUDA call is triggering the "invalid argument" error?

bsakaguchi commented

2026-10-26 09:54:36 +00:00

Author

Owner

@guodongrui-gdr commented on GitHub (Apr 25, 2025):

I haven't encountered this issue before, but I suspect it might be related to the CUDA driver version. Is there a way to pinpoint which CUDA call is triggering the "invalid argument" error?

I ran with NCCL_DEBUG=INFO, get output below:
Allocating buffer size: 2115.111296 MB ...
ubuntu:1153684:1153684 [0] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153684:1153684 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.21.5+cuda12.4
ubuntu:1153684:1153684 [0] NCCL INFO Comm config Blocking set to 1
ubuntu:1153691:1153691 [7] NCCL INFO cudaDriverVersion 12040
ubuntu:1153691:1153691 [7] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153691:1153691 [7] NCCL INFO Comm config Blocking set to 1
ubuntu:1153690:1153690 [6] NCCL INFO cudaDriverVersion 12040
ubuntu:1153690:1153690 [6] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153690:1153690 [6] NCCL INFO Comm config Blocking set to 1
ubuntu:1153686:1153686 [2] NCCL INFO cudaDriverVersion 12040
ubuntu:1153686:1153686 [2] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153688:1153688 [4] NCCL INFO cudaDriverVersion 12040
ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153686:1153686 [2] NCCL INFO Comm config Blocking set to 1
ubuntu:1153688:1153688 [4] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153688:1153688 [4] NCCL INFO Comm config Blocking set to 1
ubuntu:1153685:1153685 [1] NCCL INFO cudaDriverVersion 12040
ubuntu:1153685:1153685 [1] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153685:1153685 [1] NCCL INFO Comm config Blocking set to 1
ubuntu:1153687:1153687 [3] NCCL INFO cudaDriverVersion 12040
ubuntu:1153687:1153687 [3] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153687:1153687 [3] NCCL INFO Comm config Blocking set to 1
ubuntu:1153689:1153689 [5] NCCL INFO cudaDriverVersion 12040
ubuntu:1153689:1153689 [5] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0>
ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: Using internal network plugin.
ubuntu:1153689:1153689 [5] NCCL INFO Comm config Blocking set to 1
ubuntu:1153685:1154269 [1] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153685:1154269 [1] NCCL INFO Using non-device net plugin version 0
ubuntu:1153685:1154269 [1] NCCL INFO Using network IB
ubuntu:1153684:1154261 [0] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153684:1154261 [0] NCCL INFO Using non-device net plugin version 0
ubuntu:1153684:1154261 [0] NCCL INFO Using network IB
ubuntu:1153690:1154265 [6] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153686:1154266 [2] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153690:1154265 [6] NCCL INFO Using non-device net plugin version 0
ubuntu:1153690:1154265 [6] NCCL INFO Using network IB
ubuntu:1153686:1154266 [2] NCCL INFO Using non-device net plugin version 0
ubuntu:1153686:1154266 [2] NCCL INFO Using network IB
ubuntu:1153691:1154264 [7] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153691:1154264 [7] NCCL INFO Using non-device net plugin version 0
ubuntu:1153691:1154264 [7] NCCL INFO Using network IB
ubuntu:1153688:1154267 [4] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153688:1154267 [4] NCCL INFO Using non-device net plugin version 0
ubuntu:1153688:1154267 [4] NCCL INFO Using network IB
ubuntu:1153687:1154271 [3] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153687:1154271 [3] NCCL INFO Using non-device net plugin version 0
ubuntu:1153687:1154271 [3] NCCL INFO Using network IB
ubuntu:1153689:1154272 [5] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0>
ubuntu:1153689:1154272 [5] NCCL INFO Using non-device net plugin version 0
ubuntu:1153689:1154272 [5] NCCL INFO Using network IB
ubuntu:1153687:1154271 [3] NCCL INFO ncclCommInitRank comm 0x432a2a70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 5d000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153685:1154269 [1] NCCL INFO ncclCommInitRank comm 0x1fdd0320 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 2a000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153686:1154266 [2] NCCL INFO ncclCommInitRank comm 0x42155fa0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 3b000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153688:1154267 [4] NCCL INFO ncclCommInitRank comm 0x2dd19f30 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153689:1154272 [5] NCCL INFO ncclCommInitRank comm 0x38f020c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId ab000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153684:1154261 [0] NCCL INFO ncclCommInitRank comm 0x29404f40 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 19000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153691:1154264 [7] NCCL INFO ncclCommInitRank comm 0x3b6f23a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId db000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153690:1154265 [6] NCCL INFO ncclCommInitRank comm 0x45c952b0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId bb000 commId 0x3ddd6a65f9575625 - Init START
ubuntu:1153687:1154271 [3] NCCL INFO MNNVL busId 0x5d000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153691:1154264 [7] NCCL INFO MNNVL busId 0xdb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153690:1154265 [6] NCCL INFO MNNVL busId 0xbb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153685:1154269 [1] NCCL INFO MNNVL busId 0x2a000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153688:1154267 [4] NCCL INFO MNNVL busId 0x9b000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153684:1154261 [0] NCCL INFO MNNVL busId 0x19000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153686:1154266 [2] NCCL INFO MNNVL busId 0x3b000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153689:1154272 [5] NCCL INFO MNNVL busId 0xab000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
ubuntu:1153690:1154265 [6] NCCL INFO Setting affinity for GPU 6 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000
ubuntu:1153690:1154265 [6] NCCL INFO NVLS multicast support is available on dev 6
ubuntu:1153685:1154269 [1] NCCL INFO Setting affinity for GPU 1 to 3fffff,ff000000,00000000,00000000,3fffffff
ubuntu:1153685:1154269 [1] NCCL INFO NVLS multicast support is available on dev 1
ubuntu:1153684:1154261 [0] NCCL INFO Setting affinity for GPU 0 to 3fffff,ff000000,00000000,00000000,3fffffff
ubuntu:1153684:1154261 [0] NCCL INFO NVLS multicast support is available on dev 0
ubuntu:1153691:1154264 [7] NCCL INFO Setting affinity for GPU 7 to ffff,fffc0000,00000000,00000000,00ffffff,fc000000,00000000,00000000
ubuntu:1153691:1154264 [7] NCCL INFO NVLS multicast support is available on dev 7
ubuntu:1153686:1154266 [2] NCCL INFO Setting affinity for GPU 2 to 3fffff,ff000000,00000000,00000000,3fffffff
ubuntu:1153686:1154266 [2] NCCL INFO NVLS multicast support is available on dev 2
ubuntu:1153689:1154272 [5] NCCL INFO Setting affinity for GPU 5 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000
ubuntu:1153689:1154272 [5] NCCL INFO NVLS multicast support is available on dev 5
ubuntu:1153687:1154271 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff,ffc00000,00000000,00000000,0fffffff,c0000000
ubuntu:1153687:1154271 [3] NCCL INFO NVLS multicast support is available on dev 3
ubuntu:1153688:1154267 [4] NCCL INFO Setting affinity for GPU 4 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000
ubuntu:1153688:1154267 [4] NCCL INFO NVLS multicast support is available on dev 4
ubuntu:1153685:1154269 [1] NCCL INFO comm 0x1fdd0320 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
ubuntu:1153691:1154264 [7] NCCL INFO comm 0x3b6f23a0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
ubuntu:1153690:1154265 [6] NCCL INFO comm 0x45c952b0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
ubuntu:1153688:1154267 [4] NCCL INFO comm 0x2dd19f30 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
ubuntu:1153684:1154261 [0] NCCL INFO comm 0x29404f40 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
ubuntu:1153686:1154266 [2] NCCL INFO comm 0x42155fa0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
ubuntu:1153691:1154264 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
ubuntu:1153691:1154264 [7] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153687:1154271 [3] NCCL INFO comm 0x432a2a70 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
ubuntu:1153685:1154269 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
ubuntu:1153684:1154261 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
ubuntu:1153685:1154269 [1] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153684:1154261 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
ubuntu:1153690:1154265 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
ubuntu:1153688:1154267 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
ubuntu:1153684:1154261 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
ubuntu:1153689:1154272 [5] NCCL INFO comm 0x38f020c0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
ubuntu:1153690:1154265 [6] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153688:1154267 [4] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153684:1154261 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
ubuntu:1153687:1154271 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
ubuntu:1153684:1154261 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
ubuntu:1153686:1154266 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
ubuntu:1153687:1154271 [3] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153684:1154261 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
ubuntu:1153686:1154266 [2] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153684:1154261 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
ubuntu:1153689:1154272 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
ubuntu:1153684:1154261 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
ubuntu:1153684:1154261 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
ubuntu:1153684:1154261 [0] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153689:1154272 [5] NCCL INFO P2P Chunksize set to 524288
ubuntu:1153685:1154269 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Connected all rings
ubuntu:1153684:1154261 [0] NCCL INFO Connected all rings
ubuntu:1153685:1154269 [1] NCCL INFO Connected all rings
ubuntu:1153686:1154266 [2] NCCL INFO Connected all rings
ubuntu:1153685:1154269 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Connected all rings
ubuntu:1153691:1154264 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153685:1154269 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM
ubuntu:1153687:1154271 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Connected all rings
ubuntu:1153689:1154272 [5] NCCL INFO Connected all rings
ubuntu:1153688:1154267 [4] NCCL INFO Connected all rings
ubuntu:1153691:1154264 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153686:1154266 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153691:1154264 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153689:1154272 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153690:1154265 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153688:1154267 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/CUMEM
ubuntu:1153684:1154261 [0] NCCL INFO Connected all trees
ubuntu:1153685:1154269 [1] NCCL INFO Connected all trees
ubuntu:1153686:1154266 [2] NCCL INFO Connected all trees
ubuntu:1153691:1154264 [7] NCCL INFO Connected all trees
ubuntu:1153684:1154261 [0] NCCL INFO NVLS comm 0x29404f40 headRank 0 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153687:1154271 [3] NCCL INFO Connected all trees
ubuntu:1153685:1154269 [1] NCCL INFO NVLS comm 0x1fdd0320 headRank 1 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153688:1154267 [4] NCCL INFO Connected all trees
ubuntu:1153689:1154272 [5] NCCL INFO Connected all trees
ubuntu:1153690:1154265 [6] NCCL INFO Connected all trees
ubuntu:1153691:1154264 [7] NCCL INFO NVLS comm 0x3b6f23a0 headRank 7 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153686:1154266 [2] NCCL INFO NVLS comm 0x42155fa0 headRank 2 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153687:1154271 [3] NCCL INFO NVLS comm 0x432a2a70 headRank 3 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153689:1154272 [5] NCCL INFO NVLS comm 0x38f020c0 headRank 5 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153690:1154265 [6] NCCL INFO NVLS comm 0x45c952b0 headRank 6 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552
ubuntu:1153688:1154267 [4] NCCL INFO NVLS comm 0x2dd19f30 headRank 4 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552

ubuntu:1153688:1154267 [4] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153688:1154267 [4] NCCL INFO transport/nvls.cc:330 -> 1

ubuntu:1153689:1154272 [5] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153688:1154267 [4] NCCL INFO init.cc:1277 -> 1

ubuntu:1153690:1154265 [6] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153689:1154272 [5] NCCL INFO transport/nvls.cc:330 -> 1

ubuntu:1153687:1154271 [3] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153690:1154265 [6] NCCL INFO transport/nvls.cc:330 -> 1
ubuntu:1153689:1154272 [5] NCCL INFO init.cc:1277 -> 1

ubuntu:1153691:1154264 [7] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153687:1154271 [3] NCCL INFO transport/nvls.cc:330 -> 1
ubuntu:1153690:1154265 [6] NCCL INFO init.cc:1277 -> 1
ubuntu:1153688:1154267 [4] NCCL INFO init.cc:1548 -> 1

ubuntu:1153685:1154269 [1] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'

ubuntu:1153686:1154266 [2] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153691:1154264 [7] NCCL INFO transport/nvls.cc:330 -> 1
ubuntu:1153687:1154271 [3] NCCL INFO init.cc:1277 -> 1
ubuntu:1153688:1154267 [4] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153685:1154269 [1] NCCL INFO transport/nvls.cc:330 -> 1

ubuntu:1153684:1154261 [0] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument'
ubuntu:1153689:1154272 [5] NCCL INFO init.cc:1548 -> 1
ubuntu:1153686:1154266 [2] NCCL INFO transport/nvls.cc:330 -> 1
ubuntu:1153691:1154264 [7] NCCL INFO init.cc:1277 -> 1
ubuntu:1153685:1154269 [1] NCCL INFO init.cc:1277 -> 1
ubuntu:1153684:1154261 [0] NCCL INFO transport/nvls.cc:330 -> 1
ubuntu:1153689:1154272 [5] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153686:1154266 [2] NCCL INFO init.cc:1277 -> 1
ubuntu:1153687:1154271 [3] NCCL INFO init.cc:1548 -> 1
ubuntu:1153684:1154261 [0] NCCL INFO init.cc:1277 -> 1
ubuntu:1153690:1154265 [6] NCCL INFO init.cc:1548 -> 1
ubuntu:1153687:1154271 [3] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153690:1154265 [6] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153691:1154264 [7] NCCL INFO init.cc:1548 -> 1
ubuntu:1153691:1154264 [7] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153685:1154269 [1] NCCL INFO init.cc:1548 -> 1
ubuntu:1153684:1154261 [0] NCCL INFO init.cc:1548 -> 1
ubuntu:1153685:1154269 [1] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153684:1154261 [0] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153686:1154266 [2] NCCL INFO init.cc:1548 -> 1
ubuntu:1153686:1154266 [2] NCCL INFO group.cc:64 -> 1 [Async thread]
ubuntu:1153688:1153688 [4] NCCL INFO group.cc:418 -> 1
ubuntu:1153691:1153691 [7] NCCL INFO group.cc:418 -> 1
ubuntu:1153688:1153688 [4] NCCL INFO init.cc:1929 -> 1
ubuntu:1153691:1153691 [7] NCCL INFO init.cc:1929 -> 1
ubuntu:1153687:1153687 [3] NCCL INFO group.cc:418 -> 1
ubuntu:1153687:1153687 [3] NCCL INFO init.cc:1929 -> 1
ubuntu:1153689:1153689 [5] NCCL INFO group.cc:418 -> 1
ubuntu:1153689:1153689 [5] NCCL INFO init.cc:1929 -> 1
ubuntu:1153690:1153690 [6] NCCL INFO group.cc:418 -> 1
ubuntu:1153690:1153690 [6] NCCL INFO init.cc:1929 -> 1
ubuntu:1153686:1153686 [2] NCCL INFO group.cc:418 -> 1
ubuntu:1153684:1153684 [0] NCCL INFO group.cc:418 -> 1
ubuntu:1153686:1153686 [2] NCCL INFO init.cc:1929 -> 1
ubuntu:1153684:1153684 [0] NCCL INFO init.cc:1929 -> 1
ubuntu:1153685:1153685 [1] NCCL INFO group.cc:418 -> 1
ubuntu:1153685:1153685 [1] NCCL INFO init.cc:1929 -> 1
/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

W0425 13:50:06.741000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153684 via signal SIGTERM
W0425 13:50:06.742000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153685 via signal SIGTERM
W0425 13:50:06.742000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153686 via signal SIGTERM
W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153687 via signal SIGTERM
W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153688 via signal SIGTERM
W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153689 via signal SIGTERM
W0425 13:50:06.744000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153690 via signal SIGTERM
Traceback (most recent call last):
File "/root/DeepEP-main/tests/test_low_latency.py", line 172, in
torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 7 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/root/DeepEP-main/tests/test_low_latency.py", line 156, in test_loop
buffer = deep_ep.Buffer(group, num_rdma_bytes=num_rdma_bytes, low_latency_mode=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/deep_ep-1.0.0-py3.11-linux-x86_64.egg/deep_ep/buffer.py", line 59, in init
dist.all_gather_object(device_ids, local_device_id, group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object
all_gather(object_size_list, local_size, group=group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather
work = group.allgather([tensor_list], [tensor])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 1 'invalid argument'

@guodongrui-gdr commented on GitHub (Apr 25, 2025): > I haven't encountered this issue before, but I suspect it might be related to the CUDA driver version. Is there a way to pinpoint which CUDA call is triggering the "invalid argument" error? I ran with NCCL_DEBUG=INFO, get output below: Allocating buffer size: 2115.111296 MB ... ubuntu:1153684:1153684 [0] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153684:1153684 [0] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153684:1153684 [0] NCCL INFO cudaDriverVersion 12040 NCCL version 2.21.5+cuda12.4 ubuntu:1153684:1153684 [0] NCCL INFO Comm config Blocking set to 1 ubuntu:1153691:1153691 [7] NCCL INFO cudaDriverVersion 12040 ubuntu:1153691:1153691 [7] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153691:1153691 [7] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153691:1153691 [7] NCCL INFO Comm config Blocking set to 1 ubuntu:1153690:1153690 [6] NCCL INFO cudaDriverVersion 12040 ubuntu:1153690:1153690 [6] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153690:1153690 [6] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153690:1153690 [6] NCCL INFO Comm config Blocking set to 1 ubuntu:1153686:1153686 [2] NCCL INFO cudaDriverVersion 12040 ubuntu:1153686:1153686 [2] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153688:1153688 [4] NCCL INFO cudaDriverVersion 12040 ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153686:1153686 [2] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153686:1153686 [2] NCCL INFO Comm config Blocking set to 1 ubuntu:1153688:1153688 [4] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153688:1153688 [4] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153688:1153688 [4] NCCL INFO Comm config Blocking set to 1 ubuntu:1153685:1153685 [1] NCCL INFO cudaDriverVersion 12040 ubuntu:1153685:1153685 [1] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153685:1153685 [1] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153685:1153685 [1] NCCL INFO Comm config Blocking set to 1 ubuntu:1153687:1153687 [3] NCCL INFO cudaDriverVersion 12040 ubuntu:1153687:1153687 [3] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153687:1153687 [3] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153687:1153687 [3] NCCL INFO Comm config Blocking set to 1 ubuntu:1153689:1153689 [5] NCCL INFO cudaDriverVersion 12040 ubuntu:1153689:1153689 [5] NCCL INFO Bootstrap : Using ens10f0np0:7.55.112.5<0> ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so) ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so ubuntu:1153689:1153689 [5] NCCL INFO NET/Plugin: Using internal network plugin. ubuntu:1153689:1153689 [5] NCCL INFO Comm config Blocking set to 1 ubuntu:1153685:1154269 [1] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153685:1154269 [1] NCCL INFO Using non-device net plugin version 0 ubuntu:1153685:1154269 [1] NCCL INFO Using network IB ubuntu:1153684:1154261 [0] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153684:1154261 [0] NCCL INFO Using non-device net plugin version 0 ubuntu:1153684:1154261 [0] NCCL INFO Using network IB ubuntu:1153690:1154265 [6] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153686:1154266 [2] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153690:1154265 [6] NCCL INFO Using non-device net plugin version 0 ubuntu:1153690:1154265 [6] NCCL INFO Using network IB ubuntu:1153686:1154266 [2] NCCL INFO Using non-device net plugin version 0 ubuntu:1153686:1154266 [2] NCCL INFO Using network IB ubuntu:1153691:1154264 [7] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153691:1154264 [7] NCCL INFO Using non-device net plugin version 0 ubuntu:1153691:1154264 [7] NCCL INFO Using network IB ubuntu:1153688:1154267 [4] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153688:1154267 [4] NCCL INFO Using non-device net plugin version 0 ubuntu:1153688:1154267 [4] NCCL INFO Using network IB ubuntu:1153687:1154271 [3] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153687:1154271 [3] NCCL INFO Using non-device net plugin version 0 ubuntu:1153687:1154271 [3] NCCL INFO Using network IB ubuntu:1153689:1154272 [5] NCCL INFO NET/IB : Using [0]mlx5_7:1/RoCE [RO]; OOB ens10f0np0:7.55.112.5<0> ubuntu:1153689:1154272 [5] NCCL INFO Using non-device net plugin version 0 ubuntu:1153689:1154272 [5] NCCL INFO Using network IB ubuntu:1153687:1154271 [3] NCCL INFO ncclCommInitRank comm 0x432a2a70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 5d000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153685:1154269 [1] NCCL INFO ncclCommInitRank comm 0x1fdd0320 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 2a000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153686:1154266 [2] NCCL INFO ncclCommInitRank comm 0x42155fa0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 3b000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153688:1154267 [4] NCCL INFO ncclCommInitRank comm 0x2dd19f30 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153689:1154272 [5] NCCL INFO ncclCommInitRank comm 0x38f020c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId ab000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153684:1154261 [0] NCCL INFO ncclCommInitRank comm 0x29404f40 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 19000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153691:1154264 [7] NCCL INFO ncclCommInitRank comm 0x3b6f23a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId db000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153690:1154265 [6] NCCL INFO ncclCommInitRank comm 0x45c952b0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId bb000 commId 0x3ddd6a65f9575625 - Init START ubuntu:1153687:1154271 [3] NCCL INFO MNNVL busId 0x5d000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153691:1154264 [7] NCCL INFO MNNVL busId 0xdb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153690:1154265 [6] NCCL INFO MNNVL busId 0xbb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153685:1154269 [1] NCCL INFO MNNVL busId 0x2a000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153688:1154267 [4] NCCL INFO MNNVL busId 0x9b000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153684:1154261 [0] NCCL INFO MNNVL busId 0x19000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153686:1154266 [2] NCCL INFO MNNVL busId 0x3b000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153689:1154272 [5] NCCL INFO MNNVL busId 0xab000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 ubuntu:1153690:1154265 [6] NCCL INFO Setting affinity for GPU 6 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000 ubuntu:1153690:1154265 [6] NCCL INFO NVLS multicast support is available on dev 6 ubuntu:1153685:1154269 [1] NCCL INFO Setting affinity for GPU 1 to 3fffff,ff000000,00000000,00000000,3fffffff ubuntu:1153685:1154269 [1] NCCL INFO NVLS multicast support is available on dev 1 ubuntu:1153684:1154261 [0] NCCL INFO Setting affinity for GPU 0 to 3fffff,ff000000,00000000,00000000,3fffffff ubuntu:1153684:1154261 [0] NCCL INFO NVLS multicast support is available on dev 0 ubuntu:1153691:1154264 [7] NCCL INFO Setting affinity for GPU 7 to ffff,fffc0000,00000000,00000000,00ffffff,fc000000,00000000,00000000 ubuntu:1153691:1154264 [7] NCCL INFO NVLS multicast support is available on dev 7 ubuntu:1153686:1154266 [2] NCCL INFO Setting affinity for GPU 2 to 3fffff,ff000000,00000000,00000000,3fffffff ubuntu:1153686:1154266 [2] NCCL INFO NVLS multicast support is available on dev 2 ubuntu:1153689:1154272 [5] NCCL INFO Setting affinity for GPU 5 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000 ubuntu:1153689:1154272 [5] NCCL INFO NVLS multicast support is available on dev 5 ubuntu:1153687:1154271 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff,ffc00000,00000000,00000000,0fffffff,c0000000 ubuntu:1153687:1154271 [3] NCCL INFO NVLS multicast support is available on dev 3 ubuntu:1153688:1154267 [4] NCCL INFO Setting affinity for GPU 4 to 03ffff,fff00000,00000000,00000000,03ffffff,f0000000,00000000 ubuntu:1153688:1154267 [4] NCCL INFO NVLS multicast support is available on dev 4 ubuntu:1153685:1154269 [1] NCCL INFO comm 0x1fdd0320 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 ubuntu:1153691:1154264 [7] NCCL INFO comm 0x3b6f23a0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 ubuntu:1153690:1154265 [6] NCCL INFO comm 0x45c952b0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 ubuntu:1153688:1154267 [4] NCCL INFO comm 0x2dd19f30 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 ubuntu:1153684:1154261 [0] NCCL INFO comm 0x29404f40 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 ubuntu:1153686:1154266 [2] NCCL INFO comm 0x42155fa0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 ubuntu:1153691:1154264 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 ubuntu:1153691:1154264 [7] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153687:1154271 [3] NCCL INFO comm 0x432a2a70 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 ubuntu:1153685:1154269 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 ubuntu:1153684:1154261 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 ubuntu:1153685:1154269 [1] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153684:1154261 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 ubuntu:1153690:1154265 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 ubuntu:1153688:1154267 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 ubuntu:1153684:1154261 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 ubuntu:1153689:1154272 [5] NCCL INFO comm 0x38f020c0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 ubuntu:1153690:1154265 [6] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153688:1154267 [4] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153684:1154261 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 ubuntu:1153687:1154271 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 ubuntu:1153684:1154261 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 ubuntu:1153686:1154266 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 ubuntu:1153687:1154271 [3] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153684:1154261 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 ubuntu:1153686:1154266 [2] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153684:1154261 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 ubuntu:1153689:1154272 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 ubuntu:1153684:1154261 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 ubuntu:1153684:1154261 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 ubuntu:1153684:1154261 [0] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153689:1154272 [5] NCCL INFO P2P Chunksize set to 524288 ubuntu:1153685:1154269 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Connected all rings ubuntu:1153684:1154261 [0] NCCL INFO Connected all rings ubuntu:1153685:1154269 [1] NCCL INFO Connected all rings ubuntu:1153686:1154266 [2] NCCL INFO Connected all rings ubuntu:1153685:1154269 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Connected all rings ubuntu:1153691:1154264 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153685:1154269 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM ubuntu:1153687:1154271 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Connected all rings ubuntu:1153689:1154272 [5] NCCL INFO Connected all rings ubuntu:1153688:1154267 [4] NCCL INFO Connected all rings ubuntu:1153691:1154264 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153686:1154266 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153691:1154264 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153689:1154272 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153690:1154265 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153688:1154267 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/CUMEM ubuntu:1153684:1154261 [0] NCCL INFO Connected all trees ubuntu:1153685:1154269 [1] NCCL INFO Connected all trees ubuntu:1153686:1154266 [2] NCCL INFO Connected all trees ubuntu:1153691:1154264 [7] NCCL INFO Connected all trees ubuntu:1153684:1154261 [0] NCCL INFO NVLS comm 0x29404f40 headRank 0 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153687:1154271 [3] NCCL INFO Connected all trees ubuntu:1153685:1154269 [1] NCCL INFO NVLS comm 0x1fdd0320 headRank 1 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153688:1154267 [4] NCCL INFO Connected all trees ubuntu:1153689:1154272 [5] NCCL INFO Connected all trees ubuntu:1153690:1154265 [6] NCCL INFO Connected all trees ubuntu:1153691:1154264 [7] NCCL INFO NVLS comm 0x3b6f23a0 headRank 7 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153686:1154266 [2] NCCL INFO NVLS comm 0x42155fa0 headRank 2 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153687:1154271 [3] NCCL INFO NVLS comm 0x432a2a70 headRank 3 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153689:1154272 [5] NCCL INFO NVLS comm 0x38f020c0 headRank 5 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153690:1154265 [6] NCCL INFO NVLS comm 0x45c952b0 headRank 6 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153688:1154267 [4] NCCL INFO NVLS comm 0x2dd19f30 headRank 4 nHeads 8 buffSize 1048576 memSize 2097152 nvlsPerRankSize 150994944 nvlsTotalSize 1207959552 ubuntu:1153688:1154267 [4] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153688:1154267 [4] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153689:1154272 [5] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153688:1154267 [4] NCCL INFO init.cc:1277 -> 1 ubuntu:1153690:1154265 [6] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153689:1154272 [5] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153687:1154271 [3] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153690:1154265 [6] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153689:1154272 [5] NCCL INFO init.cc:1277 -> 1 ubuntu:1153691:1154264 [7] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153687:1154271 [3] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153690:1154265 [6] NCCL INFO init.cc:1277 -> 1 ubuntu:1153688:1154267 [4] NCCL INFO init.cc:1548 -> 1 ubuntu:1153685:1154269 [1] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153686:1154266 [2] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153691:1154264 [7] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153687:1154271 [3] NCCL INFO init.cc:1277 -> 1 ubuntu:1153688:1154267 [4] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153685:1154269 [1] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153684:1154261 [0] transport/nvls.cc:158 NCCL WARN Cuda failure 1 'invalid argument' ubuntu:1153689:1154272 [5] NCCL INFO init.cc:1548 -> 1 ubuntu:1153686:1154266 [2] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153691:1154264 [7] NCCL INFO init.cc:1277 -> 1 ubuntu:1153685:1154269 [1] NCCL INFO init.cc:1277 -> 1 ubuntu:1153684:1154261 [0] NCCL INFO transport/nvls.cc:330 -> 1 ubuntu:1153689:1154272 [5] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153686:1154266 [2] NCCL INFO init.cc:1277 -> 1 ubuntu:1153687:1154271 [3] NCCL INFO init.cc:1548 -> 1 ubuntu:1153684:1154261 [0] NCCL INFO init.cc:1277 -> 1 ubuntu:1153690:1154265 [6] NCCL INFO init.cc:1548 -> 1 ubuntu:1153687:1154271 [3] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153690:1154265 [6] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153691:1154264 [7] NCCL INFO init.cc:1548 -> 1 ubuntu:1153691:1154264 [7] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153685:1154269 [1] NCCL INFO init.cc:1548 -> 1 ubuntu:1153684:1154261 [0] NCCL INFO init.cc:1548 -> 1 ubuntu:1153685:1154269 [1] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153684:1154261 [0] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153686:1154266 [2] NCCL INFO init.cc:1548 -> 1 ubuntu:1153686:1154266 [2] NCCL INFO group.cc:64 -> 1 [Async thread] ubuntu:1153688:1153688 [4] NCCL INFO group.cc:418 -> 1 ubuntu:1153691:1153691 [7] NCCL INFO group.cc:418 -> 1 ubuntu:1153688:1153688 [4] NCCL INFO init.cc:1929 -> 1 ubuntu:1153691:1153691 [7] NCCL INFO init.cc:1929 -> 1 ubuntu:1153687:1153687 [3] NCCL INFO group.cc:418 -> 1 ubuntu:1153687:1153687 [3] NCCL INFO init.cc:1929 -> 1 ubuntu:1153689:1153689 [5] NCCL INFO group.cc:418 -> 1 ubuntu:1153689:1153689 [5] NCCL INFO init.cc:1929 -> 1 ubuntu:1153690:1153690 [6] NCCL INFO group.cc:418 -> 1 ubuntu:1153690:1153690 [6] NCCL INFO init.cc:1929 -> 1 ubuntu:1153686:1153686 [2] NCCL INFO group.cc:418 -> 1 ubuntu:1153684:1153684 [0] NCCL INFO group.cc:418 -> 1 ubuntu:1153686:1153686 [2] NCCL INFO init.cc:1929 -> 1 ubuntu:1153684:1153684 [0] NCCL INFO init.cc:1929 -> 1 ubuntu:1153685:1153685 [1] NCCL INFO group.cc:418 -> 1 ubuntu:1153685:1153685 [1] NCCL INFO init.cc:1929 -> 1 /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init /root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init W0425 13:50:06.741000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153684 via signal SIGTERM W0425 13:50:06.742000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153685 via signal SIGTERM W0425 13:50:06.742000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153686 via signal SIGTERM W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153687 via signal SIGTERM W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153688 via signal SIGTERM W0425 13:50:06.743000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153689 via signal SIGTERM W0425 13:50:06.744000 1153616 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 1153690 via signal SIGTERM Traceback (most recent call last): File "/root/DeepEP-main/tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 7 terminated with the following error: Traceback (most recent call last): File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/root/DeepEP-main/tests/test_low_latency.py", line 156, in test_loop buffer = deep_ep.Buffer(group, num_rdma_bytes=num_rdma_bytes, low_latency_mode=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/deep_ep-1.0.0-py3.11-linux-x86_64.egg/deep_ep/buffer.py", line 59, in __init__ dist.all_gather_object(device_ids, local_device_id, group) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object all_gather(object_size_list, local_size, group=group) File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather work = group.allgather([tensor_list], [tensor]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5 ncclUnhandledCudaError: Call to CUDA function failed. Last error: Cuda failure 1 'invalid argument'

bsakaguchi commented

2026-10-26 09:54:36 +00:00

Author

Owner

@ChenDolph7in commented on GitHub (Jun 13, 2025):

@guodongrui-gdr Have you solved this error?

@ChenDolph7in commented on GitHub (Jun 13, 2025): @guodongrui-gdr Have you solved this error?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#115

No description provided.

Rows
Columns