deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #172] Error: The hostname of the client socket cannot be retrieved. err=-3 #148

New issue
Closed
opened 2026-10-26 09:55:15 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:55:15 +00:00
Owner
Copy link

Originally created by @ZhenshengWu on GitHub (May 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/172

when i run the test case

root@5d23d6366618:/sgl-workspace/DeepEP# python3 tests/test_intranode.py
[W521 13:07:59.801469723 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.823210136 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.823831992 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.829379102 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.831913012 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.832534764 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.846934093 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W521 13:07:59.852386971 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3

Did you met the same question?

version of nvshmem

NVSHMEM_MAJOR := 3
NVSHMEM_MINOR := 2
NVSHMEM_PATCH := 5
NVSHMEM_PACKAGE := 1

gdrcopy work normaly

root@5d23d6366618:/sgl-workspace/nvshmem# gdrcopy_copybw
GPU id:0; name: NVIDIA H800; Bus id: 0000:19:00
GPU id:1; name: NVIDIA H800; Bus id: 0000:3b:00
GPU id:2; name: NVIDIA H800; Bus id: 0000:4c:00
GPU id:3; name: NVIDIA H800; Bus id: 0000:5d:00
GPU id:4; name: NVIDIA H800; Bus id: 0000:9b:00
GPU id:5; name: NVIDIA H800; Bus id: 0000:bb:00
GPU id:6; name: NVIDIA H800; Bus id: 0000:cb:00
GPU id:7; name: NVIDIA H800; Bus id: 0000:db:00
selecting device 0
testing size: 131072
rounded size: 131072
gpu alloc fn: cuMemAlloc
device ptr: 7f47d1e00000
map_d_ptr: 0x7f47f5a25000
info.va: 7f47d1e00000
info.mapped_size: 131072
info.page_size: 65536
info.mapped: 1
info.wc_mapping: 1
page offset: 0
user-space pointer:0x7f47f5a25000
writing test, size=131072 offset=0 num_iters=10000
write BW: 18018.8MB/s
reading test, size=131072 offset=0 num_iters=100
read BW: 713.126MB/s
unmapping buffer
unpinning buffer
closing gdrdrv

root@5d23d6366618:/sgl-workspace/DeepEP# nvidia-smi
Wed May 21 13:09:45 2025

+----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H800                    On  |   00000000:19:00.0 Off |                    0 |
| N/A   33C    P0             79W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H800                    On  |   00000000:3B:00.0 Off |                    0 |
| N/A   30C    P0             81W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H800                    On  |   00000000:4C:00.0 Off |                    0 |
| N/A   29C    P0             77W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H800                    On  |   00000000:5D:00.0 Off |                    0 |
| N/A   31C    P0             76W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H800                    On  |   00000000:9B:00.0 Off |                    0 |
| N/A   33C    P0             78W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H800                    On  |   00000000:BB:00.0 Off |                    0 |
| N/A   31C    P0             76W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H800                    On  |   00000000:CB:00.0 Off |                    0 |
| N/A   32C    P0             77W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H800                    On  |   00000000:DB:00.0 Off |                    0 |
| N/A   29C    P0             76W /  700W |       1MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+



Infiniband device 'mlx5_gdr_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:d5b0
	base lid:	 0x709
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_1' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:d7a8
	base lid:	 0x56
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_2' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:c6a0
	base lid:	 0x57
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_3' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00ac:04c2
	base lid:	 0x708
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_4' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:c660
	base lid:	 0x58
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_5' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00ac:0432
	base lid:	 0x59
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_6' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:d198
	base lid:	 0x5a
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_gdr_7' port 1 status:
	default gid:	 fe80:0000:0000:0000:9c63:c003:00b1:d378
	base lid:	 0x5b
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_stor_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:5c25:7303:00ec:fb48
	base lid:	 0xda
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 200 Gb/sec (2X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_stor_1' port 1 status:
	default gid:	 fe80:0000:0000:0000:5c25:7303:00ec:fb49
	base lid:	 0xdc
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 200 Gb/sec (2X NDR)
	link_layer:	 InfiniBand
Originally created by @ZhenshengWu on GitHub (May 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/172 when i run the test case ``` root@5d23d6366618:/sgl-workspace/DeepEP# python3 tests/test_intranode.py [W521 13:07:59.801469723 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.823210136 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.823831992 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.829379102 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.831913012 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.832534764 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.846934093 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 [W521 13:07:59.852386971 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3 ``` Did you met the same question? version of nvshmem ``` NVSHMEM_MAJOR := 3 NVSHMEM_MINOR := 2 NVSHMEM_PATCH := 5 NVSHMEM_PACKAGE := 1 ``` gdrcopy work normaly ``` root@5d23d6366618:/sgl-workspace/nvshmem# gdrcopy_copybw GPU id:0; name: NVIDIA H800; Bus id: 0000:19:00 GPU id:1; name: NVIDIA H800; Bus id: 0000:3b:00 GPU id:2; name: NVIDIA H800; Bus id: 0000:4c:00 GPU id:3; name: NVIDIA H800; Bus id: 0000:5d:00 GPU id:4; name: NVIDIA H800; Bus id: 0000:9b:00 GPU id:5; name: NVIDIA H800; Bus id: 0000:bb:00 GPU id:6; name: NVIDIA H800; Bus id: 0000:cb:00 GPU id:7; name: NVIDIA H800; Bus id: 0000:db:00 selecting device 0 testing size: 131072 rounded size: 131072 gpu alloc fn: cuMemAlloc device ptr: 7f47d1e00000 map_d_ptr: 0x7f47f5a25000 info.va: 7f47d1e00000 info.mapped_size: 131072 info.page_size: 65536 info.mapped: 1 info.wc_mapping: 1 page offset: 0 user-space pointer:0x7f47f5a25000 writing test, size=131072 offset=0 num_iters=10000 write BW: 18018.8MB/s reading test, size=131072 offset=0 num_iters=100 read BW: 713.126MB/s unmapping buffer unpinning buffer closing gdrdrv ``` root@5d23d6366618:/sgl-workspace/DeepEP# nvidia-smi Wed May 21 13:09:45 2025 ``` +----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.127.08 Driver Version: 550.127.08 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H800 On | 00000000:19:00.0 Off | 0 | | N/A 33C P0 79W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA H800 On | 00000000:3B:00.0 Off | 0 | | N/A 30C P0 81W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA H800 On | 00000000:4C:00.0 Off | 0 | | N/A 29C P0 77W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA H800 On | 00000000:5D:00.0 Off | 0 | | N/A 31C P0 76W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA H800 On | 00000000:9B:00.0 Off | 0 | | N/A 33C P0 78W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA H800 On | 00000000:BB:00.0 Off | 0 | | N/A 31C P0 76W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA H800 On | 00000000:CB:00.0 Off | 0 | | N/A 32C P0 77W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA H800 On | 00000000:DB:00.0 Off | 0 | | N/A 29C P0 76W / 700W | 1MiB / 143771MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` ``` Infiniband device 'mlx5_gdr_0' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:d5b0 base lid: 0x709 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_1' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:d7a8 base lid: 0x56 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_2' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:c6a0 base lid: 0x57 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_3' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00ac:04c2 base lid: 0x708 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_4' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:c660 base lid: 0x58 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_5' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00ac:0432 base lid: 0x59 sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_6' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:d198 base lid: 0x5a sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_gdr_7' port 1 status: default gid: fe80:0000:0000:0000:9c63:c003:00b1:d378 base lid: 0x5b sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_stor_0' port 1 status: default gid: fe80:0000:0000:0000:5c25:7303:00ec:fb48 base lid: 0xda sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: InfiniBand Infiniband device 'mlx5_stor_1' port 1 status: default gid: fe80:0000:0000:0000:5c25:7303:00ec:fb49 base lid: 0xdc sm lid: 0x3 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: InfiniBand ```
bsakaguchi commented 2026-10-26 09:55:16 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (May 23, 2025):

This appears to be an error reported by PyTorch, and is not related to NVSHMEM.

<!-- gh-comment-id:2903330359 --> @sphish commented on GitHub (May 23, 2025): This appears to be an error reported by PyTorch, and is not related to NVSHMEM.
bsakaguchi commented 2026-10-26 09:55:16 +00:00
Author
Owner
Copy link

@zengxingchen commented on GitHub (Jul 1, 2025):

Hi, Zhensheng.
Could you please share how you solved this issue?

<!-- gh-comment-id:3021669002 --> @zengxingchen commented on GitHub (Jul 1, 2025): Hi, Zhensheng. Could you please share how you solved this issue?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#148
No description provided.