mirror of
https://github.com/deepseek-ai/3FS.git
synced 2026-11-05 00:47:47 +00:00
[GH-ISSUE #255] After deploying mgmtd and meta, the list-nodes command occasionally works and occasionally fails. What could be the reason? #159
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/3FS#159
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Hansyangxingxiang on GitHub (Apr 23, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/255
After deploying mgmtd and meta, the list-nodes command occasionally works and occasionally fails. What could be the reason?
The error message is as follows:
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:39:31 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:43:01 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:43:01 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Encounter error: 6004(MgmtdClient::RoutingInfoNotReady)
mgmtd error log:
[2025-04-23T09:35:26.145277402+00:00 SvrProc0: 4740 IBConnect.cc:681 ERROR] IBSocket RDMA://testnode1ssd/mlx5_bond_0:1/8500 failed to modify QP to RTR, errno 110, peer {"hostname":"testnode1ssd","dev":2,"dev_name":"mlx5_bond_0","port":1,"mtu":3,"qp_num":8500,"linklayer":{"type":"IBConnectRoCEInfo","value":{"gid":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u001f\fr"}}}, config {"sl":0,"traffic_class":0,"gid_index":0,"pkey_index":0,"start_psn":0,"min_rnr_timer":1,"timeout":14,"retry_cnt":7,"rnr_retry":0,,
"max_sge":16,"max_rdma_wr":128,"max_rdma_wr_per_post":32,"max_rd_atomic":16,"buf_size":16384,"send_buf_cnt":32,"buf_ack_batch":8,"buf_signal_batch":8,"event_ack_batch":128,"record_latency_per_peer":false}
[2025-04-23T09:35:26.145339871+00:00 SvrProc0: 4740 IBConnect.cc:168 ERROR] IBConnectService::connect failed to accept from 172.31.12.114, error RPC::ConnectFailed(2014) IBSocket failed to modify QP to RTR.
[2025-04-23T09:35:36.642292569+00:00 SvrConn1: 4747 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_2:1/2059 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:36.642351209+00:00 SvrConn1: 4747 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_2:1/2059 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:37.742451498+00:00 SvrConn0: 4746 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_3:1/4342 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:37.742510895+00:00 SvrConn0: 4746 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_3:1/4342 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:42.484212045+00:00 SvrConn0: 4746 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_2:1/2062 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:42.484264204+00:00 SvrConn0: 4746 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_2:1/2062 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:43.582416723+00:00 SvrConn1: 4747 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_3:1/4344 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:43.582465859+00:00 SvrConn1: 4747 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_3:1/4344 still in ACCEPTED state after wait 15s
Could it be due to network timeout? However, the network latency appears normal upon inspection
@qiuxinyidian commented on GitHub (May 6, 2025):
ip addr 看到的rdma网卡的名字是啥？
@Hansyangxingxiang commented on GitHub (May 6, 2025):
ip addr |grep -E 'bond0|ens4'
6: ens39f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
7: ens39f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
8: ens41f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.10.1.114/24 brd 10.10.1.255 scope global ens41f0
9: ens41f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.10.2.114/24 brd 10.10.2.255 scope global ens41f1
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 172.31.12.114/24 brd 172.31.12.255 scope global bond0
root@testnode1ssd:~# ibdev2netdev
mlx5_2 port 1 ==> ens41f0 (Up)
mlx5_3 port 1 ==> ens41f1 (Up)
mlx5_bond_0 port 1 ==> bond0 (Up)
刚开始的时候我以为是组bond的问题，后来我把bond的这个口过滤了，剩下ens41f0，ens41f1 ，依然还是概率会出现
@qiuxinyidian commented on GitHub (May 6, 2025):
日志目录下的，admin_cli log也上传一下吧下
@Hansyangxingxiang commented on GitHub (May 7, 2025):
我现在的环境直接一直用不了，重启启动了一下mgmtd，报错如下，好像没有正确监听8000端口了：
[2025-05-07T06:39:34.339933099+00:00 mgmtd_main: 560 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_0 => ens39f0
[2025-05-07T06:39:34.341059981+00:00 mgmtd_main: 560 IfAddrs.h:102 INFO] Get ifaddr of ens41f0, addr 10.10.1.114/24, subnet 10.10.1.0/24, up true
[2025-05-07T06:39:34.341079691+00:00 mgmtd_main: 560 IfAddrs.h:102 INFO] Get ifaddr of ens41f1, addr 10.10.2.114/24, subnet 10.10.2.0/24, up true
[2025-05-07T06:39:34.341089928+00:00 mgmtd_main: 560 IfAddrs.h:102 INFO] Get ifaddr of bond0, addr 172.31.12.114/24, subnet 172.31.12.0/24, up true
[2025-05-07T06:39:34.345429131+00:00 mgmtd_main: 560 IBDevice.cc:396 INFO] IBDevice mlx5_2:1 IP addrs 10.10.1.114/24
[2025-05-07T06:39:34.345446667+00:00 mgmtd_main: 560 IBDevice.cc:441 CRITICAL] IBDevice mlx5_2:1 can't set zone by IP, fallback to UNKNOWN
[2025-05-07T06:39:34.345545609+00:00 mgmtd_main: 560 IBDevice.cc:367 INFO] IBDevice mlx5_2 add active port 1, linklayer ETHERNET, addrs 10.10.1.114/24, zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a🅰️1:72
[2025-05-07T06:39:34.345555024+00:00 mgmtd_main: 560 IBDevice.cc:256 INFO] IBDevice add mlx5_2, id 0, 1 available ports
[2025-05-07T06:39:34.347811431+00:00 mgmtd_main: 560 IBDevice.cc:396 INFO] IBDevice mlx5_3:1 IP addrs 10.10.2.114/24
[2025-05-07T06:39:34.347819971+00:00 mgmtd_main: 560 IBDevice.cc:441 CRITICAL] IBDevice mlx5_3:1 can't set zone by IP, fallback to UNKNOWN
[2025-05-07T06:39:34.347903646+00:00 mgmtd_main: 560 IBDevice.cc:367 INFO] IBDevice mlx5_3 add active port 1, linklayer ETHERNET, addrs 10.10.2.114/24, zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a🅰️2:72
[2025-05-07T06:39:34.347912039+00:00 mgmtd_main: 560 IBDevice.cc:256 INFO] IBDevice add mlx5_3, id 1, 1 available ports
[2025-05-07T06:39:34.350437905+00:00 mgmtd_main: 560 IBDevice.cc:305 INFO] Skip device mlx5_bond_0, port 1 because it's not in device filter.
[2025-05-07T06:39:34.350447345+00:00 mgmtd_main: 560 IBDevice.cc:253 INFO] IBDevice skip mlx5_bond_0 because it doesn't have available ports.
[2025-05-07T06:39:34.353705673+00:00 IBManager: 662 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-05-07T06:39:34.353766636+00:00 mgmtd_main: 560 ServerLauncher.h:43 INFO] IBDevice inited
[2025-05-07T06:39:34.358750227+00:00 mgmtd_main: 560 Utils.cc:239 INFO] Start load AppInfo from launcher
@Hansyangxingxiang commented on GitHub (May 7, 2025):
原来是概率报这个错误的时候cli的日志是报错：
[2025-04-24T08:08:48.465164712+00:00 CliBG1:18573 Transport.cc:138 DEBUG] Wait ib socket 0x563e0d5cd800 last read/write finished begin
[2025-04-24T08:08:48.465173914+00:00 CliIO1:18571 Transport.cc:194 WARNING] transport [connect RDMA://10.10.1.114:8000] read error notified by epoll
[2025-04-24T08:08:48.465203684+00:00 CliIO0:18570 Transport.cc:261 WARNING] transport [connect RDMA://10.10.1.114:8000] write error notified by epoll
[2025-04-24T08:08:48.465249659+00:00 CliBG1:18573 Transport.cc:140 DEBUG] Wait ib socket 0x563e0d5cd800 last read/write finished end
[2025-04-24T08:08:48.465259828+00:00 CliBG1:18573 IBSocket.cc:706 INFO] IBSocket [connect RDMA://10.10.1.114:8000] close
[2025-04-24T08:08:48.465266555+00:00 CliBG1:18573 IBSocket.cc:809 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] send close msg.
[2025-04-24T08:08:48.482879305+00:00 CliProc1:18569 Processor.h:153 DEBUG] receive response 11:2
[2025-04-24T08:08:48.482944228+00:00 CliConn1:18575 IBConnect.cc:418 ERROR] IBSocket [connect RDMA://10.10.1.114:8000] failed to connect, error RPC::ConnectFailed(2014) IBSocket failed to modify QP to RTR., timeout 5s
[2025-04-24T08:08:48.482991484+00:00 CliConn1:18575 IOWorker.cc:89 ERROR] error: RPC::ConnectFailed(2014) IBSocket failed to modify QP to RTR.
[2025-04-24T08:08:48.569487320+00:00 CliProc1:18569 Processor.h:153 DEBUG] receive response 11:1
[2025-04-24T08:08:48.569533007+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_2:1, remote mlx5_2:1, zone UNKNOWN}
[2025-04-24T08:08:48.569544284+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_3:1, remote mlx5_2:1, zone UNKNOWN}
[2025-04-24T08:08:48.569549340+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_bond_0:1, remote mlx5_2:1, zone UNKNOWN}
[2025-04-24T08:08:48.569554321+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_2:1, remote mlx5_3:1, zone UNKNOWN}
[2025-04-24T08:08:48.569565544+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_3:1, remote mlx5_3:1, zone UNKNOWN}
[2025-04-24T08:08:48.569570487+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_bond_0:1, remote mlx5_3:1, zone UNKNOWN}
[2025-04-24T08:08:48.569572979+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_2:1, remote mlx5_bond_0:1, zone UNKNOWN}
[2025-04-24T08:08:48.569575173+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_3:1, remote mlx5_bond_0:1, zone UNKNOWN}
[2025-04-24T08:08:48.569578827+00:00 CliConn1:18575 IBConnect.cc:255 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] find match devices {local mlx5_bond_0:1, remote mlx5_bond_0:1, zone UNKNOWN}
[2025-04-24T08:08:48.569582923+00:00 CliConn1:18575 IBConnect.cc:290 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] use devices {local mlx5_2:1, remote mlx5_bond_0:1, zone UNKNOWN}
[2025-04-24T08:08:48.569589678+00:00 CliConn1:18575 IBConnect.cc:386 INFO] IBSocket [connect RDMA://10.10.1.114:8000] connect with dev {local mlx5_2:1, remote mlx5_bond_0:1, zone UNKNOWN}
[2025-04-24T08:08:48.569794934+00:00 CliConn1:18575 IBDevice.cc:112 DEBUG] mlx5_2:1 skip GID 0, ip fe80::ba59:9fff:fe06:5b0c
[2025-04-24T08:08:48.569813169+00:00 CliConn1:18575 IBDevice.cc:112 DEBUG] mlx5_2:1 skip GID 1, ip fe80::ba59:9fff:fe06:5b0c
[2025-04-24T08:08:48.569842830+00:00 CliConn1:18575 IBDevice.cc:129 DEBUG] mlx5_2:1 skip GID 2, type IB/RoCE v1 != RoCE v2
[2025-04-24T08:08:48.569866486+00:00 CliConn1:18575 IBDevice.cc:133 DEBUG] mlx5_2:1 found RoCE v2 GID, index 3, ip ::ffff:10.10.1.114
[2025-04-24T08:08:48.569881749+00:00 CliConn1:18575 IBConnect.cc:545 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] create QP.
[2025-04-24T08:08:48.570402865+00:00 CliConn1:18575 IBConnect.cc:727 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] init socket buffers.
[2025-04-24T08:08:48.571034239+00:00 CliConn1:18575 IBDevice.cc:520 DEBUG] IBDevice mlx5_2 reg_mr, addr 0x563e0f35a000, length 1196032, access 1048577, mr 0x563e0dd1f400
[2025-04-24T08:08:48.571069632+00:00 CliConn1:18575 IBConnect.cc:606 DEBUG] IBSocket [connect RDMA://10.10.1.114:8000] init QP.
@qiuxinyidian commented on GitHub (May 7, 2025):
这个可以netstat -tpnl |grep 8000 看看，你的rdma 网卡是en bound开头 ，看代码建立监听应该没问题
@Hansyangxingxiang commented on GitHub (May 8, 2025):
netstat -tpnl |grep 8000
root@testnode1ssd:~# 没监听，刚开始部署的时候我看了是有监听起来的，只是 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.10.1.114:8000"]' "list-nodes" 会概率出现失败，过了几天后发现一直失败了，查看端口也没监听了，重新启动了mgmtd，就发出报了上面的错误，端口监听也有问题了
@Hansyangxingxiang commented on GitHub (May 8, 2025):
没监听8000原因找到了，是因为空间满foundationdb异常了，但是mgmtd启动的时候确没报任何与之相关的错误
@Hansyangxingxiang commented on GitHub (May 9, 2025):
"list-nodes" 还是会概率出现失败，而且发现如果是在mgmtd节点直接上查询，很少有失败的，但是在在storage的节点上查询出现失败的概率很大，如果是网络压力呢，但是又没有跑IO，这个就奇怪了
cli的日志如下：报错的错误确实是超时相关的
@Hansyangxingxiang commented on GitHub (May 9, 2025):
补充环境信息：
OS：ubuntu20.04
网卡：
5e:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5e:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5f:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5f:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
另外就是3FS各组件是容器化部署的，实验环境无法连接外网，安装相关的依赖比较麻烦，容器镜像也是基于ubuntu20.04，启动容器方式：
docker run -it --privileged --network host hf3fs-monitor:latest /bin/bash
docker run -it --privileged --network host hf3fs-mgmtd:latest /bin/bash
docker run --network host --privileged -it hf3fs-meta:latest /bin/bash
docker run --network host --privileged -it hf3fs-fuse:latest /bin/bash
启动容器后再进入容器里面启动各组件
@qiuxinyidian commented on GitHub (May 9, 2025):
我这边容器部署，admin_cli 注册服务成功，但是meta启动日志提示node 未注册，物理机部署没这个问题，折腾中，我这边定制的操作系统，分析中
@xlcbingo1999 commented on GitHub (Aug 8, 2025):
请问是否有解决方案呢？我也在容器部署的时候遇到问题
@qiuxinyidian commented on GitHub (Aug 8, 2025):
先使用fdbcli连下fdb， ck client连下ck server，看看是否正常