[GH-ISSUE #175] Encounter error: 6004(MgmtdClient::RoutingInfoNotReady) #108

New issue

Open

opened 2026-10-26 09:45:14 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented 2026-10-26 09:45:14 +00:00

Owner

Copy link

Originally created by @izxl007 on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/175

root@host-214:/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.234.5.197:8000"]' "set-config --type FUSE --file /opt/3fs/etc/hf3fs_fuse_main.toml"
Encounter error: 6004(MgmtdClient::RoutingInfoNotReady)
同样的命令和OS，一个节点执行成功，一个节点执行失败。
大家有没有碰到过？

Originally created by @izxl007 on GitHub (Mar 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/175 root@host-214:/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.234.5.197:8000"]' "set-config --type FUSE --file /opt/3fs/etc/hf3fs_fuse_main.toml" Encounter error: 6004(MgmtdClient::RoutingInfoNotReady) 同样的命令和OS，一个节点执行成功，一个节点执行失败。 大家有没有碰到过？

bsakaguchi commented 2026-10-26 09:45:15 +00:00

Author

Owner

Copy link

@Icedroid commented on GitHub (Mar 15, 2025):

我也遇到了这个问题，怎么解决的呢

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.3.25:8000"]' "list-nodes"
Encounter error: 6004(MgmtdClient::RoutingInfoNotReady)

然后mgmtd_main-err.log的错误如下：

[2025-03-15T10:06:53.201396617+00:00 SvrConn0:  228 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-15T10:06:53.201434547+00:00 SvrConn0:  228 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 still in ACCEPTED state after wait 15s
[2025-03-15T10:07:02.929891558+00:00 SvrConn1:  229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-15T10:07:02.929936771+00:00 SvrConn1:  229 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 still in ACCEPTED state after wait 15s
[2025-03-15T10:07:04.138540139+00:00 SvrConn1:  229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/955 accept timeout, port {mlx5_2:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}

<!-- gh-comment-id:2726407388 --> @Icedroid commented on GitHub (Mar 15, 2025): 我也遇到了这个问题，怎么解决的呢 ``` # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.3.25:8000"]' "list-nodes" Encounter error: 6004(MgmtdClient::RoutingInfoNotReady) ``` 然后mgmtd_main-err.log的错误如下： ``` [2025-03-15T10:06:53.201396617+00:00 SvrConn0: 228 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} [2025-03-15T10:06:53.201434547+00:00 SvrConn0: 228 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 still in ACCEPTED state after wait 15s [2025-03-15T10:07:02.929891558+00:00 SvrConn1: 229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} [2025-03-15T10:07:02.929936771+00:00 SvrConn1: 229 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 still in ACCEPTED state after wait 15s [2025-03-15T10:07:04.138540139+00:00 SvrConn1: 229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/955 accept timeout, port {mlx5_2:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} ```

bsakaguchi commented 2026-10-26 09:45:15 +00:00

Author

Owner

Copy link

@izxl007 commented on GitHub (Mar 15, 2025):

我也遇到了这个问题，怎么解决的呢

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.3.25:8000"]' "list-nodes"
Encounter error: 6004(MgmtdClient::RoutingInfoNotReady)

然后mgmtd_main-err.log的错误如下：

[2025-03-15T10:06:53.201396617+00:00 SvrConn0:  228 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-15T10:06:53.201434547+00:00 SvrConn0:  228 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 still in ACCEPTED state after wait 15s
[2025-03-15T10:07:02.929891558+00:00 SvrConn1:  229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-15T10:07:02.929936771+00:00 SvrConn1:  229 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 still in ACCEPTED state after wait 15s
[2025-03-15T10:07:04.138540139+00:00 SvrConn1:  229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/955 accept timeout, port {mlx5_2:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}

应该是网络的问题。我这边重启节点就好了。
3fs对网络很敏感，不要轻易动网络配置。

<!-- gh-comment-id:2726749686 --> @izxl007 commented on GitHub (Mar 15, 2025): > 我也遇到了这个问题，怎么解决的呢 > > ``` > # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.3.25:8000"]' "list-nodes" > Encounter error: 6004(MgmtdClient::RoutingInfoNotReady) > ``` > > 然后mgmtd_main-err.log的错误如下： > > ``` > [2025-03-15T10:06:53.201396617+00:00 SvrConn0: 228 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} > [2025-03-15T10:06:53.201434547+00:00 SvrConn0: 228 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/953 still in ACCEPTED state after wait 15s > [2025-03-15T10:07:02.929891558+00:00 SvrConn1: 229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 accept timeout, port {mlx5_1:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} > [2025-03-15T10:07:02.929936771+00:00 SvrConn1: 229 Listener.cc:218 ERROR] IBSocket RDMA://localhost.localdomain/mlx5_0:1/954 still in ACCEPTED state after wait 15s > [2025-03-15T10:07:04.138540139+00:00 SvrConn1: 229 IBSocket.cc:738 CRITICAL] IBSocket RDMA://localhost.localdomain/mlx5_0:1/955 accept timeout, port {mlx5_2:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE} > ``` 应该是网络的问题。我这边重启节点就好了。 3fs对网络很敏感，不要轻易动网络配置。

bsakaguchi commented 2026-10-26 09:45:15 +00:00

Author

Owner

Copy link

@Icedroid commented on GitHub (Mar 17, 2025):

@izxl007 我是容器化部署的，重启没用

<!-- gh-comment-id:2727784995 --> @Icedroid commented on GitHub (Mar 17, 2025): @izxl007 我是容器化部署的，重启没用

bsakaguchi commented 2026-10-26 09:45:15 +00:00

Author

Owner

Copy link

@feeyman commented on GitHub (Mar 28, 2025):

192.168.3.25

你好，请问192.168.3.25是IB卡的地址吗？我部署时候mgmtd_main的8000端口总是绑定到管理网地址上，请问遇到过吗？

<!-- gh-comment-id:2761544878 --> @feeyman commented on GitHub (Mar 28, 2025): > 192.168.3.25 你好，请问192.168.3.25是IB卡的地址吗？我部署时候mgmtd_main的8000端口总是绑定到管理网地址上，请问遇到过吗？

bsakaguchi commented 2026-10-26 09:45:16 +00:00

Author

Owner

Copy link

@izxl007 commented on GitHub (Apr 1, 2025):

192.168.3.25

你好，请问192.168.3.25是IB卡的地址吗？我部署时候mgmtd_main的8000端口总是绑定到管理网地址上，请问遇到过吗？

3fs的表现是和部署时候网卡的顺序相关，后面如果修改网卡的顺序，可能就会导致服务起不来

<!-- gh-comment-id:2767772669 --> @izxl007 commented on GitHub (Apr 1, 2025): > > 192.168.3.25 > > 你好，请问192.168.3.25是IB卡的地址吗？我部署时候mgmtd_main的8000端口总是绑定到管理网地址上，请问遇到过吗？ 3fs的表现是和部署时候网卡的顺序相关，后面如果修改网卡的顺序，可能就会导致服务起不来

bsakaguchi commented 2026-10-26 09:45:17 +00:00

Author

Owner

Copy link

@qiuxinyidian commented on GitHub (Apr 10, 2025):

也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试

<!-- gh-comment-id:2793999606 --> @qiuxinyidian commented on GitHub (Apr 10, 2025): 也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试

bsakaguchi commented 2026-10-26 09:45:17 +00:00

Author

Owner

Copy link

@qiuxinyidian commented on GitHub (Apr 10, 2025):

网卡填写filter直接coredump

<!-- gh-comment-id:2794008271 --> @qiuxinyidian commented on GitHub (Apr 10, 2025): 网卡填写filter直接coredump

bsakaguchi commented 2026-10-26 09:45:17 +00:00

Author

Owner

Copy link

@Funny-Boy-Max commented on GitHub (Apr 19, 2025):

也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试

我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没

<!-- gh-comment-id:2816722262 --> @Funny-Boy-Max commented on GitHub (Apr 19, 2025): > 也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试 我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没

bsakaguchi commented 2026-10-26 09:45:17 +00:00

Author

Owner

Copy link

@qiuxinyidian commented on GitHub (Apr 21, 2025):

也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试

我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没

解决了，我先修改RDMA网络链接名称，默认的网络连接名字有限制。你可以改成en0开头的即可。

<!-- gh-comment-id:2817638667 --> @qiuxinyidian commented on GitHub (Apr 21, 2025): > > 也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试 > > 我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没 解决了，我先修改RDMA网络链接名称，默认的网络连接名字有限制。你可以改成en0开头的即可。

bsakaguchi commented 2026-10-26 09:45:17 +00:00

Author

Owner

Copy link

@Funny-Boy-Max commented on GitHub (Apr 22, 2025):

也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试

我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没

解决了，我先修改RDMA网络链接名称，默认的网络连接名字有限制。你可以改成en0开头的即可。

感谢回复，请问一下你了解3FS集群扩展的问题么，我发现我这边的chaintable是删除不掉的

<!-- gh-comment-id:2821494494 --> @Funny-Boy-Max commented on GitHub (Apr 22, 2025): > > > 也遇到了，单节点部署，8000端口，rdma网卡没有被监听，list node 一直提示，还以为代码问题，明天重启试试 > > > > > > 我也遇到了这个问题，我查看了一下是默认绑定到本地TCP网卡了，没有绑定到RDMA网卡，这块我也不知道为什么，请问你解决了没 > > 解决了，我先修改RDMA网络链接名称，默认的网络连接名字有限制。你可以改成en0开头的即可。 感谢回复，请问一下你了解3FS集群扩展的问题么，我发现我这边的chaintable是删除不掉的

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

SF-Zhou-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#108

No description provided.