mirror of
https://github.com/deepseek-ai/3FS.git
synced 2026-11-05 00:47:47 +00:00
[GH-ISSUE #272] exec admin_cli "init-cluster block in container #164
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/3FS#164
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @qiuxinyidian on GitHub (May 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/272
[root@node10 foundationdb]# docker exec -it 3fs-monitor /bin/bash
[root@node10 app]# > /var/log/3fs/cli.log
[root@node10 app]# admin_cli "init-cluster --mgmtd /etc/3fs/mgmtd_main.toml 1 1048576 4"
^C
[root@node10 app]# admin_cli -cfg /etc/3fs/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["rdma://10.44.31.10:8000"]' "list-nodes"
bash: /usr/sbin/ibdev2netdev: No such file or directory
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD node10 14259 [] N/A 0(UPTODATE) 250228-dev-1-999999-dfadd598
[root@node10 app]# vi /var/log/3fs/cli.log
[root@node10 app]# > /var/log/3fs/cli.log
[root@node10 app]# admin_cli "init-cluster --mgmtd /etc/3fs/mgmtd_main.toml 1 1048576 4"
^[^C
[root@node10 app]# vi /var/log/3fs/cli.log
[root@node10 app]#
listnode not blocked
@qiuxinyidian commented on GitHub (May 9, 2025):
1、how to change log level in admin_cli ?
now monitor 、mgmt service is normal
@qiuxinyidian commented on GitHub (May 9, 2025):
if i skip init-cluster ，meta server cannot start with bellow logs
[2025-05-09T16:14:33.784769214+08:00 SvrBG0:14397 MgmtdClient.cc:183 ERROR] MgmtdClient: Heartbeat failed: Mgmtd::HeartbeatFail(5002) NodeId(100) not registered
[2025-05-09T16:14:33.784793956+08:00 SvrBG0:14397 MgmtdClient.cc:715 ERROR] MgmtdClient: Heartbeat failed: Mgmtd::HeartbeatFail(5002) NodeId(100) not registered
[2025-05-09T16:14:33.784994541+08:00 SvrBG0:14397 MgmtdClient.cc:399 ERROR] [MgmtdClientOp Heartbeat No.5] failed: Mgmtd::HeartbeatFail(5002) NodeId(100) not registered. latency: 0ns
[2025-05-09T16:14:33.785015892+08:00 SvrBG1:14398 MgmtdClient.cc:299 ERROR] MgmtdClient: AutoHeartbeat failed: Mgmtd::HeartbeatFail(5002) NodeId(100) not registered
[2025-05-09T16:14:33.786253402+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct serde::Services
[2025-05-09T16:14:33.786543343+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup
[2025-05-09T16:14:33.786550104+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.787876090+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.789095569+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.791599911+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct serde::Services
[2025-05-09T16:14:33.791865330+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup
[2025-05-09T16:14:33.791871241+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.793093650+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.794294937+08:00 meta_main:14358 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-05-09T16:14:33.797014135+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.797080509+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.797133493+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.797185337+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.797237201+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.797251122+08:00 meta_main:14358 StorageClientImpl.cc:1413 INFO] Starting storage client ClientId(f8d73fa3-75ad-4a0d-83de-26a723661db5)@node10
[2025-05-09T16:14:33.797306136+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 64, maxTokens: 4096, availableTokens: 64
[2025-05-09T16:14:33.797885058+08:00 meta_main:14358 StorageClientImpl.cc:1442 INFO] Storage client ClientId(f8d73fa3-75ad-4a0d-83de-26a723661db5)@node10 is getting the first routing info
[2025-05-09T16:14:33.797899229+08:00 meta_main:14358 StorageClientImpl.cc:1503 INFO] First routing info obtained: RoutingInfoVersion(6)
[2025-05-09T16:14:33.797921671+08:00 meta_main:14358 StorageClientImpl.cc:1461 INFO] Storage client ClientId(f8d73fa3-75ad-4a0d-83de-26a723661db5)@node10 started
[2025-05-09T16:14:33.797932141+08:00 StorageMsgrEL0:14436 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-05-09T16:14:33.798033249+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 4, maxTokens: 4096, availableTokens: 4
[2025-05-09T16:14:33.798087023+08:00 meta_main:14358 Semaphore.h:27 INFO] usableTokens: 32, maxTokens: 4096, availableTokens: 32
[2025-05-09T16:14:33.799414959+08:00 meta_main:14358 MetaOperator.cc:175 INFO] MetaOperator::init
[2025-05-09T16:14:33.799443861+08:00 meta_main:14358 StructuredTraceLog.h:238 INFO] Opening MetaEventTrace trace log: ./2025-05-09/node10/MetaEventTrace.node10.2025-05-09-16-14-33.1.parquet
[2025-05-09T16:14:33.799914765+08:00 meta_main:14358 GcManager.cc:621 INFO] GcManager::init
[2025-05-09T16:14:33.800353387+08:00 meta_main:14358 GcManager.cc:701 CRITICAL] Root or GcRoot not found, root false, gcRoot false
[2025-05-09T16:14:33.800375239+08:00 meta_main:14358 GcManager.cc:625 ERROR] GcManager::checkFs failed, error Meta::BadFileSystem(3011) src/common/kv/WithTransaction.h:45, '_result' has error
[2025-05-09T16:14:33.800385350+08:00 meta_main:14358 MetaServer.cc:74 ERROR] Init MetaOperator failed with Meta::BadFileSystem(3011) src/common/kv/WithTransaction.h:45, '_result' has error
[2025-05-09T16:14:33.800394010+08:00 meta_main:14358 Server.cc:39 ERROR] Server::beforeStart failed: Meta::BadFileSystem(3011) src/common/kv/WithTransaction.h:45, '_result' has error
[2025-05-09T16:14:33.800403111+08:00 meta_main:14358 TwoPhaseApplication.h:100 FATAL] Start server failed: Meta::BadFileSystem(3011) src/common/kv/WithTransaction.h:45, '_result' has error
@qiuxinyidian commented on GitHub (May 9, 2025):
[root@node10 ~]# ps -ef |grep admin
root 83891 83663 0 16:31 ? 00:00:00 admin_cli init-cluster --mgmtd /etc/3fs/mgmtd_main.toml 1 1048576 4
root 83988 83931 0 16:32 pts/1 00:00:00 grep --color=auto admin
[root@node10 ~]# strace -p 83891
strace: Process 83891 attached
futex(0xfffff9d271c8, FUTEX_WAIT_BITSET_PRIVATE, 4294967293, NULL, FUTEX_BITSET_MATCH_ANY
@bugwz commented on GitHub (May 9, 2025):
do you modify the foundationdb listen addr and port ？
/opt/3fs/etc/fdb.cluster/etc/foundationdb/fdb.clusterthen restart the foundationdb.
@qiuxinyidian commented on GitHub (May 9, 2025):
even change ip in /etc/foundationdb/fdb.cluster ，then copy it to /opt/3fs/etc/ ，restart fdb，
docker run -itd --net=host --name 3fs-mgmt --privileged=true -v /opt/3fs/etc/:/etc/3fs/ -v /var/log/3fs:/var/log/3fs 3b124b37eac2 /sbin/init
exec fdb in container， get the database is unavailable
@bugwz commented on GitHub (May 9, 2025):
My situation:
The
init-clusterhang was caused by FoundationDB's listener address being 127.0.0.1.Changing it to the actual IP resolved the issue.
Since you reported FDB unavailability, I suggest first fixing the FDB connection problem.
@bugwz commented on GitHub (May 9, 2025):
And also check the internal network in the container.
@qiuxinyidian commented on GitHub (May 9, 2025):
I will try it
@qiuxinyidian commented on GitHub (May 13, 2025):
both container and server all installed fdb in a node, after stopping fdb in container , I t works