[GH-ISSUE #416] deployment crashed at data placement part need expert to look into #238

New issue

Open

opened 2026-10-26 09:47:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-26 09:47:37 +00:00

Owner

Copy link

Originally created by @gaowayne on GitHub (Jun 7, 2026).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/416

hello team

below is error

(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml
DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:14500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName:
    nodes:
3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry: Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout:<nil> ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s}
INFO[0000] Running task CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data:   NODE="<LOCAL>"
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts:   NODE="<LOCAL>"
INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql  NODE=local TASK=CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: e166fa5bf7c46930cd28ac238f6d6a8cb1d42bff0938cf274fb75964865f8226  NODE="<LOCAL>"
DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response  NODE="<LOCAL>"
DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2  NODE=local TASK=CreatePgClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections  NODE="<LOCAL>"
INFO[0001] Started postgresql container 3fs-postgres successfully  NODE=local TASK=CreatePgClusterTask
INFO[0001] Running task CreateFdbClusterTask
DEBU[0001] fdb cluster file content: 3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500  NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data:   NODE="<LOCAL>"
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs:   NODE="<LOCAL>"
DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: 9bace2adf85cd6e1f1c5e6971171570ce8f864411ee588f6e3e0f89bb8e9f59e  NODE="<LOCAL>"
INFO[0001] Started fdb container 3fs-fdb successfully    NODE=local TASK=CreateFdbClusterTask
INFO[0001] Initializing fdb cluster                      NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd'  NODE="<LOCAL>"
^C
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# vim cluster.yml
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml
DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:4500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName:3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry:Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout:<nil> ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s}
INFO[0000] Running task CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data:   NODE="<LOCAL>"
DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts:   NODE="<LOCAL>"
INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql  NODE=local TASK=CreatePgClusterTask
DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: fa11a8f1cc8e28698f5616daae0c09694097d86744decce0f330ca763f54b849  NODE="<LOCAL>"
DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres  NODE="<LOCAL>"
DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response  NODE="<LOCAL>"
DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2  NODE=local TASK=CreatePgClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections  NODE="<LOCAL>"
INFO[0001] Started postgresql container 3fs-postgres successfully  NODE=local TASK=CreatePgClusterTask
INFO[0001] Running task CreateFdbClusterTask
DEBU[0001] fdb cluster file content: 45dh8lnkka:hi1ldtjncq@10.74.40.86:4500  NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data:   NODE="<LOCAL>"
DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs:   NODE="<LOCAL>"
DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63  NODE="<LOCAL>"
DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: d11b3ff5ade923a38c7408a59c3e2322902d34678701143071ee0dc77fc04087  NODE="<LOCAL>"
INFO[0001] Started fdb container 3fs-fdb successfully    NODE=local TASK=CreateFdbClusterTask
INFO[0001] Initializing fdb cluster                      NODE=local TASK=CreateFdbClusterTask
DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd'  NODE="<LOCAL>"
DEBU[0004] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd': Database created  NODE="<LOCAL>"
INFO[0004] Waiting for fdb cluster initialized           NODE=local TASK=CreateFdbClusterTask
DEBU[0004] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal'  NODE="<LOCAL>"
DEBU[0005] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available, but has issues (type 'status' for more information).  NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal'  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available.  NODE="<LOCAL>"
INFO[0010] Initialized fdb cluster                       NODE=local TASK=CreateFdbClusterTask
INFO[0010] Running task CreateClickhouseClusterTask
DEBU[0010] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX: /tmp/3fs-clickhouse.jCQ5nO  NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql:   NODE="<LOCAL>"
DEBU[0010] Run command: sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy  NODE="<LOCAL>"
DEBU[0010] Output of sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy: e63b4ffd3e8cf6f8be7d72d88f40ffe8a3c80d9502fe5d83945e899613a77861  NODE="<LOCAL>"
INFO[0015] Started clickhouse container 3fs-clickhouse successfully  NODE=local TASK=CreateClickhouseClusterTask
INFO[0015] Initializing clickhouse cluster               NODE=local TASK=CreateClickhouseClusterTask
DEBU[0015] Run command: sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql"  NODE="<LOCAL>"
DEBU[0015] Output of sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql":   NODE="<LOCAL>"
INFO[0015] Initialized clickhouse cluster                NODE=local TASK=CreateClickhouseClusterTask
DEBU[0015] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO  NODE="<LOCAL>"
DEBU[0015] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO:   NODE="<LOCAL>"
INFO[0015] Running task CreateGrafanaServiceTask
INFO[0015] Generating datasource.yaml                    NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources  NODE="<LOCAL>"
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources:   NODE="<LOCAL>"
INFO[0015] Generating dashboard.yaml                     NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards  NODE="<LOCAL>"
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards:   NODE="<LOCAL>"
INFO[0015] Generating 3fs.json                           NODE=local TASK=CreateGrafanaServiceTask
DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards  NODE="<LOCAL>"
DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards:   NODE="<LOCAL>"
DEBU[0015] Run command: sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0: 849e83e1e65ee4aeb3f4e18efdf5c1cad63ba2ce9d562e07ee03c9d4145691cc  NODE="<LOCAL>"
INFO[0016] Started grafana container 3fs-grafana successfully, service endpoint is http://10.74.40.86:3000, login with username "admin" and password "admin"  NODE=local TASK=CreateGrafanaServiceTask
INFO[0016] Running task CreateMonitorTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX: /tmp/3fs-monitor.stFhHX  NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml: 475e4148833690ba401f2855e1742f3e8fb71bb2e375deea3bf6aefdbcc93acc  NODE="<LOCAL>"
INFO[0016] Started monitor container 3fs-monitor successfully  NODE=local TASK=CreateMonitorTask
DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX:   NODE="<LOCAL>"
INFO[0016] Running task CreateMgmtdServiceTask
DEBU[0016] Node ID map: map[local:1]                     NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Admin cli config template data: map[ClusterID:open3fs-single MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]]  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.DoAci2  NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2:   NODE="<LOCAL>"
INFO[0016] Create mgmtd_main config dir /mnt/nvme3n1/3fs-lab/mgmtd  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd:   NODE="<LOCAL>"
DEBU[0016] Template data of mgmtd_main_app.toml.tmpl: map[NodeID:1]  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main_app.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_app.toml  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Config of mgmtd_main_app.toml: allow_empty_node_id = true
node_id = 1  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Template data of mgmtd_main_launcher.toml.tmpl: map[ClusterID:open3fs-single HostMountpoint:/mnt/3fsmount MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]]  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main_launcher.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_launcher.toml  NODE=local TASK=CreateMgmtdServiceTask

clusterFile = '/opt/3fs/etc/fdb.cluster'
default_backoff = 0
enableMultipleClient = false
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json'

[ib_devices]
allow_no_usable_devices = false
allow_unknown_zone = true
default_network_zone = 'UNKNOWN'
default_pkey_index = 0
default_roce_pkey_index = 0
default_traffic_class = 0
device_filter = []
fork_safe = true
prefer_ibdevice = true
skip_inactive_ports = true
skip_unusable_device = true
subnets = []

[kv_engine]
use_memkv = false

[kv_engine.fdb]
casual_read_risky = false
clusterFile = ''
default_backoff = 0
enableMultipleClient = false
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json'  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Template data of mgmtd_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8000 TCPListenPort:9000]  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating mgmtd_main.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main.toml  NODE=local TASK=CreateMgmtdServiceTask

[common.monitor]
collect_period = '1s'
num_collectors = 1

[[common.monitor.reporters]]
type = 'monitor_collector'

[common.monitor.reporters.monitor_collector]
remote_ip = "10.74.40.86:10000"

[server.base.independent_thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.base.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[[server.base.groups]]
check_connections_interval = '1min'
connection_expiration_time = '1day'
network_type = 'RDMA'
services = [ 'Mgmtd' ]
use_independent_thread_pool = false


[server.service]
allow_heartbeat_from_unregistered = true
authenticate = false
bootstrapping_length = '2min'
bump_routing_info_version_interval = '5s'
check_status_interval = '10s'
client_session_timeout = '20min'
enable_routinginfo_cache = true
extend_lease_check_release_version = true
extend_lease_interval = '10s'
heartbeat_fail_interval = '1min'
heartbeat_ignore_stale_targets = true
heartbeat_ignore_unknown_targets = false
heartbeat_timestamp_valid_window = '30s'
lease_length = '1min'
new_chain_bootstrap_interval = '2min'
only_accept_client_uuid = false
retry_times_on_txn_errors = -1
send_heartbeat = true
send_heartbeat_interval = '10s'
suspicious_lease_interval = '20s'
target_info_load_interval = '1s'
target_info_persist_batch = 1000
target_info_persist_interval = '1s'
try_adjust_target_order_as_preferred = false
update_chains_interval = '1s'
update_metrics_interval = '1s'
validate_lease_on_write = true

[server.service.retry_transaction]
max_backoff = '1s'
max_retry_count = 10

[server.service.user_cache]
buckets = 127
exist_ttl = '5min'
inexist_ttl = '10s'  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Save admin cli config to /tmp/prepare-3fs-config.DoAci2/admin_cli.toml  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Generating fdb.cluster to /tmp/prepare-3fs-config.DoAci2/fdb.cluster  NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Copying mgmtd_main configs from /tmp/prepare-3fs-config.DoAci2 to local /mnt/nvme3n1/3fs-lab/mgmtd/config.d  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log:   NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16'  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16': Init filesystem, root directory layout: chain table ChainTableId(1), chunksize 1048576, stripesize 16

Init config for MGMTD version 1  NODE="<LOCAL>"
INFO[0016] Cluster initialization success                NODE=local TASK=CreateMgmtdServiceTask
INFO[0016] Starting mgmtd_main container 3fs-mgmtd       NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml: cec202504223f22561a2b9a90f37f918146ca0389adb2ced6d7c79fcd6968934  NODE="<LOCAL>"
INFO[0016] Started mgmtd_main container 3fs-mgmtd successfully  NODE=local TASK=CreateMgmtdServiceTask
DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX  NODE="<LOCAL>"
DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX: /tmp/3fs-mgmtd.65wChW  NODE="<LOCAL>"
DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW:   NODE="<LOCAL>"
DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh:   NODE="<LOCAL>"
DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW:   NODE="<LOCAL>"
INFO[0017] Running task CreateMetaServiceTask
DEBU[0017] Node ID map: map[local:100]                   NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.57VFbH  NODE="<LOCAL>"
DEBU[0017] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH:   NODE="<LOCAL>"
INFO[0017] Create meta_main config dir /mnt/nvme3n1/3fs-lab/meta  NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta:   NODE="<LOCAL>"
DEBU[0017] Template data of meta_main_app.toml.tmpl: map[NodeID:100]  NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating meta_main_app.toml to /tmp/prepare-3fs-config.57VFbH/meta_main_app.toml  NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Config of meta_main_app.toml: allow_empty_node_id = true
node_id = 100  NODE=local TASK=CreateMetaServiceTask

[client.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[ib_devices]
allow_no_usable_devices = false
allow_unknown_zone = true
default_network_zone = 'UNKNOWN'
default_pkey_index = 0
default_roce_pkey_index = 0
default_traffic_class = 0
device_filter = []
fork_safe = true
prefer_ibdevice = true
skip_inactive_ports = true
skip_unusable_device = true
subnets = []

[mgmtd_client]
accept_incomplete_routing_info_during_mgmtd_bootstrapping = true
auto_extend_client_session_interval = '10s'
auto_heartbeat_interval = '10s'
auto_refresh_interval = '10s'
enable_auto_extend_client_session = false
enable_auto_heartbeat = true
enable_auto_refresh = true
mgmtd_server_addresses = ["RDMA://10.74.40.86:8000"]
work_queue_size = 100  NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Template data of meta_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8001 TCPListenPort:9001]  NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating meta_main.toml to /tmp/prepare-3fs-config.57VFbH/meta_main.toml  NODE=local TASK=CreateMetaServiceTask


[server.background_client.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.base.independent_thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.base.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[[server.base.groups]]
check_connections_interval = '1min'
connection_expiration_time = '1day'
network_type = 'RDMA'
services = [ 'Me
externalClientDir = ''
externalClientPath = ''
multipleClientThreadNum = 4
readonly = false
trace_file = ''
trace_format = 'json'

[server.meta]
acl_cache_time = '15s'
allow_directly_move_to_trash = false
allow_owner_change_immutable = false
allow_stat_deleted_inodes = true
authenticate = false
batch_stat_by_path_concurrent = 4
batch_stat_concurrent = 8
check_file_hole = false
dynamic_stripe = false
dynamic_stripe_growth = 2
dynamic_stripe_initial = 16
enable_new_chunk_engine = false
grv_cache = false
idempotent_record_clean = '1min'
idempotent_record_expire = '30min'
idempotent_remove = true
idempotent_rename = false
iflags_chain_allocation = false
iflags_chunk_engine = true
ignore_length_hint = false
inodeId_abort_on_duplicate = false
inodeId_check_unique = true
list_default_limit = 128
max_batch_operations = 4096
max_directory_depth = 64
max_remove_chunks_per_request = 32
max_symlink_count = 10
max_symlink_depth = 4
operation_timeout = '5s'
otrunc_replace_file = true
otrunc_replace_file
rdma_connect_timeout = '5s'
read_write_rdma_in_event_thread = false
read_write_tcp_in_event_thread = false
tcp_connect_timeout = '1s'
wait_to_retry_send = '100ms'

[server.storage_client.net_client.io_worker.connect_concurrency_limiter]
max_concurrency = 4

[server.storage_client.net_client.io_worker.ibsocket]
buf_ack_batch = 8
buf_signal_batch = 8
buf_size = 16384
drain_timeout = '5s'
drop_connections = 0
event_ack_batch = 128
max_rd_atomic = 16
max_rdma_wr = 128
max_rdma_wr_per_post = 32
max_sge = 1
min_rnr_timer = 1
record_bytes_per_peer = false
record_latency_per_peer = false
retry_cnt = 7
rnr_retry = 0
send_buf_cnt = 32
sl = 0
start_psn = 0
timeout = 14

[server.storage_client.net_client.io_worker.transport_pool]
max_connections = 1

[server.storage_client.net_client.processor]
enable_coroutines_pool = true
max_coroutines_num = 256
max_processing_requests_num = 4096
response_compression_level = 1
response_compression_threshold = '128KB
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.read]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.remove]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.truncate]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true

[server.storage_client.traffic_control.write]
max_batch_bytes = '4MB'
max_batch_size = 128
max_concurrent_requests = 32
max_concurrent_requests_per_server = 8
process_batches_in_parallel = true
random_shuffle_requests = true  NODE=local TASK=CreateMetaServiceTask
INFO[0017] Save admin cli config to /tmp/prepare-3fs-config.57VFbH/admin_cli.toml  NODE=local TASK=CreateMetaServiceTask
INFO[0017] Generating fdb.cluster to /tmp/prepare-3fs-config.57VFbH/fdb.cluster  NODE=local TASK=CreateMetaServiceTask
INFO[0017] Copying meta_main configs from /tmp/prepare-3fs-config.57VFbH to local /mnt/nvme3n1/3fs-lab/meta/config.d  NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d:   NODE="<LOCAL>"
DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH:   NODE="<LOCAL>"
INFO[0017] Upload meta_main main config                  NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml'  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml': Succeed
ConfigVersion  1  NODE="<LOCAL>"
INFO[0017] Service meta_main main config uploaded        NODE=local TASK=CreateMetaServiceTask
INFO[0017] Starting meta_main container 3fs-meta         NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml  NODE="<LOCAL>"
DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml: 05d1527d43e491187ab14a66170e261b3bbb109aa78cb8fab5de0e8567900902  NODE="<LOCAL>"
INFO[0017] Waiting for service meta_main active...       NODE=local TASK=CreateMetaServiceTask
DEBU[0017] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes"  NODE="<LOCAL>"
DEBU[0018] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id  Type   Status         Hostname              Pid  Tags  LastHeartbeatTime  ConfigVersion  ReleaseVersion
1   MGMTD  PRIMARY_MGMTD  salab-hpedl380g11-02  1    []    N/A                1(UPTODATE)    250228-dev-1-999999-ee9a5cee  NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes"  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id   Type   Status               Hostname              Pid  Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1    MGMTD  PRIMARY_MGMTD        salab-hpedl380g11-02  1    []    N/A                  1(UPTODATE)    250228-dev-1-999999-ee9a5cee
100  META   HEARTBEAT_CONNECTED  salab-hpedl380g11-02  1    []    2026-06-07 04:20:13  0              250228-dev-1-999999-ee9a5cee  NODE="<LOCAL>"
INFO[0020] Service meta_main is not active               NODE=local TASK=CreateMetaServiceTask
INFO[0020] Started meta_main container 3fs-meta successfully  NODE=local TASK=CreateMetaServiceTask
INFO[0020] Running task CreateStorageServiceTask
DEBU[0020] Node ID map: map[local:10001]                 NODE=local TASK=CreateStorageServiceTask
INFO[0020] Start to run script disk_tool.sh on node      NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX: /tmp/remote-run-script.pwn6yQ  NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ:   NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage:   NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -p /tmp  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.LO9bgmrBXL  NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL:   NODE="<LOCAL>"
INFO[0020] Scp /tmp/remote-run-script.pwn6yQ/tmp_script.sh to /tmp/tmp.LO9bgmrBXL  NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL:   NODE="<LOCAL>"
INFO[0020] Run disk_tool.sh with [/mnt/nvme3n1/3fs-lab/storage 1 dir prepare]  NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare: + '[' 4 -lt 4 ']'
+ MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata
+ DISK_NUM=1
+ DISK_TYPE=dir
+ ACTION=prepare
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata
+ [[ dir == \n\v\m\e ]]
+ [[ prepare == \p\r\e\p\a\r\e ]]
+ prepare_dir_disk
++ seq 0 0
+ for i in $(seq 0 $((DISK_NUM-1)))
+ MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs  NODE="<LOCAL>"
DEBU[0020] Run disk_tool.sh output: + '[' 4 -lt 4 ']'
+ MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata
+ DISK_NUM=1
+ DISK_TYPE=dir
+ ACTION=prepare
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata
+ [[ dir == \n\v\m\e ]]
+ [[ prepare == \p\r\e\p\a\r\e ]]
+ prepare_dir_disk
++ seq 0 0
+ for i in $(seq 0 $((DISK_NUM-1)))
+ MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0
+ mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs  NODE=local TASK=CreateStorageServiceTask
INFO[0020] Run disk_tool.sh success                      NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL:   NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ:   NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.2QLDXp  NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp:   NODE="<LOCAL>"
INFO[0020] Create storage_main config dir /mnt/nvme3n1/3fs-lab/storage  NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage  NODE="<LOCAL>"

[client.rdma_control]
max_concurrent_transmission = 64

[client.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[ib_devices]
allow_no_usable_devices = false
allow_unknown_zone = true
default_network_zone = 'UNKNOWN'
default_pkey_index = 0
default_roce_pkey_index = 0
default_traffic_class = 0
device_filter = []
fork_safe = true
prefer_ibdevice = true
skip_inactive_ports = true
skip_unusable_device = true
subnets = []

[mgmtd_client]
accept_incomplete_routing_info_during_mgmtd_bootstrapping = true
auto_extend_client_session_interval = '10s'
auto_heartbeat_interval = '10s'
auto_refresh_interval = '10s'
enable_auto_extend_client_session = false
enable_auto_heartbeat = true
enable_auto_refresh = true
mgmtd_server_addresses = ["RDMA://10.74.40.86:8000"]
work_queue_size = 100  NODE=local TASK=CreateStorageServiceTask


[server.base.groups.io_worker.ibsocket]
buf_ack_batch = 8
buf_signal_batch = 8
buf_size = 16384
drain_timeout = '5s'
drop_connections = 0
event_ack_batch = 128
max_rd_atomic = 16
max_rdma_wr = 128
max_rdma_wr_per_post = 32
max_sge = 1
min_rnr_timer = 1
record_bytes_per_peer = false
record_latency_per_peer = false
retry_cnt = 7
rnr_retry = 0
send_buf_cnt = 32
sl = 0
start_psn = 0
timeout = 14

[server.base.groups.io_worker.transport_pool]
max_connections = 1

[server.base.groups.listener]
domain_socket_index = 1
filter_list = []
listen_port = 8002
listen_queue_depth = 4096
rdma_accept_timeout = '15s'
rdma_listen_ethernet = true
reuse_port = false

[server.base.groups.processor]
enable_coroutines_pool = true
max_coroutines_num = 256
max_processing_requests_num = 4096
re
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.coroutines_pool_default]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_read]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_sync]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.coroutines_pool_update]
coroutines_num = 64
queue_size = 1024
threads_num = 8

[server.dump_worker]
dump_interval = '1day'
dump_root_path = ''
high_cpu_usage_threshold = 100

[server.forward_client]
default_compression_level = 0
default_compression_threshold = '128KB'
default_log_long_running_threshold = '0ns'
default_report_metrics = false
default_send_retry_times = 1
default_timeout = '1s'
enable_r
enable_work_stealing = false
queue_size = 1024

[server.targets]
allow_disk_without_uuid = true
collect_all_fds = true
create_engine_path = true
space_info_cache_timeout = '5s'
target_num_per_path = 0
target_paths = ["/mnt/3fsdata/data0/3fs"]

[server.targets.storage_target]
force_persist = true
kv_path = ''
migrate_kv_store = false
mutex_num = 257
point_query_strategy = 'NONE'

[server.targets.storage_target.file_store]
preopen_chunk_size_list = []

[server.targets.storage_target.kv_store]
create_if_missing = false
integrate_leveldb_log = false
leveldb_block_cache_size = '8GB'
leveldb_iterator_fill_cache = true
leveldb_shared_block_cache = true
leveldb_sst_file_size = '16MB'
leveldb_write_buffer_size = '16MB'
rocksdb_avoid_flush_during_recovery = false
rocksdb_avoid_flush_during_shutdown = false
rocksdb_avoid_unnecessary_blocking_io = false
rocksdb_block_cache_size = '8GB'
rocksdb_block_size = '4KB'
rocksdb_bloom_filter_bits_per_key = 10
rocksdb_compression = 'kNoCompression'
rocksdb_enable_bloom_filter = true
rocksdb_enable_pipelined_write = false
rocksdb_enable_prefix_transform = true
rocksdb_keep_log_file_num = 10
rocksdb_level0_file_num_compaction_trigger = 4
rocksdb_lowest_used_cache_tier = 'kNonVolatileBlockTier'
rocksdb_max_manifest_file_size = '64MB'
rocksdb_num_levels = 7
rocksdb_prepopulate_block_cache = 'kDisable'
rocksdb_readahead_size = '2MB'
rocksdb_shared_block_cache = true
rocksdb_stats_dump_period = '2min'
rocksdb_target_file_size_base = '64MB'
rocksdb_target_file_size_multiplier = 1
rocksdb_threads_num = 8
rocksdb_unordered_write = false
rocksdb_wal_recovery_mode = 'kTolerateCorruptedTailRecords'
rocksdb_write_buffer_size = '16MB'
sync_when_write = true
type = 'LevelDB'

[server.targets.storage_target.meta_store]
allocate_size = '256MB'
punch_hole_batch_size = 16
recycle_batch_size = 256
removed_chunk_expiration_time = '3day'
removed_chunk_force_recycled_time = '1h'  NODE=local TASK=CreateStorageServiceTask
INFO[0020] Save admin cli config to /tmp/prepare-3fs-config.2QLDXp/admin_cli.toml  NODE=local TASK=CreateStorageServiceTask
INFO[0020] Generating fdb.cluster to /tmp/prepare-3fs-config.2QLDXp/fdb.cluster  NODE=local TASK=CreateStorageServiceTask
INFO[0020] Copying storage_main configs from /tmp/prepare-3fs-config.2QLDXp to local /mnt/nvme3n1/3fs-lab/storage/config.d  NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d:   NODE="<LOCAL>"
DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp  NODE="<LOCAL>"
DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp:   NODE="<LOCAL>"
INFO[0020] Upload storage_main main config               NODE=local TASK=CreateStorageServiceTask
DEBU[0020] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml'  NODE="<LOCAL>"
DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml': Succeed
ConfigVersion  1  NODE="<LOCAL>"
INFO[0021] Service storage_main main config uploaded     NODE=local TASK=CreateStorageServiceTask
INFO[0021] Starting storage_main container 3fs-storage   NODE=local TASK=CreateStorageServiceTask
DEBU[0021] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml  NODE="<LOCAL>"
DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml: bc8b2bb426eabe682f7c232948fad7f057e1a59c6874d727371b4fa60878c9a4  NODE="<LOCAL>"
INFO[0021] Waiting for service storage_main active...    NODE=local TASK=CreateStorageServiceTask
DEBU[0021] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes"  NODE="<LOCAL>"
DEBU[0021] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id   Type   Status               Hostname              Pid  Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1    MGMTD  PRIMARY_MGMTD        salab-hpedl380g11-02  1    []    N/A                  1(UPTODATE)    250228-dev-1-999999-ee9a5cee
100  META   HEARTBEAT_CONNECTED  salab-hpedl380g11-02  1    []    2026-06-07 04:20:13  0              250228-dev-1-999999-ee9a5cee  NODE="<LOCAL>"
DEBU[0023] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes"  NODE="<LOCAL>"
DEBU[0023] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id   Type   Status               Hostname              Pid  Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1    MGMTD  PRIMARY_MGMTD        salab-hpedl380g11-02  1    []    N/A                  1(UPTODATE)    250228-dev-1-999999-ee9a5cee
100  META   HEARTBEAT_CONNECTED  salab-hpedl380g11-02  1    []    2026-06-07 04:20:13  0              250228-dev-1-999999-ee9a5cee  NODE="<LOCAL>"
DEBU[0025] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes"  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id     Type     Status               Hostname              Pid  Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1      MGMTD    PRIMARY_MGMTD        salab-hpedl380g11-02  1    []    N/A                  1(UPTODATE)    250228-dev-1-999999-ee9a5cee
100    META     HEARTBEAT_CONNECTED  salab-hpedl380g11-02  1    []    2026-06-07 04:20:13  0              250228-dev-1-999999-ee9a5cee
10001  STORAGE  HEARTBEAT_CONNECTED  salab-hpedl380g11-02  1    []    2026-06-07 04:20:19  1(UPTODATE)    250228-dev-1-999999-ee9a5cee  NODE="<LOCAL>"
INFO[0026] Service storage_main is not active            NODE=local TASK=CreateStorageServiceTask
INFO[0026] Started storage_main container 3fs-storage successfully  NODE=local TASK=CreateStorageServiceTask
INFO[0026] Creating mount-3fs-disks script and mount-3fs-disks.service service...  NODE=local TASK=CreateStorageServiceTask
DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.P7fFFvwgTu  NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.CIlnSMAAuU  NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU:   NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service: Created symlink /etc/systemd/system/multi-user.target.wants/mount-3fs-disks.service → /usr/lib/systemd/system/mount-3fs-disks.service.  NODE="<LOCAL>"
DEBU[0026] Run command: sudo -S /bin/bash -c systemctl daemon-reload  NODE="<LOCAL>"
DEBU[0026] Output of sudo -S /bin/bash -c systemctl daemon-reload:   NODE="<LOCAL>"
DEBU[0026] Created dir type disk: /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0, index: 0  NODE=local TASK=CreateStorageServiceTask
INFO[0026] Running task InitUserAndChainTask
DEBU[0026] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add--root --admin 0 root"  NODE="<LOCAL>"
DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add --root --admin 0 root": Uid                0
Name               root
Token              AAD1pfWU8QAgLseW2wDulnc0(Expired at N/A)
IsRootUser         true
IsAdmin            true
Gid                0
SupplementaryGids    NODE="<LOCAL>"
DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt"  NODE="<LOCAL>"
DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt":   NODE="<LOCAL>"
DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1  NODE="<LOCAL>"
DEBU[0028] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1: NODE="<LOCAL>"
DEBU[0028] Command failed stacktrace: exit status 1
Traceback (most recent call last):
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in <module>
    main()
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main
    ).run(
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run
    num_loops = self.max_recovery_traffic_on_peer*2
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer
    return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1))
ZeroDivisionError: division by zero

pkg/external/manager.go:50:(*externalBase).run: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1]
pkg/external/docker.go:168:(*dockerExternal).Exec:
pkg/mgmtd/steps.go:239:(*initUserAndChainStep).initChainFiles: run data_placement.py
pkg/mgmtd/steps.go:177:(*initUserAndChainStep).Execute:
pkg/task/task.go:101:(*BaseTask).ExecuteSteps.(*BaseTask).newStepExecuter.func1:
pkg/task/task.go:139:(*BaseTask).ExecuteSteps:
pkg/task/runner.go:252:(*Runner).Run: run task InitUserAndChainTask
cmd/m3fs/cluster.go:211:createCluster: create cluster
2026/06/07 04:20:24 create cluster: run task InitUserAndChainTask: run data_placement.py: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -typeCR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1]: exit status 1
Traceback (most recent call last):
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in <module>
    main()
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main
    ).run(
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run
    num_loops = self.max_recovery_traffic_on_peer*2
  File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer
    return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1))
ZeroDivisionError: division by zero
(3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs#

Originally created by @gaowayne on GitHub (Jun 7, 2026). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/416 hello team below is error ``` (3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:14500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName: nodes: 3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry: Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout:<nil> ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s} INFO[0000] Running task CreatePgClusterTask DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data: NODE="<LOCAL>" DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts: NODE="<LOCAL>" INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql NODE=local TASK=CreatePgClusterTask DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0 NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-9Oql -e POSTGRESQL_REPLICATION_PASSWORD=WvYV7MJh3qKUyfXf -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: e166fa5bf7c46930cd28ac238f6d6a8cb1d42bff0938cf274fb75964865f8226 NODE="<LOCAL>" DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response NODE="<LOCAL>" DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2 NODE=local TASK=CreatePgClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections NODE="<LOCAL>" INFO[0001] Started postgresql container 3fs-postgres successfully NODE=local TASK=CreatePgClusterTask INFO[0001] Running task CreateFdbClusterTask DEBU[0001] fdb cluster file content: 3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 NODE=local TASK=CreateFdbClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data: NODE="<LOCAL>" DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs: NODE="<LOCAL>" DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63 NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=3lxpzmask2:gl6f7qh7ni@10.74.40.86:14500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: 9bace2adf85cd6e1f1c5e6971171570ce8f864411ee588f6e3e0f89bb8e9f59e NODE="<LOCAL>" INFO[0001] Started fdb container 3fs-fdb successfully NODE=local TASK=CreateFdbClusterTask INFO[0001] Initializing fdb cluster NODE=local TASK=CreateFdbClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd' NODE="<LOCAL>" ^C (3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# vim cluster.yml (3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ./bin/m3fs --debug cluster create -c cluster.yml DEBU[0000] Cluster config: &{Name:open3fs-single WorkDir:/mnt/nvme3n1/3fs-lab NetworkType:RXE LogLevel:INFO Nodes:[{Name:local Host:10.74.40.86 Port:22 Username:root Password:0xc00036dd70 RDMAAddresses:[]}] NodeGroups:[] Services:{Pg:{ContainerName:3fs-postgres Nodes:[local] NodeGroups:[] Port:5432 Username:postgres Password:pgpassword Database:open3fs ReadOnlyUsername:postgres_readonly ReadOnlyPassword:pgpassword_readonly WaitReadyTimeout:30s} Fdb:{ContainerName:3fs-fdb Nodes:[local] NodeGroups:[] Port:4500 WaitClusterTimeout:2m0s} Clickhouse:{ContainerName:3fs-clickhouse Nodes:[local] NodeGroups:[] Db:3fs User:default Password:Pass@w0rd TCPPort:8999} Grafana:{ContainerName:3fs-grafana Nodes:[local] NodeGroups:[] Port:3000} Monitor:{ContainerName:3fs-monitor Nodes:[local] NodeGroups:[] Port:10000} Mgmtd:{ContainerName:3fs-mgmtd Nodes:[local] NodeGroups:[] ChunkSize:1048576 StripeSize:16 RDMAListenPort:8000 TCPListenPort:9000 WaitTargetOnlineTimeout:10m0s} Meta:{ContainerName:3fs-meta Nodes:[local] NodeGroups:[] RDMAListenPort:8001 TCPListenPort:9001} Storage:{ContainerName:3fs-storage Nodes:[local] NodeGroups:[] DiskType:dir SectorSize:4096 DiskNumPerNode:1 RDMAListenPort:8002 TCPListenPort:9002 ReplicationFactor:1 TargetNumPerDisk:1 TargetIDPrefix:1 ChainIDPrefix:1 WaitChainsServingTimeout:10m0s WaitTargetUpdateSleep:10s} Client:{ContainerName:3fs-client Nodes:[local] NodeGroups:[] HostMountpoint:/mnt/3fsmount}} Images:{Registry:Arch:amd64 FFFS:{Repo:open3fs/3fs Tag:20250410-avx2} Clickhouse:{Repo:open3fs/clickhouse Tag:25.1-jammy} Grafana:{Repo:open3fs/grafana Tag:12.0.0} Fdb:{Repo:open3fs/foundationdb Tag:7.3.63} Pg:{Repo:open3fs/postgresql Tag:17.5.0}} UI:{TaskInfoColor:green} CmdMaxExitTimeout:<nil> ServiceBasePath:/lib/systemd/system/ CheckStatusTimeout:5m0s CheckStatusInterval:5s} INFO[0000] Running task CreatePgClusterTask DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/data: NODE="<LOCAL>" DEBU[0000] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/postgresql/scripts: NODE="<LOCAL>" INFO[0000] Generated init_db script /mnt/nvme3n1/3fs-lab/postgresql/scripts/init_db.sql NODE=local TASK=CreatePgClusterTask DEBU[0000] Run command: sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0 NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c docker run --name 3fs-postgres --detach --network host -e POSTGRESQL_USERNAME=postgres -e POSTGRESQL_PASSWORD=pgpassword -e POSTGRESQL_REPLICATION_USER=open3fs-replica-A3Pg -e POSTGRESQL_REPLICATION_PASSWORD=d2A6LqJnCLVBLjEr -e POSTGRESQL_PORT_NUMBER=5432 --volume /mnt/nvme3n1/3fs-lab/postgresql/data:/bitnami/postgresql --volume /mnt/nvme3n1/3fs-lab/postgresql/scripts:/docker-entrypoint-initdb.d --restart unless-stopped open3fs/postgresql:17.5.0: fa11a8f1cc8e28698f5616daae0c09694097d86744decce0f330ca763f54b849 NODE="<LOCAL>" DEBU[0000] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE="<LOCAL>" DEBU[0000] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - no response NODE="<LOCAL>" DEBU[0000] Run pg_isready failed: sudo run cmd [cmd: docker exec 3fs-postgres pg_isready -U postgres]: exit status 2 NODE=local TASK=CreatePgClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c docker exec 3fs-postgres pg_isready -U postgres: /var/run/postgresql:5432 - accepting connections NODE="<LOCAL>" INFO[0001] Started postgresql container 3fs-postgres successfully NODE=local TASK=CreatePgClusterTask INFO[0001] Running task CreateFdbClusterTask DEBU[0001] fdb cluster file content: 45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 NODE=local TASK=CreateFdbClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/data: NODE="<LOCAL>" DEBU[0001] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/fdb/logs: NODE="<LOCAL>" DEBU[0001] Run command: sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63 NODE="<LOCAL>" DEBU[0001] Output of sudo -S /bin/bash -c docker run --name 3fs-fdb --detach --network host -e FDB_CLUSTER_FILE_CONTENTS=45dh8lnkka:hi1ldtjncq@10.74.40.86:4500 --volume /mnt/nvme3n1/3fs-lab/fdb/data:/var/fdb/data --volume /mnt/nvme3n1/3fs-lab/fdb/logs:/var/fdb/logs --restart unless-stopped open3fs/foundationdb:7.3.63: d11b3ff5ade923a38c7408a59c3e2322902d34678701143071ee0dc77fc04087 NODE="<LOCAL>" INFO[0001] Started fdb container 3fs-fdb successfully NODE=local TASK=CreateFdbClusterTask INFO[0001] Initializing fdb cluster NODE=local TASK=CreateFdbClusterTask DEBU[0001] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd' NODE="<LOCAL>" DEBU[0004] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'configure new single ssd': Database created NODE="<LOCAL>" INFO[0004] Waiting for fdb cluster initialized NODE=local TASK=CreateFdbClusterTask DEBU[0004] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal' NODE="<LOCAL>" DEBU[0005] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available, but has issues (type 'status' for more information). NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal' NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c docker exec 3fs-fdb fdbcli --exec 'status minimal': The database is available. NODE="<LOCAL>" INFO[0010] Initialized fdb cluster NODE=local TASK=CreateFdbClusterTask INFO[0010] Running task CreateClickhouseClusterTask DEBU[0010] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-clickhouse.XXXXXX: /tmp/3fs-clickhouse.jCQ5nO NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-clickhouse.jCQ5nO: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/data: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/log: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/config.d: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/config.xml /mnt/nvme3n1/3fs-lab/clickhouse/config.d/config.xml: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/clickhouse/sql: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-clickhouse.jCQ5nO/3fs-monitor.sql /mnt/nvme3n1/3fs-lab/clickhouse/sql/3fs-monitor.sql: NODE="<LOCAL>" DEBU[0010] Run command: sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy NODE="<LOCAL>" DEBU[0010] Output of sudo -S /bin/bash -c docker run --name 3fs-clickhouse --detach --network host -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=Pass@w0rd --volume /mnt/nvme3n1/3fs-lab/clickhouse/data:/var/lib/clickhouse --volume /mnt/nvme3n1/3fs-lab/clickhouse/log:/var/log/clickhouse-server --volume /mnt/nvme3n1/3fs-lab/clickhouse/config.d:/etc/clickhouse-server/config.d --volume /mnt/nvme3n1/3fs-lab/clickhouse/sql:/tmp/sql --restart unless-stopped open3fs/clickhouse:25.1-jammy: e63b4ffd3e8cf6f8be7d72d88f40ffe8a3c80d9502fe5d83945e899613a77861 NODE="<LOCAL>" INFO[0015] Started clickhouse container 3fs-clickhouse successfully NODE=local TASK=CreateClickhouseClusterTask INFO[0015] Initializing clickhouse cluster NODE=local TASK=CreateClickhouseClusterTask DEBU[0015] Run command: sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql" NODE="<LOCAL>" DEBU[0015] Output of sudo -S /bin/bash -c docker exec 3fs-clickhouse bash -c "clickhouse-client --port 8999 -n < /tmp/sql/3fs-monitor.sql": NODE="<LOCAL>" INFO[0015] Initialized clickhouse cluster NODE=local TASK=CreateClickhouseClusterTask DEBU[0015] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO NODE="<LOCAL>" DEBU[0015] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-clickhouse.jCQ5nO: NODE="<LOCAL>" INFO[0015] Running task CreateGrafanaServiceTask INFO[0015] Generating datasource.yaml NODE=local TASK=CreateGrafanaServiceTask DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources NODE="<LOCAL>" DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/datasources: NODE="<LOCAL>" INFO[0015] Generating dashboard.yaml NODE=local TASK=CreateGrafanaServiceTask DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards NODE="<LOCAL>" DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards: NODE="<LOCAL>" INFO[0015] Generating 3fs.json NODE=local TASK=CreateGrafanaServiceTask DEBU[0015] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards NODE="<LOCAL>" DEBU[0015] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/grafana/dashboards: NODE="<LOCAL>" DEBU[0015] Run command: sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0 NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-grafana --detach --network host --volume /mnt/nvme3n1/3fs-lab/grafana/datasources:/etc/grafana/provisioning/datasources --volume /mnt/nvme3n1/3fs-lab/grafana/dashboards:/etc/grafana/provisioning/dashboards --restart unless-stopped open3fs/grafana:12.0.0: 849e83e1e65ee4aeb3f4e18efdf5c1cad63ba2ce9d562e07ee03c9d4145691cc NODE="<LOCAL>" INFO[0016] Started grafana container 3fs-grafana successfully, service endpoint is http://10.74.40.86:3000, login with username "admin" and password "admin" NODE=local TASK=CreateGrafanaServiceTask INFO[0016] Running task CreateMonitorTask DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-monitor.XXXXXX: /tmp/3fs-monitor.stFhHX NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-monitor.stFhHX: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/etc: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-monitor.stFhHX/monitor_collector_main.toml /mnt/nvme3n1/3fs-lab/monitor/etc/monitor_collector_main.toml: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/monitor/log: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-monitor --detach --network host --privileged --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/monitor/etc:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/monitor/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/monitor_collector_main --cfg /opt/3fs/etc/monitor_collector_main.toml: 475e4148833690ba401f2855e1742f3e8fb71bb2e375deea3bf6aefdbcc93acc NODE="<LOCAL>" INFO[0016] Started monitor container 3fs-monitor successfully NODE=local TASK=CreateMonitorTask DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-monitor.stFhHX: NODE="<LOCAL>" INFO[0016] Running task CreateMgmtdServiceTask DEBU[0016] Node ID map: map[local:1] NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Admin cli config template data: map[ClusterID:open3fs-single MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]] NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.DoAci2 NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2 NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.DoAci2: NODE="<LOCAL>" INFO[0016] Create mgmtd_main config dir /mnt/nvme3n1/3fs-lab/mgmtd NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd: NODE="<LOCAL>" DEBU[0016] Template data of mgmtd_main_app.toml.tmpl: map[NodeID:1] NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Generating mgmtd_main_app.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_app.toml NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Config of mgmtd_main_app.toml: allow_empty_node_id = true node_id = 1 NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Template data of mgmtd_main_launcher.toml.tmpl: map[ClusterID:open3fs-single HostMountpoint:/mnt/3fsmount MgmtdServerAddresses:["RDMA://10.74.40.86:8000"]] NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Generating mgmtd_main_launcher.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main_launcher.toml NODE=local TASK=CreateMgmtdServiceTask clusterFile = '/opt/3fs/etc/fdb.cluster' default_backoff = 0 enableMultipleClient = false externalClientDir = '' externalClientPath = '' multipleClientThreadNum = 4 readonly = false trace_file = '' trace_format = 'json' [ib_devices] allow_no_usable_devices = false allow_unknown_zone = true default_network_zone = 'UNKNOWN' default_pkey_index = 0 default_roce_pkey_index = 0 default_traffic_class = 0 device_filter = [] fork_safe = true prefer_ibdevice = true skip_inactive_ports = true skip_unusable_device = true subnets = [] [kv_engine] use_memkv = false [kv_engine.fdb] casual_read_risky = false clusterFile = '' default_backoff = 0 enableMultipleClient = false externalClientDir = '' externalClientPath = '' multipleClientThreadNum = 4 readonly = false trace_file = '' trace_format = 'json' NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Template data of mgmtd_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8000 TCPListenPort:9000] NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Generating mgmtd_main.toml to /tmp/prepare-3fs-config.DoAci2/mgmtd_main.toml NODE=local TASK=CreateMgmtdServiceTask [common.monitor] collect_period = '1s' num_collectors = 1 [[common.monitor.reporters]] type = 'monitor_collector' [common.monitor.reporters.monitor_collector] remote_ip = "10.74.40.86:10000" [server.base.independent_thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [server.base.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [[server.base.groups]] check_connections_interval = '1min' connection_expiration_time = '1day' network_type = 'RDMA' services = [ 'Mgmtd' ] use_independent_thread_pool = false [server.service] allow_heartbeat_from_unregistered = true authenticate = false bootstrapping_length = '2min' bump_routing_info_version_interval = '5s' check_status_interval = '10s' client_session_timeout = '20min' enable_routinginfo_cache = true extend_lease_check_release_version = true extend_lease_interval = '10s' heartbeat_fail_interval = '1min' heartbeat_ignore_stale_targets = true heartbeat_ignore_unknown_targets = false heartbeat_timestamp_valid_window = '30s' lease_length = '1min' new_chain_bootstrap_interval = '2min' only_accept_client_uuid = false retry_times_on_txn_errors = -1 send_heartbeat = true send_heartbeat_interval = '10s' suspicious_lease_interval = '20s' target_info_load_interval = '1s' target_info_persist_batch = 1000 target_info_persist_interval = '1s' try_adjust_target_order_as_preferred = false update_chains_interval = '1s' update_metrics_interval = '1s' validate_lease_on_write = true [server.service.retry_transaction] max_backoff = '1s' max_retry_count = 10 [server.service.user_cache] buckets = 127 exist_ttl = '5min' inexist_ttl = '10s' NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Save admin cli config to /tmp/prepare-3fs-config.DoAci2/admin_cli.toml NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Generating fdb.cluster to /tmp/prepare-3fs-config.DoAci2/fdb.cluster NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Copying mgmtd_main configs from /tmp/prepare-3fs-config.DoAci2 to local /mnt/nvme3n1/3fs-lab/mgmtd/config.d NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.DoAci2 /mnt/nvme3n1/3fs-lab/mgmtd/config.d: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2 NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.DoAci2: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/mgmtd/log: NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16' NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --network host --entrypoint '' --rm --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml 'init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16': Init filesystem, root directory layout: chain table ChainTableId(1), chunksize 1048576, stripesize 16 Init config for MGMTD version 1 NODE="<LOCAL>" INFO[0016] Cluster initialization success NODE=local TASK=CreateMgmtdServiceTask INFO[0016] Starting mgmtd_main container 3fs-mgmtd NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Run command: sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c docker run --name 3fs-mgmtd --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/mgmtd/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/mgmtd/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/mgmtd_main --launcher_cfg /opt/3fs/etc/mgmtd_main_launcher.toml --app_cfg /opt/3fs/etc/mgmtd_main_app.toml: cec202504223f22561a2b9a90f37f918146ca0389adb2ced6d7c79fcd6968934 NODE="<LOCAL>" INFO[0016] Started mgmtd_main container 3fs-mgmtd successfully NODE=local TASK=CreateMgmtdServiceTask DEBU[0016] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX NODE="<LOCAL>" DEBU[0016] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t 3fs-mgmtd.XXXXXX: /tmp/3fs-mgmtd.65wChW NODE="<LOCAL>" DEBU[0016] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/3fs-mgmtd.65wChW: NODE="<LOCAL>" DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/3fs-mgmtd.65wChW/admin_cli.sh /mnt/nvme3n1/3fs-lab/admin_cli.sh: NODE="<LOCAL>" DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/3fs-mgmtd.65wChW: NODE="<LOCAL>" INFO[0017] Running task CreateMetaServiceTask DEBU[0017] Node ID map: map[local:100] NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.57VFbH NODE="<LOCAL>" DEBU[0017] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.57VFbH: NODE="<LOCAL>" INFO[0017] Create meta_main config dir /mnt/nvme3n1/3fs-lab/meta NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/meta: NODE="<LOCAL>" DEBU[0017] Template data of meta_main_app.toml.tmpl: map[NodeID:100] NODE=local TASK=CreateMetaServiceTask INFO[0017] Generating meta_main_app.toml to /tmp/prepare-3fs-config.57VFbH/meta_main_app.toml NODE=local TASK=CreateMetaServiceTask DEBU[0017] Config of meta_main_app.toml: allow_empty_node_id = true node_id = 100 NODE=local TASK=CreateMetaServiceTask [client.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [ib_devices] allow_no_usable_devices = false allow_unknown_zone = true default_network_zone = 'UNKNOWN' default_pkey_index = 0 default_roce_pkey_index = 0 default_traffic_class = 0 device_filter = [] fork_safe = true prefer_ibdevice = true skip_inactive_ports = true skip_unusable_device = true subnets = [] [mgmtd_client] accept_incomplete_routing_info_during_mgmtd_bootstrapping = true auto_extend_client_session_interval = '10s' auto_heartbeat_interval = '10s' auto_refresh_interval = '10s' enable_auto_extend_client_session = false enable_auto_heartbeat = true enable_auto_refresh = true mgmtd_server_addresses = ["RDMA://10.74.40.86:8000"] work_queue_size = 100 NODE=local TASK=CreateMetaServiceTask DEBU[0017] Template data of meta_main.toml.tmpl: map[LogLevel:INFO MgmtdProtocol:RDMA MgmtdServerAddresses:["RDMA://10.74.40.86:8000"] MonitorRemoteIP:10.74.40.86:10000 RDMAListenPort:8001 TCPListenPort:9001] NODE=local TASK=CreateMetaServiceTask INFO[0017] Generating meta_main.toml to /tmp/prepare-3fs-config.57VFbH/meta_main.toml NODE=local TASK=CreateMetaServiceTask [server.background_client.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [server.base.independent_thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [server.base.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [[server.base.groups]] check_connections_interval = '1min' connection_expiration_time = '1day' network_type = 'RDMA' services = [ 'Me externalClientDir = '' externalClientPath = '' multipleClientThreadNum = 4 readonly = false trace_file = '' trace_format = 'json' [server.meta] acl_cache_time = '15s' allow_directly_move_to_trash = false allow_owner_change_immutable = false allow_stat_deleted_inodes = true authenticate = false batch_stat_by_path_concurrent = 4 batch_stat_concurrent = 8 check_file_hole = false dynamic_stripe = false dynamic_stripe_growth = 2 dynamic_stripe_initial = 16 enable_new_chunk_engine = false grv_cache = false idempotent_record_clean = '1min' idempotent_record_expire = '30min' idempotent_remove = true idempotent_rename = false iflags_chain_allocation = false iflags_chunk_engine = true ignore_length_hint = false inodeId_abort_on_duplicate = false inodeId_check_unique = true list_default_limit = 128 max_batch_operations = 4096 max_directory_depth = 64 max_remove_chunks_per_request = 32 max_symlink_count = 10 max_symlink_depth = 4 operation_timeout = '5s' otrunc_replace_file = true otrunc_replace_file rdma_connect_timeout = '5s' read_write_rdma_in_event_thread = false read_write_tcp_in_event_thread = false tcp_connect_timeout = '1s' wait_to_retry_send = '100ms' [server.storage_client.net_client.io_worker.connect_concurrency_limiter] max_concurrency = 4 [server.storage_client.net_client.io_worker.ibsocket] buf_ack_batch = 8 buf_signal_batch = 8 buf_size = 16384 drain_timeout = '5s' drop_connections = 0 event_ack_batch = 128 max_rd_atomic = 16 max_rdma_wr = 128 max_rdma_wr_per_post = 32 max_sge = 1 min_rnr_timer = 1 record_bytes_per_peer = false record_latency_per_peer = false retry_cnt = 7 rnr_retry = 0 send_buf_cnt = 32 sl = 0 start_psn = 0 timeout = 14 [server.storage_client.net_client.io_worker.transport_pool] max_connections = 1 [server.storage_client.net_client.processor] enable_coroutines_pool = true max_coroutines_num = 256 max_processing_requests_num = 4096 response_compression_level = 1 response_compression_threshold = '128KB max_concurrent_requests = 32 max_concurrent_requests_per_server = 8 process_batches_in_parallel = true random_shuffle_requests = true [server.storage_client.traffic_control.read] max_batch_bytes = '4MB' max_batch_size = 128 max_concurrent_requests = 32 max_concurrent_requests_per_server = 8 process_batches_in_parallel = true random_shuffle_requests = true [server.storage_client.traffic_control.remove] max_batch_bytes = '4MB' max_batch_size = 128 max_concurrent_requests = 32 max_concurrent_requests_per_server = 8 process_batches_in_parallel = true random_shuffle_requests = true [server.storage_client.traffic_control.truncate] max_batch_bytes = '4MB' max_batch_size = 128 max_concurrent_requests = 32 max_concurrent_requests_per_server = 8 process_batches_in_parallel = true random_shuffle_requests = true [server.storage_client.traffic_control.write] max_batch_bytes = '4MB' max_batch_size = 128 max_concurrent_requests = 32 max_concurrent_requests_per_server = 8 process_batches_in_parallel = true random_shuffle_requests = true NODE=local TASK=CreateMetaServiceTask INFO[0017] Save admin cli config to /tmp/prepare-3fs-config.57VFbH/admin_cli.toml NODE=local TASK=CreateMetaServiceTask INFO[0017] Generating fdb.cluster to /tmp/prepare-3fs-config.57VFbH/fdb.cluster NODE=local TASK=CreateMetaServiceTask INFO[0017] Copying meta_main configs from /tmp/prepare-3fs-config.57VFbH to local /mnt/nvme3n1/3fs-lab/meta/config.d NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.57VFbH /mnt/nvme3n1/3fs-lab/meta/config.d: NODE="<LOCAL>" DEBU[0017] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.57VFbH: NODE="<LOCAL>" INFO[0017] Upload meta_main main config NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml' NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta-cfg-OOfRyH --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type META --file /opt/3fs/etc/meta_main.toml': Succeed ConfigVersion 1 NODE="<LOCAL>" INFO[0017] Service meta_main main config uploaded NODE=local TASK=CreateMetaServiceTask INFO[0017] Starting meta_main container 3fs-meta NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml NODE="<LOCAL>" DEBU[0017] Output of sudo -S /bin/bash -c docker run --name 3fs-meta --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/meta/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/meta/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/meta_main --launcher_cfg /opt/3fs/etc/meta_main_launcher.toml --app_cfg /opt/3fs/etc/meta_main_app.toml: 05d1527d43e491187ab14a66170e261b3bbb109aa78cb8fab5de0e8567900902 NODE="<LOCAL>" INFO[0017] Waiting for service meta_main active... NODE=local TASK=CreateMetaServiceTask DEBU[0017] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE="<LOCAL>" DEBU[0018] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c docker exec 3fs-meta /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee 100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE="<LOCAL>" INFO[0020] Service meta_main is not active NODE=local TASK=CreateMetaServiceTask INFO[0020] Started meta_main container 3fs-meta successfully NODE=local TASK=CreateMetaServiceTask INFO[0020] Running task CreateStorageServiceTask DEBU[0020] Node ID map: map[local:10001] NODE=local TASK=CreateStorageServiceTask INFO[0020] Start to run script disk_tool.sh on node NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t remote-run-script.XXXXXX: /tmp/remote-run-script.pwn6yQ NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/remote-run-script.pwn6yQ: NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage: NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.LO9bgmrBXL NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.LO9bgmrBXL: NODE="<LOCAL>" INFO[0020] Scp /tmp/remote-run-script.pwn6yQ/tmp_script.sh to /tmp/tmp.LO9bgmrBXL NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/remote-run-script.pwn6yQ/tmp_script.sh /tmp/tmp.LO9bgmrBXL: NODE="<LOCAL>" INFO[0020] Run disk_tool.sh with [/mnt/nvme3n1/3fs-lab/storage 1 dir prepare] NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c bash /tmp/tmp.LO9bgmrBXL /mnt/nvme3n1/3fs-lab/storage 1 dir prepare: + '[' 4 -lt 4 ']' + MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata + DISK_NUM=1 + DISK_TYPE=dir + ACTION=prepare + mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata + [[ dir == \n\v\m\e ]] + [[ prepare == \p\r\e\p\a\r\e ]] + prepare_dir_disk ++ seq 0 0 + for i in $(seq 0 $((DISK_NUM-1))) + MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0 + mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs NODE="<LOCAL>" DEBU[0020] Run disk_tool.sh output: + '[' 4 -lt 4 ']' + MOUNT_DIR=/mnt/nvme3n1/3fs-lab/storage/3fsdata + DISK_NUM=1 + DISK_TYPE=dir + ACTION=prepare + mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata + [[ dir == \n\v\m\e ]] + [[ prepare == \p\r\e\p\a\r\e ]] + prepare_dir_disk ++ seq 0 0 + for i in $(seq 0 $((DISK_NUM-1))) + MOUNT_POINT=/mnt/nvme3n1/3fs-lab/storage/3fsdata/data0 + mkdir -p /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0/3fs NODE=local TASK=CreateStorageServiceTask INFO[0020] Run disk_tool.sh success NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c rm -f /tmp/tmp.LO9bgmrBXL: NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/remote-run-script.pwn6yQ: NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c mktemp -d -p /tmp -t prepare-3fs-config.XXXXXX: /tmp/prepare-3fs-config.2QLDXp NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c chmod 0777 /tmp/prepare-3fs-config.2QLDXp: NODE="<LOCAL>" INFO[0020] Create storage_main config dir /mnt/nvme3n1/3fs-lab/storage NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/storage NODE="<LOCAL>" [client.rdma_control] max_concurrent_transmission = 64 [client.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [ib_devices] allow_no_usable_devices = false allow_unknown_zone = true default_network_zone = 'UNKNOWN' default_pkey_index = 0 default_roce_pkey_index = 0 default_traffic_class = 0 device_filter = [] fork_safe = true prefer_ibdevice = true skip_inactive_ports = true skip_unusable_device = true subnets = [] [mgmtd_client] accept_incomplete_routing_info_during_mgmtd_bootstrapping = true auto_extend_client_session_interval = '10s' auto_heartbeat_interval = '10s' auto_refresh_interval = '10s' enable_auto_extend_client_session = false enable_auto_heartbeat = true enable_auto_refresh = true mgmtd_server_addresses = ["RDMA://10.74.40.86:8000"] work_queue_size = 100 NODE=local TASK=CreateStorageServiceTask [server.base.groups.io_worker.ibsocket] buf_ack_batch = 8 buf_signal_batch = 8 buf_size = 16384 drain_timeout = '5s' drop_connections = 0 event_ack_batch = 128 max_rd_atomic = 16 max_rdma_wr = 128 max_rdma_wr_per_post = 32 max_sge = 1 min_rnr_timer = 1 record_bytes_per_peer = false record_latency_per_peer = false retry_cnt = 7 rnr_retry = 0 send_buf_cnt = 32 sl = 0 start_psn = 0 timeout = 14 [server.base.groups.io_worker.transport_pool] max_connections = 1 [server.base.groups.listener] domain_socket_index = 1 filter_list = [] listen_port = 8002 listen_queue_depth = 4096 rdma_accept_timeout = '15s' rdma_listen_ethernet = true reuse_port = false [server.base.groups.processor] enable_coroutines_pool = true max_coroutines_num = 256 max_processing_requests_num = 4096 re num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [server.coroutines_pool_default] coroutines_num = 64 queue_size = 1024 threads_num = 8 [server.coroutines_pool_read] coroutines_num = 64 queue_size = 1024 threads_num = 8 [server.coroutines_pool_sync] coroutines_num = 64 queue_size = 1024 threads_num = 8 [server.coroutines_pool_update] coroutines_num = 64 queue_size = 1024 threads_num = 8 [server.dump_worker] dump_interval = '1day' dump_root_path = '' high_cpu_usage_threshold = 100 [server.forward_client] default_compression_level = 0 default_compression_threshold = '128KB' default_log_long_running_threshold = '0ns' default_report_metrics = false default_send_retry_times = 1 default_timeout = '1s' enable_r enable_work_stealing = false queue_size = 1024 [server.targets] allow_disk_without_uuid = true collect_all_fds = true create_engine_path = true space_info_cache_timeout = '5s' target_num_per_path = 0 target_paths = ["/mnt/3fsdata/data0/3fs"] [server.targets.storage_target] force_persist = true kv_path = '' migrate_kv_store = false mutex_num = 257 point_query_strategy = 'NONE' [server.targets.storage_target.file_store] preopen_chunk_size_list = [] [server.targets.storage_target.kv_store] create_if_missing = false integrate_leveldb_log = false leveldb_block_cache_size = '8GB' leveldb_iterator_fill_cache = true leveldb_shared_block_cache = true leveldb_sst_file_size = '16MB' leveldb_write_buffer_size = '16MB' rocksdb_avoid_flush_during_recovery = false rocksdb_avoid_flush_during_shutdown = false rocksdb_avoid_unnecessary_blocking_io = false rocksdb_block_cache_size = '8GB' rocksdb_block_size = '4KB' rocksdb_bloom_filter_bits_per_key = 10 rocksdb_compression = 'kNoCompression' rocksdb_enable_bloom_filter = true rocksdb_enable_pipelined_write = false rocksdb_enable_prefix_transform = true rocksdb_keep_log_file_num = 10 rocksdb_level0_file_num_compaction_trigger = 4 rocksdb_lowest_used_cache_tier = 'kNonVolatileBlockTier' rocksdb_max_manifest_file_size = '64MB' rocksdb_num_levels = 7 rocksdb_prepopulate_block_cache = 'kDisable' rocksdb_readahead_size = '2MB' rocksdb_shared_block_cache = true rocksdb_stats_dump_period = '2min' rocksdb_target_file_size_base = '64MB' rocksdb_target_file_size_multiplier = 1 rocksdb_threads_num = 8 rocksdb_unordered_write = false rocksdb_wal_recovery_mode = 'kTolerateCorruptedTailRecords' rocksdb_write_buffer_size = '16MB' sync_when_write = true type = 'LevelDB' [server.targets.storage_target.meta_store] allocate_size = '256MB' punch_hole_batch_size = 16 recycle_batch_size = 256 removed_chunk_expiration_time = '3day' removed_chunk_force_recycled_time = '1h' NODE=local TASK=CreateStorageServiceTask INFO[0020] Save admin cli config to /tmp/prepare-3fs-config.2QLDXp/admin_cli.toml NODE=local TASK=CreateStorageServiceTask INFO[0020] Generating fdb.cluster to /tmp/prepare-3fs-config.2QLDXp/fdb.cluster NODE=local TASK=CreateStorageServiceTask INFO[0020] Copying storage_main configs from /tmp/prepare-3fs-config.2QLDXp to local /mnt/nvme3n1/3fs-lab/storage/config.d NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c cp -r /tmp/prepare-3fs-config.2QLDXp /mnt/nvme3n1/3fs-lab/storage/config.d: NODE="<LOCAL>" DEBU[0020] Run command: sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp NODE="<LOCAL>" DEBU[0020] Output of sudo -S /bin/bash -c rm -fr /tmp/prepare-3fs-config.2QLDXp: NODE="<LOCAL>" INFO[0020] Upload storage_main main config NODE=local TASK=CreateStorageServiceTask DEBU[0020] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml' NODE="<LOCAL>" DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage-cfg-hZgekS --network host --entrypoint '' --rm --privileged --ulimit nofile=1048576:1048576 --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev open3fs/3fs:20250410-avx2 /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' 'set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml': Succeed ConfigVersion 1 NODE="<LOCAL>" INFO[0021] Service storage_main main config uploaded NODE=local TASK=CreateStorageServiceTask INFO[0021] Starting storage_main container 3fs-storage NODE=local TASK=CreateStorageServiceTask DEBU[0021] Run command: sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml NODE="<LOCAL>" DEBU[0021] Output of sudo -S /bin/bash -c docker run --name 3fs-storage --detach --network host --privileged --ulimit nofile=1048576:1048576 --volume /dev:/dev --volume /mnt/nvme3n1/3fs-lab/storage/config.d:/opt/3fs/etc/ --volume /mnt/nvme3n1/3fs-lab/storage/log:/var/log/3fs --volume /mnt/nvme3n1/3fs-lab/storage/3fsdata:/mnt/3fsdata --volume /mnt/nvme3n1/3fs-lab/bin/ibdev2netdev:/usr/sbin/ibdev2netdev --restart unless-stopped open3fs/3fs:20250410-avx2 /opt/3fs/bin/storage_main --launcher_cfg /opt/3fs/etc/storage_main_launcher.toml --app_cfg /opt/3fs/etc/storage_main_app.toml: bc8b2bb426eabe682f7c232948fad7f057e1a59c6874d727371b4fa60878c9a4 NODE="<LOCAL>" INFO[0021] Waiting for service storage_main active... NODE=local TASK=CreateStorageServiceTask DEBU[0021] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE="<LOCAL>" DEBU[0021] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee 100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE="<LOCAL>" DEBU[0023] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE="<LOCAL>" DEBU[0023] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee 100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee NODE="<LOCAL>" DEBU[0025] Run command: sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c docker exec 3fs-storage /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes": Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-hpedl380g11-02 1 [] N/A 1(UPTODATE) 250228-dev-1-999999-ee9a5cee 100 META HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:13 0 250228-dev-1-999999-ee9a5cee 10001 STORAGE HEARTBEAT_CONNECTED salab-hpedl380g11-02 1 [] 2026-06-07 04:20:19 1(UPTODATE) 250228-dev-1-999999-ee9a5cee NODE="<LOCAL>" INFO[0026] Service storage_main is not active NODE=local TASK=CreateStorageServiceTask INFO[0026] Started storage_main container 3fs-storage successfully NODE=local TASK=CreateStorageServiceTask INFO[0026] Creating mount-3fs-disks script and mount-3fs-disks.service service... NODE=local TASK=CreateStorageServiceTask DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.P7fFFvwgTu NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.P7fFFvwgTu: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /mnt/nvme3n1/3fs-lab/bin: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.P7fFFvwgTu /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.P7fFFvwgTu: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c chmod +x /mnt/nvme3n1/3fs-lab/bin/mount-3fs-disks: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c mktemp -p /tmp NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c mktemp -p /tmp: /tmp/tmp.CIlnSMAAuU NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c chmod 0777 /tmp/tmp.CIlnSMAAuU: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c mkdir -m 0777 -p /lib/systemd/system: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c cp -r /tmp/tmp.CIlnSMAAuU /lib/systemd/system/mount-3fs-disks.service: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c rm -fr /tmp/tmp.CIlnSMAAuU: NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c systemctl enable mount-3fs-disks.service: Created symlink /etc/systemd/system/multi-user.target.wants/mount-3fs-disks.service → /usr/lib/systemd/system/mount-3fs-disks.service. NODE="<LOCAL>" DEBU[0026] Run command: sudo -S /bin/bash -c systemctl daemon-reload NODE="<LOCAL>" DEBU[0026] Output of sudo -S /bin/bash -c systemctl daemon-reload: NODE="<LOCAL>" DEBU[0026] Created dir type disk: /mnt/nvme3n1/3fs-lab/storage/3fsdata/data0, index: 0 NODE=local TASK=CreateStorageServiceTask INFO[0026] Running task InitUserAndChainTask DEBU[0026] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add--root --admin 0 root" NODE="<LOCAL>" DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://10.74.40.86:8000"]' "user-add --root --admin 0 root": Uid 0 Name root Token AAD1pfWU8QAgLseW2wDulnc0(Expired at N/A) IsRootUser true IsAdmin true Gid 0 SupplementaryGids NODE="<LOCAL>" DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt" NODE="<LOCAL>" DEBU[0027] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd bash -c "echo AAD1pfWU8QAgLseW2wDulnc0 > /opt/3fs/etc/token.txt": NODE="<LOCAL>" DEBU[0027] Run command: sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1 NODE="<LOCAL>" DEBU[0028] Output of sudo -S /bin/bash -c docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1: NODE="<LOCAL>" DEBU[0028] Command failed stacktrace: exit status 1 Traceback (most recent call last): File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in <module> main() File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main ).run( File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run num_loops = self.max_recovery_traffic_on_peer*2 File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1)) ZeroDivisionError: division by zero pkg/external/manager.go:50:(*externalBase).run: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1] pkg/external/docker.go:168:(*dockerExternal).Exec: pkg/mgmtd/steps.go:239:(*initUserAndChainStep).initChainFiles: run data_placement.py pkg/mgmtd/steps.go:177:(*initUserAndChainStep).Execute: pkg/task/task.go:101:(*BaseTask).ExecuteSteps.(*BaseTask).newStepExecuter.func1: pkg/task/task.go:139:(*BaseTask).ExecuteSteps: pkg/task/runner.go:252:(*Runner).Run: run task InitUserAndChainTask cmd/m3fs/cluster.go:211:createCluster: create cluster 2026/06/07 04:20:24 create cluster: run task InitUserAndChainTask: run data_placement.py: sudo run cmd [cmd: docker exec 3fs-mgmtd python3 /opt/3fs/data_placement/src/model/data_placement.py -ql -relax -typeCR --num_nodes 1 --replication_factor 1 --min_targets_per_disk 1]: exit status 1 Traceback (most recent call last): File "/opt/3fs/data_placement/src/model/data_placement.py", line 549, in <module> main() File "/opt/3fs/data_placement/src/model/data_placement.py", line 515, in main ).run( File "/opt/3fs/data_placement/src/model/data_placement.py", line 120, in run num_loops = self.max_recovery_traffic_on_peer*2 File "/opt/3fs/data_placement/src/model/data_placement.py", line 100, in max_recovery_traffic_on_peer return math.ceil(self.sum_recovery_traffic_per_failure / (self.num_nodes-1)) ZeroDivisionError: division by zero (3fs) root@salab-hpedl380g11-02:/mnt/nvme0n1/wayne/3fs/m3fs# ```

bsakaguchi commented 2026-10-26 09:47:40 +00:00

Author

Owner

Copy link

@hiSandog commented on GitHub (Jun 25, 2026):

The pasted config is helpful, but please redact secrets and include the few log lines immediately after the data-placement failure. One suspicious detail is NetworkType: RXE with RDMAAddresses:[]; if placement expects RDMA/RXE addresses to be discovered or declared, an empty address list could fail later in a non-obvious place. It would also help to say whether the same single-node config works with a non-RDMA/TCP mode, so the deployment issue can be separated from RXE setup.

<!-- gh-comment-id:4799031800 --> @hiSandog commented on GitHub (Jun 25, 2026): The pasted config is helpful, but please redact secrets and include the few log lines immediately after the data-placement failure. One suspicious detail is `NetworkType: RXE` with `RDMAAddresses:[]`; if placement expects RDMA/RXE addresses to be discovered or declared, an empty address list could fail later in a non-obvious place. It would also help to say whether the same single-node config works with a non-RDMA/TCP mode, so the deployment issue can be separated from RXE setup.

bsakaguchi referenced this issue 2026-10-26 09:48:53 +00:00

[PR #256] [MERGED] Fix index out of bounds issue in batch_remove #344

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

SF-Zhou-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#238

No description provided.