deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #39] [failed to setup single node demo] can expert write one single node run minimum system for study code and architecture. #23

New issue
Open
opened 2026-10-26 09:43:34 +00:00 by bsakaguchi · 24 comments
bsakaguchi commented 2026-10-26 09:43:34 +00:00
Owner
Copy link

Originally created by @gaowayne on GitHub (Mar 2, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/39

it will be great that create one single node deployment steps with concise steps.

Originally created by @gaowayne on GitHub (Mar 2, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/39 it will be great that create one single node deployment steps with concise steps.
bsakaguchi commented 2026-10-26 09:43:35 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 2, 2025):

There is a test script that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with sleep infinity. Then run it using the command: bash run.sh <path to 3fs binary> <path to test directory>. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem.

Note: 3FS still requires RDMA network even for single-node deployment.

<!-- gh-comment-id:2692722696 --> @yiyuanliu commented on GitHub (Mar 2, 2025): There is a [test script](https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh) that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with `sleep infinity`. Then run it using the command: `bash run.sh <path to 3fs binary> <path to test directory>`. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem. Note: 3FS still requires RDMA network even for single-node deployment.
bsakaguchi commented 2026-10-26 09:43:35 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 2, 2025):

There is a test script that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with sleep infinity. Then run it using the command: bash run.sh <path to 3fs binary> <path to test directory>. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem.

Note: 3FS still requires RDMA network even for single-node deployment.

that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍

<!-- gh-comment-id:2692730304 --> @gaowayne commented on GitHub (Mar 2, 2025): > There is a [test script](https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh) that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with `sleep infinity`. Then run it using the command: `bash run.sh <path to 3fs binary> <path to test directory>`. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem. > > Note: 3FS still requires RDMA network even for single-node deployment. that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍
bsakaguchi commented 2026-10-26 09:43:35 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 2, 2025):

There is a test script that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with sleep infinity. Then run it using the command: bash run.sh <path to 3fs binary> <path to test directory>. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem.
Note: 3FS still requires RDMA network even for single-node deployment.

that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍

No, you still need an RDMA network even when using FUSE. Our storage servers only provide RDMA-based data access interfaces.

<!-- gh-comment-id:2692734365 --> @yiyuanliu commented on GitHub (Mar 2, 2025): > > There is a [test script](https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh) that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with `sleep infinity`. Then run it using the command: `bash run.sh <path to 3fs binary> <path to test directory>`. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem. > > Note: 3FS still requires RDMA network even for single-node deployment. > > that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍 No, you still need an RDMA network even when using FUSE. Our storage servers only provide RDMA-based data access interfaces.
bsakaguchi commented 2026-10-26 09:43:36 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 8, 2025):

There is a test script that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with sleep infinity. Then run it using the command: bash run.sh <path to 3fs binary> <path to test directory>. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem.
Note: 3FS still requires RDMA network even for single-node deployment.

that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍

No, you still need an RDMA network even when using FUSE. Our storage servers only provide RDMA-based data access interfaces.

buddy to double check with you. I need remove all scripts code
between

echo "Fuse mounted"

and

echo "### test client io during update chain table"
(head -n 1 ${CONFIG}/chain-table.csv && tail -n +2 ${CONFIG}/chain-table.csv | shuf) > ${CONFIG}/chain-table-2.csv
${ADMIN_CLI} upload-chain-table 1 ${CONFIG}/chain-table-2.csv --desc replica-1-version-2
${ADMIN_CLI} list-chain-tables

echo "### test chattr +i"
sudo chattr +i ${MOUNT1}/test
lsattr ${MOUNT1}
sudo chattr -i ${MOUNT1}/test

replace all these with sleep infinity, is this correct?

<!-- gh-comment-id:2708308366 --> @gaowayne commented on GitHub (Mar 8, 2025): > > > There is a [test script](https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh) that performs functionality testing by deploying a simple file system on a single node. You can modify this script by replacing all test commands with `sleep infinity`. Then run it using the command: `bash run.sh <path to 3fs binary> <path to test directory>`. The script will start all necessary processes locally (fdb, mgmtd, meta, storage) and mount the filesystem with FUSE. When the script exits, it will attempt to kill all processes and unmount the filesystem. > > > Note: 3FS still requires RDMA network even for single-node deployment. > > > > > > that is great. if I want to just test fuse client, not native client, can I skip RDMA network? I already build code well, I will try different options. 👍 > > No, you still need an RDMA network even when using FUSE. Our storage servers only provide RDMA-based data access interfaces. buddy to double check with you. I need remove all scripts code between ``` echo "Fuse mounted" ```` and ``` echo "### test client io during update chain table" (head -n 1 ${CONFIG}/chain-table.csv && tail -n +2 ${CONFIG}/chain-table.csv | shuf) > ${CONFIG}/chain-table-2.csv ${ADMIN_CLI} upload-chain-table 1 ${CONFIG}/chain-table-2.csv --desc replica-1-version-2 ${ADMIN_CLI} list-chain-tables echo "### test chattr +i" sudo chattr +i ${MOUNT1}/test lsattr ${MOUNT1} sudo chattr -i ${MOUNT1}/test ``` replace all these with sleep infinity, is this correct?
bsakaguchi commented 2026-10-26 09:43:36 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 8, 2025):

@yiyuanliu buddy, I am trying to run the tests directly. but I suffer below error.

root@salab-bncbeta03:~/wayne/3fs/3FS/tests/fuse# bash run1.sh /root/wayne/3fs/3FS/build/bin /mnt/nvme2n1/3fsdir
### check network
NIC address 10.74.40.68
FDB port 8179
MGMTD port 9605
### check directory and file
### create subdirectory
generate app config
generate config
- generate admin_cli.toml
- generate hf3fs_fuse_main_launcher.toml
- generate hf3fs_fuse_main.toml
- generate meta_bench.toml
- generate meta_main_launcher.toml
- generate meta_main.toml
- generate mgmtd_main_launcher.toml
- generate mgmtd_main.toml
- generate pjdfstest.toml
- generate storage_main_launcher.toml
- generate storage_main.toml
### create foundationdb
fdbserver: /usr/sbin/fdbserver
fdbclient: /usr/bin/fdbcli
start fdb @ port 8179
sleep 5s then create database
Database created
sleep 5s then check foundationdb status
The database is available.
### init cluster
admin_cli: /root/wayne/3fs/3FS/build/bin/admin_cli --cfg /mnt/nvme2n1/3fsdir/config/admin_cli.toml --
Uid                0
Name               root
Token              AADHHSOs8QA92iRe2wB1fmuL(Expired at N/A)
IsRootUser         true
IsAdmin            true
Gid                0
SupplementaryGids  
Uid                0
Name               root
Token              AAD11kWy8QABtiJq2wDOIh8i(Expired at N/A)
Token              AADHHSOs8QA92iRe2wB1fmuL(Expired at 2025-03-15 07:28:03)
IsRootUser         true
IsAdmin            true
Gid                0
SupplementaryGids  

Uid  Name  Token                     IsRoot  IsAdmin  Gids
0    root  AAD11kWy8QABtiJq2wDOIh8i  true    true     

Init filesystem, root directory layout: chain table ChainTableId(1), chunksize 524288, stripesize 8

Init config for MGMTD version 1
Init config for META version 1
Init config for STORAGE version 1
Init config for FUSE version 1
### start mgmtd
run1.sh: line 240: 66373 Aborted                 (core dumped) LOG_FILE=${LOG}/mgmtd.log PORT=${MGMTD_PORT} ${BINARY}/mgmtd_main --app_cfg ${CONFIG}/mgmtd_main_app.toml --launcher_cfg ${CONFIG}/mgmtd_main_launcher.toml --cfg ${CONFIG}/mgmtd_main.toml > ${LOG}/mgmtd_main.stdout 2> ${LOG}/mgmtd_main.stderr

### start meta & storage
run1.sh: line 265: 66409 Aborted                 (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr
run1.sh: line 265: 66410 Aborted                 (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr
run1.sh: line 265: 66412 Aborted                 (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr
run1.sh: line 265: 66416 Aborted                 (core dumped) LOG_FILE=${LOG}/meta-${i}.log PORT=0 ${BINARY}/meta_main --app_cfg ${CONFIG}/meta_main_app_$(expr 50 + ${i}).toml --launcher_cfg ${CONFIG}/meta_main_launcher.toml --cfg ${CONFIG}/meta_main.toml > ${LOG}/meta_main-${i}.stdout 2> ${LOG}/meta_main-${i}.stderr
run1.sh: line 265: 66417 Aborted                 (core dumped) LOG_FILE=${LOG}/meta-${i}.log PORT=0 ${BINARY}/meta_main --app_cfg ${CONFIG}/meta_main_app_$(expr 50 + ${i}).toml --launcher_cfg ${CONFIG}/meta_main_launcher.toml --cfg ${CONFIG}/meta_main.toml > ${LOG}/meta_main-${i}.stdout 2> ${LOG}/meta_main-${i}.stderr

### list nodes
*** Aborted at 1741418903 (Unix time, try 'date -d @1741418903') ***
*** Signal 6 (SIGABRT) (0x10420) received by PID 66592 (pthread TID 0x706d78e22700) (linux TID 66592) (maybe from PID 66592, UID 0) (code: -6), stack trace: ***
    @ 0000000001fd4daf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
<!-- gh-comment-id:2708310178 --> @gaowayne commented on GitHub (Mar 8, 2025): @yiyuanliu buddy, I am trying to run the tests directly. but I suffer below error. ``` root@salab-bncbeta03:~/wayne/3fs/3FS/tests/fuse# bash run1.sh /root/wayne/3fs/3FS/build/bin /mnt/nvme2n1/3fsdir ### check network NIC address 10.74.40.68 FDB port 8179 MGMTD port 9605 ### check directory and file ### create subdirectory generate app config generate config - generate admin_cli.toml - generate hf3fs_fuse_main_launcher.toml - generate hf3fs_fuse_main.toml - generate meta_bench.toml - generate meta_main_launcher.toml - generate meta_main.toml - generate mgmtd_main_launcher.toml - generate mgmtd_main.toml - generate pjdfstest.toml - generate storage_main_launcher.toml - generate storage_main.toml ### create foundationdb fdbserver: /usr/sbin/fdbserver fdbclient: /usr/bin/fdbcli start fdb @ port 8179 sleep 5s then create database Database created sleep 5s then check foundationdb status The database is available. ### init cluster admin_cli: /root/wayne/3fs/3FS/build/bin/admin_cli --cfg /mnt/nvme2n1/3fsdir/config/admin_cli.toml -- Uid 0 Name root Token AADHHSOs8QA92iRe2wB1fmuL(Expired at N/A) IsRootUser true IsAdmin true Gid 0 SupplementaryGids Uid 0 Name root Token AAD11kWy8QABtiJq2wDOIh8i(Expired at N/A) Token AADHHSOs8QA92iRe2wB1fmuL(Expired at 2025-03-15 07:28:03) IsRootUser true IsAdmin true Gid 0 SupplementaryGids Uid Name Token IsRoot IsAdmin Gids 0 root AAD11kWy8QABtiJq2wDOIh8i true true Init filesystem, root directory layout: chain table ChainTableId(1), chunksize 524288, stripesize 8 Init config for MGMTD version 1 Init config for META version 1 Init config for STORAGE version 1 Init config for FUSE version 1 ### start mgmtd run1.sh: line 240: 66373 Aborted (core dumped) LOG_FILE=${LOG}/mgmtd.log PORT=${MGMTD_PORT} ${BINARY}/mgmtd_main --app_cfg ${CONFIG}/mgmtd_main_app.toml --launcher_cfg ${CONFIG}/mgmtd_main_launcher.toml --cfg ${CONFIG}/mgmtd_main.toml > ${LOG}/mgmtd_main.stdout 2> ${LOG}/mgmtd_main.stderr ### start meta & storage run1.sh: line 265: 66409 Aborted (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr run1.sh: line 265: 66410 Aborted (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr run1.sh: line 265: 66412 Aborted (core dumped) LOG_FILE=${LOG}/storage-${i}.log PORT=0 STORAGE_DIR=${DATA}/storage-${i}/ ${BINARY}/storage_main --app_cfg ${CONFIG}/storage_main_app_$(expr 10000 + ${i}).toml --launcher_cfg ${CONFIG}/storage_main_launcher.toml --cfg ${CONFIG}/storage_main.toml > ${LOG}/storage_main-${i}.stdout 2> ${LOG}/storage_main-${i}.stderr run1.sh: line 265: 66416 Aborted (core dumped) LOG_FILE=${LOG}/meta-${i}.log PORT=0 ${BINARY}/meta_main --app_cfg ${CONFIG}/meta_main_app_$(expr 50 + ${i}).toml --launcher_cfg ${CONFIG}/meta_main_launcher.toml --cfg ${CONFIG}/meta_main.toml > ${LOG}/meta_main-${i}.stdout 2> ${LOG}/meta_main-${i}.stderr run1.sh: line 265: 66417 Aborted (core dumped) LOG_FILE=${LOG}/meta-${i}.log PORT=0 ${BINARY}/meta_main --app_cfg ${CONFIG}/meta_main_app_$(expr 50 + ${i}).toml --launcher_cfg ${CONFIG}/meta_main_launcher.toml --cfg ${CONFIG}/meta_main.toml > ${LOG}/meta_main-${i}.stdout 2> ${LOG}/meta_main-${i}.stderr ### list nodes *** Aborted at 1741418903 (Unix time, try 'date -d @1741418903') *** *** Signal 6 (SIGABRT) (0x10420) received by PID 66592 (pthread TID 0x706d78e22700) (linux TID 66592) (maybe from PID 66592, UID 0) (code: -6), stack trace: *** @ 0000000001fd4daf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 ```
bsakaguchi commented 2026-10-26 09:43:37 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 8, 2025):

I do have one RDMA NIC there. :)

see my ibstat:

root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/log# ibstat
CA 'mlx5_0'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.37.1700
        Hardware version: 0
        Node GUID: 0x88e9a4ffff70566e
        System image GUID: 0x88e9a4ffff70566e
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x8ae9a4fffe70566e
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.37.1700
        Hardware version: 0
        Node GUID: 0x88e9a4ffff70566f
        System image GUID: 0x88e9a4ffff70566e
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x8ae9a4fffe70566f
                Link layer: Ethernet

[2025-03-08T07:28:03.697984722+00:00 mgmtd_main:66373 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_0 port 1 ==> ens1f0np0 (Down)
[2025-03-08T07:28:03.697984722+00:00 mgmtd_main:66373 IBDevice.cc:169 INFO] mlx5_1 port 1 ==> ens1f1np1 (Down)
[2025-03-08T07:28:03.698084200+00:00 mgmtd_main:66373 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_0 => ens1f0np0
[2025-03-08T07:28:03.698101705+00:00 mgmtd_main:66373 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_1 => ens1f1np1
[2025-03-08T07:28:03.699837640+00:00 mgmtd_main:66373 IfAddrs.h:102 INFO] Get ifaddr of ens27f0np0, addr 10.74.40.68/24, subnet 10.74.40.0/24, up true
[2025-03-08T07:28:03.710611968+00:00 mgmtd_main:66373 IBDevice.cc:341 WARNING] IBDevice mlx5_0 port 1 is not active, state IBV_PORT_DOWN, skip true
[2025-03-08T07:28:03.710631347+00:00 mgmtd_main:66373 IBDevice.cc:253 INFO] IBDevice skip mlx5_0 because it doesn't have available ports.
[2025-03-08T07:28:03.717253998+00:00 mgmtd_main:66373 IBDevice.cc:341 WARNING] IBDevice mlx5_1 port 1 is not active, state IBV_PORT_DOWN, skip true
[2025-03-08T07:28:03.717263528+00:00 mgmtd_main:66373 IBDevice.cc:253 INFO] IBDevice skip mlx5_1 because it doesn't have available ports.
[2025-03-08T07:28:03.717742308+00:00 mgmtd_main:66373 IBDevice.cc:694 ERROR] IBManager can't find available device!
[2025-03-08T07:28:03.717780958+00:00 mgmtd_main:66373 ServerLauncher.h:42 FATAL] Failed to start IBManager: RPC::IBDeviceNotFound(2016)
*** Aborted at 1741418883 (Unix time, try 'date -d @1741418883') ***
*** Signal 6 (SIGABRT) (0x10345) received by PID 66373 (pthread TID 0x71a28841b600) (linux TID 66373) (maybe from PID 66373, UID 0) (code: -6), stack trace: ***
    @ 0000000000740dcf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 000000000004251f (unknown)
    @ 00000000000969fc pthread_kill
    @ 0000000000042475 raise
    @ 00000000000287f2 abort
    @ 00000000007a8a90 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71
    @ 00000000007a7f46 _ZN5folly18LogStreamProcessor6logNowEv
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190
    @ 00000000007a8199 _ZN5folly16LogStreamVoidifyILb1EEanERSo
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222
    @ 000000000087596a _ZN5hf3fs19TwoPhaseApplicationINS_5mgmtd11MgmtdServerEE15initApplicationEv
                       /root/wayne/3fs/3FS/src/core/app/ServerLauncher.h:42
                       -> /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp
    @ 00000000004bcd16 _ZN5hf3fs15ApplicationBase3runEiPPc
                       /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65
    @ 0000000000872a8d main
                       /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp:7
    @ 0000000000029d8f (unknown)
    @ 0000000000029e3f __libc_start_main
    @ 0000000000219e24 _start
<!-- gh-comment-id:2708311285 --> @gaowayne commented on GitHub (Mar 8, 2025): I do have one RDMA NIC there. :) see my ibstat: ``` root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/log# ibstat CA 'mlx5_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.37.1700 Hardware version: 0 Node GUID: 0x88e9a4ffff70566e System image GUID: 0x88e9a4ffff70566e Port 1: State: Down Physical state: Disabled Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x8ae9a4fffe70566e Link layer: Ethernet CA 'mlx5_1' CA type: MT4125 Number of ports: 1 Firmware version: 22.37.1700 Hardware version: 0 Node GUID: 0x88e9a4ffff70566f System image GUID: 0x88e9a4ffff70566e Port 1: State: Down Physical state: Disabled Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x8ae9a4fffe70566f Link layer: Ethernet ``` ``` [2025-03-08T07:28:03.697984722+00:00 mgmtd_main:66373 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_0 port 1 ==> ens1f0np0 (Down) [2025-03-08T07:28:03.697984722+00:00 mgmtd_main:66373 IBDevice.cc:169 INFO] mlx5_1 port 1 ==> ens1f1np1 (Down) [2025-03-08T07:28:03.698084200+00:00 mgmtd_main:66373 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_0 => ens1f0np0 [2025-03-08T07:28:03.698101705+00:00 mgmtd_main:66373 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_1 => ens1f1np1 [2025-03-08T07:28:03.699837640+00:00 mgmtd_main:66373 IfAddrs.h:102 INFO] Get ifaddr of ens27f0np0, addr 10.74.40.68/24, subnet 10.74.40.0/24, up true [2025-03-08T07:28:03.710611968+00:00 mgmtd_main:66373 IBDevice.cc:341 WARNING] IBDevice mlx5_0 port 1 is not active, state IBV_PORT_DOWN, skip true [2025-03-08T07:28:03.710631347+00:00 mgmtd_main:66373 IBDevice.cc:253 INFO] IBDevice skip mlx5_0 because it doesn't have available ports. [2025-03-08T07:28:03.717253998+00:00 mgmtd_main:66373 IBDevice.cc:341 WARNING] IBDevice mlx5_1 port 1 is not active, state IBV_PORT_DOWN, skip true [2025-03-08T07:28:03.717263528+00:00 mgmtd_main:66373 IBDevice.cc:253 INFO] IBDevice skip mlx5_1 because it doesn't have available ports. [2025-03-08T07:28:03.717742308+00:00 mgmtd_main:66373 IBDevice.cc:694 ERROR] IBManager can't find available device! [2025-03-08T07:28:03.717780958+00:00 mgmtd_main:66373 ServerLauncher.h:42 FATAL] Failed to start IBManager: RPC::IBDeviceNotFound(2016) *** Aborted at 1741418883 (Unix time, try 'date -d @1741418883') *** *** Signal 6 (SIGABRT) (0x10345) received by PID 66373 (pthread TID 0x71a28841b600) (linux TID 66373) (maybe from PID 66373, UID 0) (code: -6), stack trace: *** @ 0000000000740dcf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 @ 000000000004251f (unknown) @ 00000000000969fc pthread_kill @ 0000000000042475 raise @ 00000000000287f2 abort @ 00000000007a8a90 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71 @ 00000000007a7f46 _ZN5folly18LogStreamProcessor6logNowEv /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190 @ 00000000007a8199 _ZN5folly16LogStreamVoidifyILb1EEanERSo /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222 @ 000000000087596a _ZN5hf3fs19TwoPhaseApplicationINS_5mgmtd11MgmtdServerEE15initApplicationEv /root/wayne/3fs/3FS/src/core/app/ServerLauncher.h:42 -> /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp @ 00000000004bcd16 _ZN5hf3fs15ApplicationBase3runEiPPc /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65 @ 0000000000872a8d main /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp:7 @ 0000000000029d8f (unknown) @ 0000000000029e3f __libc_start_main @ 0000000000219e24 _start ```
bsakaguchi commented 2026-10-26 09:43:38 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 11, 2025):

@yiyuanliu buddym I fixed RDMA NIC, I can run ib_send_bw well in the NIC.

now I run below commands
bash run1.sh /root/wayne/3fs/3FS/build/bin /mnt/nvme2n1/3fsdir

I still see below error

[2025-03-11T05:03:26.780714878+00:00 meta_main:81998 ApplicationBase.cc:214 INFO] Try to load config from file [/mnt/nvme2n1/3fsdir/config/meta_main_app_51.toml][2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] Full AppConfig:
[2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] allow_empty_node_id = false
[2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] node_id = 51
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] Full LauncherConfig:[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_dev_version = true
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] cluster_id = 'ci_test'[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_compression_level = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_compression_threshold = '128KB' 
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_log_long_running_threshold = '0ns'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_report_metrics = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_send_retry_times = 1
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_timeout = '1s'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_rdma_control = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] force_use_tcp = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] 
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_event_loop = 1
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] rdma_connect_timeout = '5s'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] read_write_rdma_in_event_thread = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] read_write_tcp_in_event_thread = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] tcp_connect_timeout = '1s'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] wait_to_retry_send = '100ms'[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.connect_concurrency_limiter][2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_concurrency = 4
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] 
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.ibsocket][2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_ack_batch = 8
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_signal_batch = 8[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_size = 16384
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] drain_timeout = '5s'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] drop_connections = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] event_ack_batch = 128 
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rd_atomic = 16
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rdma_wr = 128[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rdma_wr_per_post = 32
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_sge = 16
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] min_rnr_timer = 1
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] record_bytes_per_peer = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] record_latency_per_peer = false 
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] retry_cnt = 7
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] rnr_retry = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] send_buf_cnt = 32
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] sl = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] start_psn = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] timeout = 14
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.transport_pool]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_connections = 1
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.processor]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_coroutines_pool = true
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_coroutines_num = 256
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_processing_requests_num = 4096
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] response_compression_level = 1
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] response_compression_threshold = '128KB'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.rdma_control]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_concurrent_transmission = 64
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.thread_pool]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] bg_thread_pool_stratetry = 'SHARED_QUEUE'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] collect_stats = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_work_stealing = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] io_thread_pool_stratetry = 'SHARED_QUEUE'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_bg_threads = 2
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_connect_threads = 2
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_io_threads = 2
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_proc_threads = 2
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] proc_thread_pool_stratetry = 'SHARED_QUEUE'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [ib_devices]
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_no_usable_devices = false
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_unknown_zone = true
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_network_zone = 'UNKNOWN'
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_pkey_index = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_roce_pkey_index = 0
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] subnets = []
[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] auto_refresh_interval = '10s'
[2025-03-11T05:03:26.899201198+00:00 meta_main:81998 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_0 port 1 ==> ens1f0np0 (Up)
[2025-03-11T05:03:26.899201198+00:00 meta_main:81998 IBDevice.cc:169 INFO] mlx5_1 port 1 ==> ens1f1np1 (Down)
[2025-03-11T05:03:26.899407457+00:00 meta_main:81998 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_0 => ens1f0np0
[2025-03-11T05:03:26.900519707+00:00 meta_main:81998 IfAddrs.h:102 INFO] Get ifaddr of ens1f0np0, addr 192.168.1.101/24, subnet 192.168.1.0/24, up true
[2025-03-11T05:03:26.915503624+00:00 meta_main:81998 IBDevice.cc:396 INFO] IBDevice mlx5_0:1 IP addrs 192.168.1.101/24
[2025-03-11T05:03:26.915551884+00:00 meta_main:81998 IBDevice.cc:441 CRITICAL] IBDevice mlx5_0:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-11T05:03:26.915715748+00:00 meta_main:81998 IBDevice.cc:367 INFO] IBDevice mlx5_0 add active port 1, linklayer ETHERNET, addrs 192.168.1.101/24, zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:c0:a8:1:65
[2025-03-11T05:03:26.915735383+00:00 meta_main:81998 IBDevice.cc:256 INFO] IBDevice add mlx5_0, id 0, 1 available ports
[2025-03-11T05:03:26.930228755+00:00 meta_main:81998 IBDevice.cc:341 WARNING] IBDevice mlx5_1 port 1 is not active, state IBV_PORT_DOWN, skip true
[2025-03-11T05:03:26.930337623+00:00 meta_main:81998 IBDevice.cc:253 INFO] IBDevice skip mlx5_1 because it doesn't have available ports.
[2025-03-11T05:03:26.936929193+00:00 IBManager:82177 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-03-11T05:03:26.937153354+00:00 meta_main:81998 ServerLauncher.h:43 INFO] IBDevice inited
[2025-03-11T05:03:26.937197380+00:00 meta_main:81998 Utils.cc:239 INFO] Start load AppInfo from launcher
[2025-03-11T05:03:26.937342369+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct serde::Services
[2025-03-11T05:03:26.944655891+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup
[2025-03-11T05:03:26.961066851+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-03-11T05:03:26.971831332+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-03-11T05:03:26.982513256+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-03-11T05:03:27.002484684+00:00 meta_main:81998 Semaphore.h:27 INFO] usableTokens: 64, maxTokens: 4096, availableTokens: 64
[2025-03-11T05:03:27.007065854+00:00 meta_main:81998 MgmtdClient.cc:222 INFO] MgmtdClient: started
[2025-03-11T05:03:27.007097329+00:00 launcherEL0:82221 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-03-11T05:03:27.007378919+00:00 CliBG1:82213 MgmtdClient.cc:390 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.1] start execution. from:
[2025-03-11T05:03:27.007571502+00:00 CliBG1:82213 MgmtdClient.cc:424 INFO] MgmtdClient: start probe for connecting ...
[2025-03-11T05:03:27.007595082+00:00 CliBG1:82213 MgmtdClient.cc:542 INFO] MgmtdClient: start probe from unknown address RDMA://192.168.1.101:9860
[2025-03-11T05:03:27.007628202+00:00 CliBG1:82213 MgmtdClient.cc:504 INFO] MgmtdClient: probePrimary NodeId(0) via RDMA://192.168.1.101:9860 ...
[2025-03-11T05:03:27.018499277+00:00 CliConn0:82218 TcpSocket.cc:34 ERROR] connect TCP://192.168.1.101:9860 failed: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused)
[2025-03-11T05:03:27.018612854+00:00 CliConn0:82218 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused)
[2025-03-11T05:03:27.127104034+00:00 CliConn0:82218 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused)
[2025-03-11T05:03:27.127633688+00:00 CliConn1:82220 IBConnect.cc:368 ERROR] IBSocket [connect RDMA://192.168.1.101:9860] failed to query, error RPC::SendFailed(2007), timeout 5s
[2025-03-11T05:03:27.127709438+00:00 CliConn1:82220 IOWorker.cc:89 ERROR] error: RPC::SendFailed(2007)
[2025-03-11T05:03:27.232160033+00:00 CliConn1:82220 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused)
[2025-03-11T05:03:27.334542442+00:00 CliConn1:82220 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused)
[2025-03-11T05:03:27.334892732+00:00 CliConn0:82218 IOWorker.cc:89 ERROR] error: RPC::SendFailed(2007)
[2025-03-11T05:03:27.335236754+00:00 CliBG1:82213 MgmtdClient.cc:509 ERROR] MgmtdClient: probePrimary NodeId(0) failed: RPC::SendFailed(2007)
[2025-03-11T05:03:27.335340339+00:00 CliBG1:82213 MgmtdClient.cc:162 INFO] MgmtdClient: mark node as Skipped NodeId(0) ["RDMA://192.168.1.101:9860"]
[2025-03-11T05:03:27.335364419+00:00 CliBG1:82213 MgmtdClient.cc:434 WARNING] MgmtdClient: primary not found
[2025-03-11T05:03:27.335930298+00:00 CliBG1:82213 MgmtdClient.cc:390 ERROR] [MgmtdClientOp RefreshRoutingInfo force=false No.1] failed: MgmtdClient::PrimaryMgmtdNotFound(6000) ./src/client/mgmtd/MgmtdClient.cc:173, '_result' has error. latency: 0ns
[2025-03-11T05:03:27.336142164+00:00 meta_main:81998 MgmtdClient.cc:271 INFO] MgmtdClient: wait for stop
[2025-03-11T05:03:27.336196578+00:00 CliBG1:82213 MgmtdClient.cc:381 INFO] MgmtdClient: receive ExitItem
[2025-03-11T05:03:27.336293718+00:00 meta_main:81998 MgmtdClient.cc:277 INFO] MgmtdClient: stopped
[2025-03-11T05:03:27.336360529+00:00 meta_main:81998 Utils.cc:254 CRITICAL] Load AppInfo from launcher failed: MgmtdClient::PrimaryMgmtdNotFound(6000) ./src/client/mgmtd/MgmtdClient.cc:173, '_result' has error
[2025-03-11T05:03:27.336360529+00:00 meta_main:81998 Utils.cc:254 CRITICAL] retryCount: 0
[2025-03-11T05:03:27.346630104+00:00 meta_main:81998 MgmtdClient.cc:222 INFO] MgmtdClient: started
<!-- gh-comment-id:2713939847 --> @gaowayne commented on GitHub (Mar 11, 2025): @yiyuanliu buddym I fixed RDMA NIC, I can run ib_send_bw well in the NIC. now I run below commands bash run1.sh /root/wayne/3fs/3FS/build/bin /mnt/nvme2n1/3fsdir I still see below error ``` [2025-03-11T05:03:26.780714878+00:00 meta_main:81998 ApplicationBase.cc:214 INFO] Try to load config from file [/mnt/nvme2n1/3fsdir/config/meta_main_app_51.toml][2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] Full AppConfig: [2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] allow_empty_node_id = false [2025-03-11T05:03:26.781446659+00:00 meta_main:81998 ServerLauncher.h:38 INFO] node_id = 51 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] Full LauncherConfig:[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_dev_version = true [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] cluster_id = 'ci_test'[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_compression_level = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_compression_threshold = '128KB' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_log_long_running_threshold = '0ns' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_report_metrics = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_send_retry_times = 1 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_timeout = '1s' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_rdma_control = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] force_use_tcp = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_event_loop = 1 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] rdma_connect_timeout = '5s' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] read_write_rdma_in_event_thread = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] read_write_tcp_in_event_thread = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] tcp_connect_timeout = '1s' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] wait_to_retry_send = '100ms'[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.connect_concurrency_limiter][2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_concurrency = 4 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.ibsocket][2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_ack_batch = 8 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_signal_batch = 8[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] buf_size = 16384 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] drain_timeout = '5s' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] drop_connections = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] event_ack_batch = 128 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rd_atomic = 16 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rdma_wr = 128[2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_rdma_wr_per_post = 32 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_sge = 16 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] min_rnr_timer = 1 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] record_bytes_per_peer = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] record_latency_per_peer = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] retry_cnt = 7 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] rnr_retry = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] send_buf_cnt = 32 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] sl = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] start_psn = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] timeout = 14 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.io_worker.transport_pool] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_connections = 1 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.processor] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_coroutines_pool = true [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_coroutines_num = 256 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_processing_requests_num = 4096 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] response_compression_level = 1 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] response_compression_threshold = '128KB' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.rdma_control] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] max_concurrent_transmission = 64 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [client.thread_pool] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] bg_thread_pool_stratetry = 'SHARED_QUEUE' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] collect_stats = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] enable_work_stealing = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] io_thread_pool_stratetry = 'SHARED_QUEUE' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_bg_threads = 2 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_connect_threads = 2 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_io_threads = 2 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] num_proc_threads = 2 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] proc_thread_pool_stratetry = 'SHARED_QUEUE' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] [ib_devices] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_no_usable_devices = false [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] allow_unknown_zone = true [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_network_zone = 'UNKNOWN' [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_pkey_index = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] default_roce_pkey_index = 0 [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] subnets = [] [2025-03-11T05:03:26.781705624+00:00 meta_main:81998 ServerLauncher.h:39 INFO] auto_refresh_interval = '10s' [2025-03-11T05:03:26.899201198+00:00 meta_main:81998 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_0 port 1 ==> ens1f0np0 (Up) [2025-03-11T05:03:26.899201198+00:00 meta_main:81998 IBDevice.cc:169 INFO] mlx5_1 port 1 ==> ens1f1np1 (Down) [2025-03-11T05:03:26.899407457+00:00 meta_main:81998 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_0 => ens1f0np0 [2025-03-11T05:03:26.900519707+00:00 meta_main:81998 IfAddrs.h:102 INFO] Get ifaddr of ens1f0np0, addr 192.168.1.101/24, subnet 192.168.1.0/24, up true [2025-03-11T05:03:26.915503624+00:00 meta_main:81998 IBDevice.cc:396 INFO] IBDevice mlx5_0:1 IP addrs 192.168.1.101/24 [2025-03-11T05:03:26.915551884+00:00 meta_main:81998 IBDevice.cc:441 CRITICAL] IBDevice mlx5_0:1 can't set zone by IP, fallback to UNKNOWN [2025-03-11T05:03:26.915715748+00:00 meta_main:81998 IBDevice.cc:367 INFO] IBDevice mlx5_0 add active port 1, linklayer ETHERNET, addrs 192.168.1.101/24, zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:c0:a8:1:65 [2025-03-11T05:03:26.915735383+00:00 meta_main:81998 IBDevice.cc:256 INFO] IBDevice add mlx5_0, id 0, 1 available ports [2025-03-11T05:03:26.930228755+00:00 meta_main:81998 IBDevice.cc:341 WARNING] IBDevice mlx5_1 port 1 is not active, state IBV_PORT_DOWN, skip true [2025-03-11T05:03:26.930337623+00:00 meta_main:81998 IBDevice.cc:253 INFO] IBDevice skip mlx5_1 because it doesn't have available ports. [2025-03-11T05:03:26.936929193+00:00 IBManager:82177 EventLoop.cc:116 INFO] EventLoop::loop() started. [2025-03-11T05:03:26.937153354+00:00 meta_main:81998 ServerLauncher.h:43 INFO] IBDevice inited [2025-03-11T05:03:26.937197380+00:00 meta_main:81998 Utils.cc:239 INFO] Start load AppInfo from launcher [2025-03-11T05:03:26.937342369+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct serde::Services [2025-03-11T05:03:26.944655891+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup [2025-03-11T05:03:26.961066851+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-03-11T05:03:26.971831332+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-03-11T05:03:26.982513256+00:00 meta_main:81998 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-03-11T05:03:27.002484684+00:00 meta_main:81998 Semaphore.h:27 INFO] usableTokens: 64, maxTokens: 4096, availableTokens: 64 [2025-03-11T05:03:27.007065854+00:00 meta_main:81998 MgmtdClient.cc:222 INFO] MgmtdClient: started [2025-03-11T05:03:27.007097329+00:00 launcherEL0:82221 EventLoop.cc:116 INFO] EventLoop::loop() started. [2025-03-11T05:03:27.007378919+00:00 CliBG1:82213 MgmtdClient.cc:390 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.1] start execution. from: [2025-03-11T05:03:27.007571502+00:00 CliBG1:82213 MgmtdClient.cc:424 INFO] MgmtdClient: start probe for connecting ... [2025-03-11T05:03:27.007595082+00:00 CliBG1:82213 MgmtdClient.cc:542 INFO] MgmtdClient: start probe from unknown address RDMA://192.168.1.101:9860 [2025-03-11T05:03:27.007628202+00:00 CliBG1:82213 MgmtdClient.cc:504 INFO] MgmtdClient: probePrimary NodeId(0) via RDMA://192.168.1.101:9860 ... [2025-03-11T05:03:27.018499277+00:00 CliConn0:82218 TcpSocket.cc:34 ERROR] connect TCP://192.168.1.101:9860 failed: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused) [2025-03-11T05:03:27.018612854+00:00 CliConn0:82218 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused) [2025-03-11T05:03:27.127104034+00:00 CliConn0:82218 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused) [2025-03-11T05:03:27.127633688+00:00 CliConn1:82220 IBConnect.cc:368 ERROR] IBSocket [connect RDMA://192.168.1.101:9860] failed to query, error RPC::SendFailed(2007), timeout 5s [2025-03-11T05:03:27.127709438+00:00 CliConn1:82220 IOWorker.cc:89 ERROR] error: RPC::SendFailed(2007) [2025-03-11T05:03:27.232160033+00:00 CliConn1:82220 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused) [2025-03-11T05:03:27.334542442+00:00 CliConn1:82220 IOWorker.cc:91 ERROR] error: RPC::ConnectFailed(2014) error: folly::AsyncSocketException: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused) [2025-03-11T05:03:27.334892732+00:00 CliConn0:82218 IOWorker.cc:89 ERROR] error: RPC::SendFailed(2007) [2025-03-11T05:03:27.335236754+00:00 CliBG1:82213 MgmtdClient.cc:509 ERROR] MgmtdClient: probePrimary NodeId(0) failed: RPC::SendFailed(2007) [2025-03-11T05:03:27.335340339+00:00 CliBG1:82213 MgmtdClient.cc:162 INFO] MgmtdClient: mark node as Skipped NodeId(0) ["RDMA://192.168.1.101:9860"] [2025-03-11T05:03:27.335364419+00:00 CliBG1:82213 MgmtdClient.cc:434 WARNING] MgmtdClient: primary not found [2025-03-11T05:03:27.335930298+00:00 CliBG1:82213 MgmtdClient.cc:390 ERROR] [MgmtdClientOp RefreshRoutingInfo force=false No.1] failed: MgmtdClient::PrimaryMgmtdNotFound(6000) ./src/client/mgmtd/MgmtdClient.cc:173, '_result' has error. latency: 0ns [2025-03-11T05:03:27.336142164+00:00 meta_main:81998 MgmtdClient.cc:271 INFO] MgmtdClient: wait for stop [2025-03-11T05:03:27.336196578+00:00 CliBG1:82213 MgmtdClient.cc:381 INFO] MgmtdClient: receive ExitItem [2025-03-11T05:03:27.336293718+00:00 meta_main:81998 MgmtdClient.cc:277 INFO] MgmtdClient: stopped [2025-03-11T05:03:27.336360529+00:00 meta_main:81998 Utils.cc:254 CRITICAL] Load AppInfo from launcher failed: MgmtdClient::PrimaryMgmtdNotFound(6000) ./src/client/mgmtd/MgmtdClient.cc:173, '_result' has error [2025-03-11T05:03:27.336360529+00:00 meta_main:81998 Utils.cc:254 CRITICAL] retryCount: 0 [2025-03-11T05:03:27.346630104+00:00 meta_main:81998 MgmtdClient.cc:222 INFO] MgmtdClient: started ```
bsakaguchi commented 2026-10-26 09:43:38 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 11, 2025):

anyone know the fix for this issue?

[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [[server.base.groups]]
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [server.base.groups.listener]
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] listen_port = 0
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO]
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [server.service]
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] allow_heartbeat_from_unregistered = true
[2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO]
[2025-03-11T10:07:39.165942944+00:00 mgmtd_main:83589 ConfigBase.cc:166 ERROR] Parse config failed: Error while parsing inline table: expected key or closing '}', saw '{'
[2025-03-11T10:07:39.165942944+00:00 mgmtd_main:83589 ConfigBase.cc:166 ERROR]  (error occurred at line 14, column 16)
[2025-03-11T10:07:39.166117828+00:00 mgmtd_main:83589 Utils.cc:290 FATAL] Init config failed: ConfigParseError(71)
*** Aborted at 1741687659 (Unix time, try 'date -d @1741687659') ***
*** Signal 6 (SIGABRT) (0x14685) received by PID 83589 (pthread TID 0x754ced61b600) (linux TID 83589) (maybe from PID 83589, UID 0) (code: -6), stack trace: ***
    @ 0000000000740dcf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 000000000004251f (unknown)
    @ 00000000000969fc pthread_kill
    @ 0000000000042475 raise
    @ 00000000000287f2 abort
    @ 00000000007a8a90 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71
    @ 00000000007a7f46 _ZN5folly18LogStreamProcessor6logNowEv
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190
    @ 00000000007a8199 _ZN5folly16LogStreamVoidifyILb1EEanERSo
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222
    @ 00000000004c6840 _ZN5hf3fs10app_detail10initConfigERNS_6config7IConfigERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESB_ESaISC_EERKNS_4flat7AppInfoESt8functionIFN5folly8ExpectedISC_NS_6StatusEEEvEE
                       /root/wayne/3fs/3FS/src/common/app/Utils.cc:290
    @ 000000000087418d _ZN5hf3fs19TwoPhaseApplicationINS_5mgmtd11MgmtdServerEE15initApplicationEv
                       /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:46
                       -> /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp
    @ 00000000004bcd16 _ZN5hf3fs15ApplicationBase3runEiPPc
                       /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65
    @ 0000000000872a8d main
                       /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp:7
    @ 0000000000029d8f (unknown)
    @ 0000000000029e3f __libc_start_main
    @ 0000000000219e24 _start
<!-- gh-comment-id:2715185979 --> @gaowayne commented on GitHub (Mar 11, 2025): anyone know the fix for this issue? ``` [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [[server.base.groups]] [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [server.base.groups.listener] [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] listen_port = 0 [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [server.service] [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] allow_heartbeat_from_unregistered = true [2025-03-11T10:07:39.164430344+00:00 mgmtd_main:83589 Utils.cc:285 INFO] [2025-03-11T10:07:39.165942944+00:00 mgmtd_main:83589 ConfigBase.cc:166 ERROR] Parse config failed: Error while parsing inline table: expected key or closing '}', saw '{' [2025-03-11T10:07:39.165942944+00:00 mgmtd_main:83589 ConfigBase.cc:166 ERROR] (error occurred at line 14, column 16) [2025-03-11T10:07:39.166117828+00:00 mgmtd_main:83589 Utils.cc:290 FATAL] Init config failed: ConfigParseError(71) *** Aborted at 1741687659 (Unix time, try 'date -d @1741687659') *** *** Signal 6 (SIGABRT) (0x14685) received by PID 83589 (pthread TID 0x754ced61b600) (linux TID 83589) (maybe from PID 83589, UID 0) (code: -6), stack trace: *** @ 0000000000740dcf _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 @ 000000000004251f (unknown) @ 00000000000969fc pthread_kill @ 0000000000042475 raise @ 00000000000287f2 abort @ 00000000007a8a90 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71 @ 00000000007a7f46 _ZN5folly18LogStreamProcessor6logNowEv /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190 @ 00000000007a8199 _ZN5folly16LogStreamVoidifyILb1EEanERSo /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222 @ 00000000004c6840 _ZN5hf3fs10app_detail10initConfigERNS_6config7IConfigERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESB_ESaISC_EERKNS_4flat7AppInfoESt8functionIFN5folly8ExpectedISC_NS_6StatusEEEvEE /root/wayne/3fs/3FS/src/common/app/Utils.cc:290 @ 000000000087418d _ZN5hf3fs19TwoPhaseApplicationINS_5mgmtd11MgmtdServerEE15initApplicationEv /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:46 -> /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp @ 00000000004bcd16 _ZN5hf3fs15ApplicationBase3runEiPPc /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65 @ 0000000000872a8d main /root/wayne/3fs/3FS/src/mgmtd/mgmtd.cpp:7 @ 0000000000029d8f (unknown) @ 0000000000029e3f __libc_start_main @ 0000000000219e24 _start ```
bsakaguchi commented 2026-10-26 09:43:38 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 11, 2025):

This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.

<!-- gh-comment-id:2715243636 --> @yiyuanliu commented on GitHub (Mar 11, 2025): This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.
bsakaguchi commented 2026-10-26 09:43:38 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 12, 2025):

This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.

yeah, I just re-use default source code's config file. if there is any work around that I can continue?

<!-- gh-comment-id:2716578733 --> @gaowayne commented on GitHub (Mar 12, 2025): > This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later. yeah, I just re-use default source code's config file. if there is any work around that I can continue?
bsakaguchi commented 2026-10-26 09:43:38 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 12, 2025):

@yiyuanliu if this confige file is wrong, please suggest? :)

root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/config# cat mgmtd_main_app.toml
allow_empty_node_id = false
node_id = 1
root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/config# pwd
/mnt/nvme2n1/3fsdir/config
<!-- gh-comment-id:2716610688 --> @gaowayne commented on GitHub (Mar 12, 2025): @yiyuanliu if this confige file is wrong, please suggest? :) ``` root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/config# cat mgmtd_main_app.toml allow_empty_node_id = false node_id = 1 root@salab-bncbeta03:/mnt/nvme2n1/3fsdir/config# pwd /mnt/nvme2n1/3fsdir/config ```
bsakaguchi commented 2026-10-26 09:43:39 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 12, 2025):

This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.

buddy, when you fix this issue, could you please let me know the PR, I can help you test it out

<!-- gh-comment-id:2718117086 --> @gaowayne commented on GitHub (Mar 12, 2025): > This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later. buddy, when you fix this issue, could you please let me know the PR, I can help you test it out
bsakaguchi commented 2026-10-26 09:43:39 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 12, 2025):

This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.

buddy, when you fix this issue, could you please let me know the PR, I can help you test it out

#161 I modified the run.sh and configuration files, which should have resolved this issue. You can try it out, and if it works fine, I will merge it into the main branch.

<!-- gh-comment-id:2718313789 --> @yiyuanliu commented on GitHub (Mar 12, 2025): > > This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later. > > buddy, when you fix this issue, could you please let me know the PR, I can help you test it out #161 I modified the run.sh and configuration files, which should have resolved this issue. You can try it out, and if it works fine, I will merge it into the main branch.
bsakaguchi commented 2026-10-26 09:43:39 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 13, 2025):

This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later.

buddy, when you fix this issue, could you please let me know the PR, I can help you test it out

#161 I modified the run.sh and configuration files, which should have resolved this issue. You can try it out, and if it works fine, I will merge it into the main branch.

there is one build issue like below after I git pull latest include your fix as latest PR. could you please take a look at?

-- Installing: /root/wayne/3fs/3FS/build/include/parquet/metadata.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/page_index.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/pch.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/platform.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/printer.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/properties.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/schema.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/statistics.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/stream_reader.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/stream_writer.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/test_util.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/type_fwd.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/types.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/windows_compatibility.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/windows_fixup.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/xxhasher.h
-- Installing: /root/wayne/3fs/3FS/build/include/parquet/parquet_version.h
-- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/LICENSE.txt
-- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/NOTICE.txt
-- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/README.md
-- Installing: /root/wayne/3fs/3FS/build/share/arrow/gdb/gdb_arrow.py
[ 51%] Completed 'apache-arrow-cpp'
[ 51%] Built target apache-arrow-cpp
gmake: *** [Makefile:146: all] Error 2
<!-- gh-comment-id:2721700460 --> @gaowayne commented on GitHub (Mar 13, 2025): > > > This should be a configuration file issue. Some environment variables in the configuration file used by the test scripts need to be rendered using tools like Jinja. This step is missing in the test script. I will try to fix this problem later. > > > > > > buddy, when you fix this issue, could you please let me know the PR, I can help you test it out > > [#161](https://github.com/deepseek-ai/3FS/pull/161) I modified the run.sh and configuration files, which should have resolved this issue. You can try it out, and if it works fine, I will merge it into the main branch. there is one build issue like below after I git pull latest include your fix as latest PR. could you please take a look at? ``` -- Installing: /root/wayne/3fs/3FS/build/include/parquet/metadata.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/page_index.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/pch.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/platform.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/printer.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/properties.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/schema.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/statistics.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/stream_reader.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/stream_writer.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/test_util.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/type_fwd.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/types.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/windows_compatibility.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/windows_fixup.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/xxhasher.h -- Installing: /root/wayne/3fs/3FS/build/include/parquet/parquet_version.h -- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/LICENSE.txt -- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/NOTICE.txt -- Installing: /root/wayne/3fs/3FS/build/share/doc/arrow/README.md -- Installing: /root/wayne/3fs/3FS/build/share/arrow/gdb/gdb_arrow.py [ 51%] Completed 'apache-arrow-cpp' [ 51%] Built target apache-arrow-cpp gmake: *** [Makefile:146: all] Error 2 ```
bsakaguchi commented 2026-10-26 09:43:39 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 14, 2025):

My PR doesn't change any code, it only modifies scripts and configuration files. Could you reduce the number of parallel compilation jobs and try again? The process might have been killed due to using too much memory during compilation.

<!-- gh-comment-id:2723121322 --> @yiyuanliu commented on GitHub (Mar 14, 2025): My PR doesn't change any code, it only modifies scripts and configuration files. Could you reduce the number of parallel compilation jobs and try again? The process might have been killed due to using too much memory during compilation.
bsakaguchi commented 2026-10-26 09:43:39 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 14, 2025):

My PR doesn't change any code, it only modifies scripts and configuration files. Could you reduce the number of parallel compilation jobs and try again? The process might have been killed due to using too much memory during compilation.

others change the code, I update rust build now it works, and I verified mgmtd issue is fixed. :)

### Start mgmtd
### Start meta & storage
<!-- gh-comment-id:2724133125 --> @gaowayne commented on GitHub (Mar 14, 2025): > My PR doesn't change any code, it only modifies scripts and configuration files. Could you reduce the number of parallel compilation jobs and try again? The process might have been killed due to using too much memory during compilation. others change the code, I update rust build now it works, and I verified mgmtd issue is fixed. :) ``` ### Start mgmtd ### Start meta & storage ```
bsakaguchi commented 2026-10-26 09:43:40 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 14, 2025):

@yiyuanliu I met new error now

Init config for MGMTD version 1
Init config for META version 1
Init config for STORAGE version 1
Init config for FUSE version 1
### Start mgmtd
### Start meta & storage
run2.sh: line 178: 143219 Aborted                 (core dumped) "${BINARY}/storage_main" --app_cfg "${CONFIG}/storage_main_app.toml" --launcher_cfg "${CONFIG}/storage_main_launcher.toml" --cfg "${CONFIG}/storage_main.toml" > "${LOG}/storage_main.stdout" 2> "${LOG}/storage_main.stderr"
### List nodes
/root/.profile: line 10: /.cargo/env: No such file or directory
Id  Type   Status               Hostname         Pid     Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1   MGMTD  PRIMARY_MGMTD        salab-bncbeta03  143160  []    N/A                  0              250228-dev-1-999999-3c100b90
50  META   HEARTBEAT_CONNECTED  salab-bncbeta03  143220  []    2025-03-14 02:23:25  0              250228-dev-1-999999-3c100b90
### Create targets and chain table
/root/.profile: line 10: /.cargo/env: No such file or directory
Encounter error: 7004(StorageClient::RoutingError)
./src/client/cli/admin/CreateTarget.cc:69, '_result' has error
Exit with 1
fusermount3: failed to unmount /mnt/nvme2n1/3fsdir/mnt: Invalid argument

here is crashed storage server

[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "handlers": [
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "normal"
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       ]
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   },
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   "handlers": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     "normal": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "options": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "path": "/mnt/nvme2n1/3fsdir/log/storage.log",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "max_files": "100",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "max_file_size": "10485760",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] }
[2025-03-14T02:23:11.458187230+00:00 storage_main:143219 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/mnt/nvme2n1/3fsdir/log/storage.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}}
*** Aborted at 1741918993 (Unix time, try 'date -d @1741918993') ***
*** Signal 6 (SIGABRT) (0x22f73) received by PID 143219 (pthread TID 0x73b6b921abc0) (linux TID 143219) (maybe from PID 143219, UID 0) (code: -6), stack trace: ***
    @ 000000000121e96f _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 000000000004251f (unknown)
    @ 00000000000969fc pthread_kill
    @ 0000000000042475 raise
    @ 00000000000287f2 abort
    @ 00000000012850a0 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71
    @ 0000000001284556 _ZN5folly18LogStreamProcessor6logNowEv
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190
    @ 00000000012847a9 _ZN5folly16LogStreamVoidifyILb1EEanERSo
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222
    @ 0000000002384482 _ZN5hf3fs19TwoPhaseApplicationINS_7storage13StorageServerEE15initApplicationEv
                       /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:100
                       -> /root/wayne/3fs/3FS/src/storage/storage.cpp
    @ 0000000000fb27d6 _ZN5hf3fs15ApplicationBase3runEiPPc
                       /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65
    @ 00000000023811a3 main
                       /root/wayne/3fs/3FS/src/storage/storage.cpp:7
    @ 0000000000029d8f (unknown)
    @ 0000000000029e3f __libc_start_main
    @ 0000000000820264 _start
<!-- gh-comment-id:2724139830 --> @gaowayne commented on GitHub (Mar 14, 2025): @yiyuanliu I met new error now ``` Init config for MGMTD version 1 Init config for META version 1 Init config for STORAGE version 1 Init config for FUSE version 1 ### Start mgmtd ### Start meta & storage run2.sh: line 178: 143219 Aborted (core dumped) "${BINARY}/storage_main" --app_cfg "${CONFIG}/storage_main_app.toml" --launcher_cfg "${CONFIG}/storage_main_launcher.toml" --cfg "${CONFIG}/storage_main.toml" > "${LOG}/storage_main.stdout" 2> "${LOG}/storage_main.stderr" ### List nodes /root/.profile: line 10: /.cargo/env: No such file or directory Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD salab-bncbeta03 143160 [] N/A 0 250228-dev-1-999999-3c100b90 50 META HEARTBEAT_CONNECTED salab-bncbeta03 143220 [] 2025-03-14 02:23:25 0 250228-dev-1-999999-3c100b90 ### Create targets and chain table /root/.profile: line 10: /.cargo/env: No such file or directory Encounter error: 7004(StorageClient::RoutingError) ./src/client/cli/admin/CreateTarget.cc:69, '_result' has error Exit with 1 fusermount3: failed to unmount /mnt/nvme2n1/3fsdir/mnt: Invalid argument ``` here is crashed storage server ``` [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "handlers": [ [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "normal" [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] ] [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] }, [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "handlers": { [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "normal": { [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "type": "file", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "options": { [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "path": "/mnt/nvme2n1/3fsdir/log/storage.log", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "async": "true", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "rotate": "true", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "max_files": "100", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "max_file_size": "10485760", [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "rotate_on_open": "false" [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } [2025-03-14T02:23:11.458187230+00:00 storage_main:143219 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/mnt/nvme2n1/3fsdir/log/storage.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}} *** Aborted at 1741918993 (Unix time, try 'date -d @1741918993') *** *** Signal 6 (SIGABRT) (0x22f73) received by PID 143219 (pthread TID 0x73b6b921abc0) (linux TID 143219) (maybe from PID 143219, UID 0) (code: -6), stack trace: *** @ 000000000121e96f _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 @ 000000000004251f (unknown) @ 00000000000969fc pthread_kill @ 0000000000042475 raise @ 00000000000287f2 abort @ 00000000012850a0 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71 @ 0000000001284556 _ZN5folly18LogStreamProcessor6logNowEv /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190 @ 00000000012847a9 _ZN5folly16LogStreamVoidifyILb1EEanERSo /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222 @ 0000000002384482 _ZN5hf3fs19TwoPhaseApplicationINS_7storage13StorageServerEE15initApplicationEv /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:100 -> /root/wayne/3fs/3FS/src/storage/storage.cpp @ 0000000000fb27d6 _ZN5hf3fs15ApplicationBase3runEiPPc /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65 @ 00000000023811a3 main /root/wayne/3fs/3FS/src/storage/storage.cpp:7 @ 0000000000029d8f (unknown) @ 0000000000029e3f __libc_start_main @ 0000000000820264 _start ```
bsakaguchi commented 2026-10-26 09:43:40 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 14, 2025):

You may see error message in <your_test_dir>/log/storage.log.

I suspect the storage server process failed to start due to insufficient memory or the process unable to lock memory. This PR #180 reduces memory usage.

<!-- gh-comment-id:2725459311 --> @yiyuanliu commented on GitHub (Mar 14, 2025): You may see error message in `<your_test_dir>/log/storage.log`. I suspect the storage server process failed to start due to insufficient memory or the process unable to lock memory. This PR #180 reduces memory usage.
bsakaguchi commented 2026-10-26 09:43:40 +00:00
Author
Owner
Copy link

@gaowayne commented on GitHub (Mar 17, 2025):

You may see error message in <your_test_dir>/log/storage.log.

I suspect the storage server process failed to start due to insufficient memory or the process unable to lock memory. This PR #180 reduces memory usage.

I rebuild, now I did not see storage server crash issue. but I failed to write 1.txt under test folder now.

Image

I did not find any error in the error log. @yiyuanliu

<!-- gh-comment-id:2728165213 --> @gaowayne commented on GitHub (Mar 17, 2025): > You may see error message in `<your_test_dir>/log/storage.log`. > > I suspect the storage server process failed to start due to insufficient memory or the process unable to lock memory. This PR [#180](https://github.com/deepseek-ai/3FS/pull/180) reduces memory usage. I rebuild, now I did not see storage server crash issue. but I failed to write 1.txt under test folder now. ![Image](https://github.com/user-attachments/assets/e9cbb3c1-6224-4bc4-9dfb-0cceed0c1f0d) I did not find any error in the error log. @yiyuanliu
bsakaguchi commented 2026-10-26 09:43:40 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 17, 2025):

Please modify this line to --skip-config-check 1 524288 1 and try again. https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh#L154

<!-- gh-comment-id:2730207061 --> @yiyuanliu commented on GitHub (Mar 17, 2025): Please modify this line to `--skip-config-check 1 524288 1` and try again. https://github.com/deepseek-ai/3FS/blob/main/tests/fuse/run.sh#L154
bsakaguchi commented 2026-10-26 09:43:40 +00:00
Author
Owner
Copy link

@wiryls commented on GitHub (Mar 19, 2025):

@yiyuanliu hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes?

<!-- gh-comment-id:2736485679 --> @wiryls commented on GitHub (Mar 19, 2025): @yiyuanliu hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes?
bsakaguchi commented 2026-10-26 09:43:41 +00:00
Author
Owner
Copy link

@wiryls commented on GitHub (Mar 24, 2025):

@yiyuanliu hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes?

OK, i made it in a ubuntu 20.04 container after some tossing.

Some of the changes made here are as follows:

  1. Installation of dependent libraries, such as libjemalloc-dev, libatomic1, gettext, iproute2 and so on.
  2. Modify scripts:
    • add "bond" into grep -E '(en|eth|ib|bond)' in run.sh
    • resize devices to kMaxDeviceCnt in function IBDevice::openAll of src/common/net/ib/IBDevice.cc.
      • I've tried adding device_filter into tests/fuse/config/meta_main_launcher.toml, but it not works. Even device_filter is empty in log files.
<!-- gh-comment-id:2747597180 --> @wiryls commented on GitHub (Mar 24, 2025): > [@yiyuanliu](https://github.com/yiyuanliu) hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes? OK, i made it in a ubuntu 20.04 container after some tossing. Some of the changes made here are as follows: 1. Installation of dependent libraries, such as libjemalloc-dev, libatomic1, gettext, iproute2 and so on. 2. Modify scripts: - add "bond" into `grep -E '(en|eth|ib|bond)'` in `run.sh` - resize devices to `kMaxDeviceCnt` in function `IBDevice::openAll` of `src/common/net/ib/IBDevice.cc`. - I've tried adding `device_filter` into `tests/fuse/config/meta_main_launcher.toml`, but it not works. Even `device_filter` is empty in log files.
bsakaguchi commented 2026-10-26 09:43:41 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (Mar 25, 2025):

@yiyuanliu hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes?

OK, i made it in a ubuntu 20.04 container after some tossing.

Some of the changes made here are as follows:

  1. Installation of dependent libraries, such as libjemalloc-dev, libatomic1, gettext, iproute2 and so on.

  2. Modify scripts:

    • add "bond" into grep -E '(en|eth|ib|bond)' in run.sh

    • resize devices to kMaxDeviceCnt in function IBDevice::openAll of src/common/net/ib/IBDevice.cc.

      • I've tried adding device_filter into tests/fuse/config/meta_main_launcher.toml, but it not works. Even device_filter is empty in log files.

Could you describe what error you encountered? It seems you've run into an RDMA network-related error, but without knowing the specific error details, I cannot provide concrete suggestions. When using RDMA networks in container environments, using privileged containers can avoid many issues.

<!-- gh-comment-id:2751557682 --> @yiyuanliu commented on GitHub (Mar 25, 2025): > > [@yiyuanliu](https://github.com/yiyuanliu) hi, can I run this nice script in a docker container? I tried it and it doesn't seem to work, do I need to make more changes? > > OK, i made it in a ubuntu 20.04 container after some tossing. > > Some of the changes made here are as follows: > > 1. Installation of dependent libraries, such as libjemalloc-dev, libatomic1, gettext, iproute2 and so on. > 2. Modify scripts: > > * add "bond" into `grep -E '(en|eth|ib|bond)'` in `run.sh` > * resize devices to `kMaxDeviceCnt` in function `IBDevice::openAll` of `src/common/net/ib/IBDevice.cc`. > > * I've tried adding `device_filter` into `tests/fuse/config/meta_main_launcher.toml`, but it not works. Even `device_filter` is empty in log files. Could you describe what error you encountered? It seems you've run into an RDMA network-related error, but without knowing the specific error details, I cannot provide concrete suggestions. When using RDMA networks in container environments, using privileged containers can avoid many issues.
bsakaguchi commented 2026-10-26 09:43:42 +00:00
Author
Owner
Copy link

@wiryls commented on GitHub (Apr 2, 2025):

Could you describe what error you encountered? It seems you've run into an RDMA network-related error, but without knowing the specific error details, I cannot provide concrete suggestions. When using RDMA networks in container environments, using privileged containers can avoid many issues.

The original question was because, when checking the logs for run failures, there was a warning like “maybe run in container“. Makes me think that scripts don't support running in containers.

It's currently working so most of the previous logs are missing. The errors that can be recalled so far are:

ADDRESS is empty thus the rest of the script to report other errors. I fix it by adding "bond" into grep.

export ADDRESS=$(ip -o -4 addr show | grep -E '(en|eth|ib)' | awk '{print $4}' | head -n 1 | awk -F'[/ ]' '{print $1}')

And also our node have with more than 4 IBs. It makes the following code reports an error:

github.com/deepseek-ai/3FS@dfadd59868/src/common/net/ib/IBDevice.cc (L260-L266)

I tried adding device_filter = ['...', '...'] into tests/fuse/config/meta_main_launcher.toml but it didn't work. I can confirm that the configuration file has the field, but it is still empty in the related logs. I ended up modifying the code directly.

The rest are errors caused by missing dependencies, scattered in different log files.

<!-- gh-comment-id:2771608604 --> @wiryls commented on GitHub (Apr 2, 2025): > Could you describe what error you encountered? It seems you've run into an RDMA network-related error, but without knowing the specific error details, I cannot provide concrete suggestions. When using RDMA networks in container environments, using privileged containers can avoid many issues. The original question was because, when checking the logs for run failures, there was a warning like “maybe run in container“. Makes me think that scripts don't support running in containers. It's currently working so most of the previous logs are missing. The errors that can be recalled so far are: `ADDRESS` is empty thus the rest of the script to report other errors. I fix it by adding "bond" into grep. ```bash export ADDRESS=$(ip -o -4 addr show | grep -E '(en|eth|ib)' | awk '{print $4}' | head -n 1 | awk -F'[/ ]' '{print $1}') ``` And also our node have with more than 4 IBs. It makes the following code reports an error: https://github.com/deepseek-ai/3FS/blob/dfadd59868954fe016affc6bfac506864077026c/src/common/net/ib/IBDevice.cc#L260-L266 I tried adding `device_filter = ['...', '...']` into `tests/fuse/config/meta_main_launcher.toml` but it didn't work. I can confirm that the configuration file has the field, but it is still empty in the related logs. I ended up modifying the code directly. The rest are errors caused by missing dependencies, scattered in different log files.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#23
No description provided.