deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #192] mgmtd_main and admin_cli core when deploy with E810 RDMA card #120

New issue
Open
opened 2026-10-26 09:45:30 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:45:30 +00:00
Owner
Copy link

Originally created by @junzeng1 on GitHub (Mar 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/192

  1. we try to deploy a 3FS cluster with Intel E810 RDMA card;
  2. the admin_cli exit abnormally when running "/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes"
    " command after we have start mgmtd_main service successfully;
  3. after a while the mgmtd_main also exit abnormally.

It seems that the two abnormal exit due to same reason, just happen when trying to close a IBSocket.

A. ###########the core stack of admin_cli
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD spr-9 374052 [] N/A 1(UPTODATE) 250228-dev-1-999999-3a30c53b
100 META HEARTBEAT_CONNECTING spr-9 0 [] N/A 0 0-dev-0-0-00000000
*** Aborted at 1742303109 (Unix time, try 'date -d @1742303109') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 374105 (pthread TID 0x7f7c6702ed40) (linux TID 374105) (code: address not mapped to object), stack trace: ***
@ 000000000245250f folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
/home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
@ 000000000004251f (unknown)
@ 0000000000006cf0 (unknown)
@ 00000000022a373f hf3fs::net::IBSocket::closeGracefully()
/usr/include/infiniband/verbs.h:3327
-> /home/3fs/3fs/src/common/net/ib/IBSocket.cc
@ 00000000022a50c2 hf3fs::net::IBSocket::Drainer::create(std::unique_ptr<hf3fs::net::IBSocket, std::default_deletehf3fs::net::IBSocket >, std::weak_ptrhf3fs::net::IBSocketManager)
/home/3fs/3fs/src/common/net/ib/IBSocket.cc:1094
@ 00000000022a6968 hf3fs::net::IBSocketManager::close(std::unique_ptr<hf3fs::net::IBSocket, std::default_deletehf3fs::net::IBSocket >)
/home/3fs/3fs/src/common/net/ib/IBSocket.cc:1203
@ 00000000022915f3 hf3fs::net::IBManager::close(std::unique_ptr<hf3fs::net::IBSocket, std::default_deletehf3fs::net::IBSocket >)
/home/3fs/3fs/src/common/net/ib/IBDevice.cc:757
@ 0000000002255746 hf3fs::net::Transport::~Transport()
/home/3fs/3fs/src/common/net/Transport.cc:64
@ 000000000367d539 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:175
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 00000000037460a6 std::_Rb_tree<std::shared_ptrhf3fs::net::Transport, std::pair<std::shared_ptrhf3fs::net::Transport const, unsigned int>, std::_Select1st<std::pair<std::shared_ptrhf3fs::net::Transport const, unsigned int> >, std::less<std::shared_ptrhf3fs::net::Transport >, std::allocator<std::pair<std::shared_ptrhf3fs::net::Transport const, unsigned int> > >::_M_erase(std::_Rb_tree_node<std::pair<std::shared_ptrhf3fs::net::Transport const, unsigned int> >*)
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:353
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 00000000022605bd hf3fs::net::TransportPool::dropConnections(bool, bool)
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_tree.h:986
-> /home/3fs/3fs/src/common/net/TransportPool.cc
@ 00000000022315f6 hf3fs::net::IOWorker::stopAndJoin()
/home/3fs/3fs/src/common/net/IOWorker.h:76
-> /home/3fs/3fs/src/common/net/IOWorker.cc
@ 00000000036649c3 main
/home/3fs/3fs/src/common/net/Client.h:51
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 0000000000029d8f (unknown)
@ 0000000000029e3f __libc_start_main
@ 0000000001a03ca4 _start

B. ########## the core stack of mgmtd_main
mgmtd_main[374052]: *** Aborted at 1742303146 (Unix time, try 'date -d @1742303146') ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Signal 11 (SIGSEGV) (0x0) received by PID 374052 (pthread TID 0x7efeb4ff9640) (linux TID 374084) (code: address not mapped to object), stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007445ef folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000004251f (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000006cf0 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005a0588 hf3fs::net::IBSocket::check()
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/include/infiniband/verbs.h:3327
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ib/IBSocket.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005589c9 hf3fs::net::TransportSet::checkAll(hf3fs::Duration)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/Transport.h:64
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/TransportPool.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000559e51 hf3fs::net::TransportPool::checkConnections(hf3fs::net::Address, hf3fs::Duration)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/TransportPool.cc:198
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000536454 hf3fs::net::ServiceGroup::checkConnectionsRegularly() [clone .resume]
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/IOWorker.h:82
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ServiceGroup.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007d0183 folly::resumeCoroutineWithNewAsyncStackRoot(std::__n4861::coroutine_handle, folly::AsyncStackFrame&)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/coroutine:135
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/tracing/AsyncStack.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000227498 folly::coro::detail::ViaCoroutinePromiseBase::executeContinuation()
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:90
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/mgmtd/MgmtdConfigFetcher.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000070f52a folly::ThreadPoolExecutor::runTask(std::shared_ptrfolly::ThreadPoolExecutor::Thread const&, folly::ThreadPoolExecutor::Task&&)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/Function.h:375
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000006fa5aa folly::CPUThreadPoolExecutor::threadRun(std::shared_ptrfolly::ThreadPoolExecutor::Thread)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/executors/CPUThreadPoolExecutor.cpp:333
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000711edf void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptrfolly::ThreadPoolExecutor::Thread), folly::ThreadPo>
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:74
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000000dc252 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000094ac2 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000012684f (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Check failure async stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack root: 0x7efeb4bdd988, normal stack frame pointer holding async stack root: 0x7efeb4bdd9c0, return address: 0x561e113b8499 ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack frame pointer: 0x561e5195c018, return address: 0x561e116929e7, async stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005019e7 folly::coro::detail::InlineTaskDetached folly::coro::TaskWithExecutor::startImpl<folly::coro::TaskWithExecutor::start() &&:>
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:0

C. ######### the ibv_devinfo of E810 RDMA card
hca_id: irdma0
transport: InfiniBand (0)
fw_ver: 1.74
node_guid: 6efe:54ff:fe40:c0d0
sys_image_guid: 6efe:54ff:fe40:c0d0
vendor_id: 0x8086
vendor_part_id: 5522
hw_ver: 0x2
phys_port_cnt: 1
max_mr_size: 0x200000000000
page_size_cap: 0x40201000
max_qp: 4092
max_qp_wr: 8127
device_cap_flags: 0x01221000
RC_RNR_NAK_GEN
MEM_WINDOW
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
max_sge: 6
max_sge_rd: 6
max_cq: 8189
max_cqe: 1048574
max_mr: 4194301
max_pd: 262141
max_qp_rd_atom: 8
max_ee_rd_atom: 0
max_res_rd_atom: 0
max_qp_init_rd_atom: 8
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_NONE (0)
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 4096
max_mcast_qp_attach: 8
max_total_mcast_qp_attach: 32768
max_ah: 131072
max_fmr: 0
max_srq: 0
max_pkeys: 1
local_ca_ack_delay: 0
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
completion_timestamp_mask not supported
core clock not supported
device_cap_flags_ex: 0x1221000
tso_caps:
max_tso: 0
rss_caps:
max_rwq_indirection_tables: 0
max_rwq_indirection_table_size: 0
rx_hash_function: 0x0
rx_hash_fields_mask: 0x0
max_wq_type_rq: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
tag matching not supported
num_comp_vectors: 65
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x7fffffff
port_cap_flags: 0x04050000
port_cap_flags2: 0x0000
max_vl_num: invalid value (0)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 1
gid_tbl_len: 32
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
phys_state: LINK_UP (5)
GID[ 0]: fe80::6efe:54ff:fe40:c0d0, RoCE v2
GID[ 1]: ::ffff:192.100.100.112, RoCE v2

D. ####### and we have change the max_sge = 1 and max_rd_atomic = 8 according to the capacity of irdma0 device

E. we can run ib_write_bw with the RDMA device successfully

Actually, there are _opt_3fs_bin_admin_cli.0.crash and _opt_3fs_bin_mgmtd_main.0.crash under /var/crash directory, but we are not able to analyze them, as they not a type of file known by gdb.

Originally created by @junzeng1 on GitHub (Mar 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/192 1. we try to deploy a 3FS cluster with Intel E810 RDMA card; 2. the admin_cli exit abnormally when running "/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes" " command after we have start mgmtd_main service successfully; 3. after a while the mgmtd_main also exit abnormally. It seems that the two abnormal exit due to same reason, just happen when trying to close a IBSocket. A. ###########the core stack of admin_cli /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes" Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD spr-9 374052 [] N/A 1(UPTODATE) 250228-dev-1-999999-3a30c53b 100 META HEARTBEAT_CONNECTING spr-9 0 [] N/A 0 0-dev-0-0-00000000 *** Aborted at 1742303109 (Unix time, try 'date -d @1742303109') *** *** Signal 11 (SIGSEGV) (0x0) received by PID 374105 (pthread TID 0x7f7c6702ed40) (linux TID 374105) (code: address not mapped to object), stack trace: *** @ 000000000245250f folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*) /home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 @ 000000000004251f (unknown) @ 0000000000006cf0 (unknown) @ 00000000022a373f hf3fs::net::IBSocket::closeGracefully() /usr/include/infiniband/verbs.h:3327 -> /home/3fs/3fs/src/common/net/ib/IBSocket.cc @ 00000000022a50c2 hf3fs::net::IBSocket::Drainer::create(std::unique_ptr<hf3fs::net::IBSocket, std::default_delete<hf3fs::net::IBSocket> >, std::weak_ptr<hf3fs::net::IBSocketManager>) /home/3fs/3fs/src/common/net/ib/IBSocket.cc:1094 @ 00000000022a6968 hf3fs::net::IBSocketManager::close(std::unique_ptr<hf3fs::net::IBSocket, std::default_delete<hf3fs::net::IBSocket> >) /home/3fs/3fs/src/common/net/ib/IBSocket.cc:1203 @ 00000000022915f3 hf3fs::net::IBManager::close(std::unique_ptr<hf3fs::net::IBSocket, std::default_delete<hf3fs::net::IBSocket> >) /home/3fs/3fs/src/common/net/ib/IBDevice.cc:757 @ 0000000002255746 hf3fs::net::Transport::~Transport() /home/3fs/3fs/src/common/net/Transport.cc:64 @ 000000000367d539 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:175 -> /home/3fs/3fs/src/client/bin/admin_cli.cc @ 00000000037460a6 std::_Rb_tree<std::shared_ptr<hf3fs::net::Transport>, std::pair<std::shared_ptr<hf3fs::net::Transport> const, unsigned int>, std::_Select1st<std::pair<std::shared_ptr<hf3fs::net::Transport> const, unsigned int> >, std::less<std::shared_ptr<hf3fs::net::Transport> >, std::allocator<std::pair<std::shared_ptr<hf3fs::net::Transport> const, unsigned int> > >::_M_erase(std::_Rb_tree_node<std::pair<std::shared_ptr<hf3fs::net::Transport> const, unsigned int> >*) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:353 -> /home/3fs/3fs/src/client/bin/admin_cli.cc @ 00000000022605bd hf3fs::net::TransportPool::dropConnections(bool, bool) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_tree.h:986 -> /home/3fs/3fs/src/common/net/TransportPool.cc @ 00000000022315f6 hf3fs::net::IOWorker::stopAndJoin() /home/3fs/3fs/src/common/net/IOWorker.h:76 -> /home/3fs/3fs/src/common/net/IOWorker.cc @ 00000000036649c3 main /home/3fs/3fs/src/common/net/Client.h:51 -> /home/3fs/3fs/src/client/bin/admin_cli.cc @ 0000000000029d8f (unknown) @ 0000000000029e3f __libc_start_main @ 0000000001a03ca4 _start B. ########## the core stack of mgmtd_main mgmtd_main[374052]: *** Aborted at 1742303146 (Unix time, try 'date -d @1742303146') *** Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Signal 11 (SIGSEGV) (0x0) received by PID 374052 (pthread TID 0x7efeb4ff9640) (linux TID 374084) (code: address not mapped to object), stack trace: *** Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007445ef folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000004251f (unknown) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000006cf0 (unknown) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005a0588 hf3fs::net::IBSocket::check() Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/include/infiniband/verbs.h:3327 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ib/IBSocket.cc Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005589c9 hf3fs::net::TransportSet::checkAll(hf3fs::Duration) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/Transport.h:64 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/TransportPool.cc Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000559e51 hf3fs::net::TransportPool::checkConnections(hf3fs::net::Address, hf3fs::Duration) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/TransportPool.cc:198 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000536454 hf3fs::net::ServiceGroup::checkConnectionsRegularly() [clone .resume] Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/IOWorker.h:82 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ServiceGroup.cc Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007d0183 folly::resumeCoroutineWithNewAsyncStackRoot(std::__n4861::coroutine_handle<void>, folly::AsyncStackFrame&) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/coroutine:135 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/tracing/AsyncStack.cpp Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000227498 folly::coro::detail::ViaCoroutinePromiseBase::executeContinuation() Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:90 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/mgmtd/MgmtdConfigFetcher.cc Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000070f52a folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/Function.h:375 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000006fa5aa folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/executors/CPUThreadPoolExecutor.cpp:333 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000711edf void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPo> Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:74 Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000000dc252 (unknown) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000094ac2 (unknown) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000012684f (unknown) Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Check failure async stack trace: *** Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack root: 0x7efeb4bdd988, normal stack frame pointer holding async stack root: 0x7efeb4bdd9c0, return address: 0x561e113b8499 *** Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack frame pointer: 0x561e5195c018, return address: 0x561e116929e7, async stack trace: *** Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005019e7 folly::coro::detail::InlineTaskDetached folly::coro::TaskWithExecutor<void>::startImpl<folly::coro::TaskWithExecutor<void>::start() &&:> Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:0 C. ######### the ibv_devinfo of E810 RDMA card hca_id: irdma0 transport: InfiniBand (0) fw_ver: 1.74 node_guid: 6efe:54ff:fe40:c0d0 sys_image_guid: 6efe:54ff:fe40:c0d0 vendor_id: 0x8086 vendor_part_id: 5522 hw_ver: 0x2 phys_port_cnt: 1 max_mr_size: 0x200000000000 page_size_cap: 0x40201000 max_qp: 4092 max_qp_wr: 8127 device_cap_flags: 0x01221000 RC_RNR_NAK_GEN MEM_WINDOW MEM_MGT_EXTENSIONS MEM_WINDOW_TYPE_2B max_sge: 6 max_sge_rd: 6 max_cq: 8189 max_cqe: 1048574 max_mr: 4194301 max_pd: 262141 max_qp_rd_atom: 8 max_ee_rd_atom: 0 max_res_rd_atom: 0 max_qp_init_rd_atom: 8 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_NONE (0) max_ee: 0 max_rdd: 0 max_mw: 0 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 0 max_mcast_grp: 4096 max_mcast_qp_attach: 8 max_total_mcast_qp_attach: 32768 max_ah: 131072 max_fmr: 0 max_srq: 0 max_pkeys: 1 local_ca_ack_delay: 0 general_odp_caps: rc_odp_caps: NO SUPPORT uc_odp_caps: NO SUPPORT ud_odp_caps: NO SUPPORT xrc_odp_caps: NO SUPPORT completion_timestamp_mask not supported core clock not supported device_cap_flags_ex: 0x1221000 tso_caps: max_tso: 0 rss_caps: max_rwq_indirection_tables: 0 max_rwq_indirection_table_size: 0 rx_hash_function: 0x0 rx_hash_fields_mask: 0x0 max_wq_type_rq: 0 packet_pacing_caps: qp_rate_limit_min: 0kbps qp_rate_limit_max: 0kbps tag matching not supported num_comp_vectors: 65 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 1 port_lmc: 0x00 link_layer: Ethernet max_msg_sz: 0x7fffffff port_cap_flags: 0x04050000 port_cap_flags2: 0x0000 max_vl_num: invalid value (0) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 1 gid_tbl_len: 32 subnet_timeout: 0 init_type_reply: 0 active_width: 4X (2) active_speed: 25.0 Gbps (32) phys_state: LINK_UP (5) GID[ 0]: fe80::6efe:54ff:fe40:c0d0, RoCE v2 GID[ 1]: ::ffff:192.100.100.112, RoCE v2 D. ####### and we have change the max_sge = 1 and max_rd_atomic = 8 according to the capacity of irdma0 device E. we can run ib_write_bw with the RDMA device successfully Actually, there are _opt_3fs_bin_admin_cli.0.crash and _opt_3fs_bin_mgmtd_main.0.crash under /var/crash directory, but we are not able to analyze them, as they not a type of file known by gdb.
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@junzeng1 commented on GitHub (Mar 19, 2025):

through I have update the parameter in config file according to the capability of E810 rdma card (like max_sge=1 and max_rd_atomic =8), but from the log of mgmtd_main, the parameters for these two are still the default value. Can any one do me a favor about how to fix this ? or whether it's the cause for the crash problem above ?

parameters printed out from the mgmted_main.log

[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] num_event_loop = 1
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] rdma_connect_timeout = '5s'
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] read_write_rdma_in_event_thread = false
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] read_write_tcp_in_event_thread = false
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] tcp_connect_timeout = '1s'
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] wait_to_retry_send = '100ms'
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker.connect_concurrency_limiter]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_concurrency = 4
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker.ibsocket]
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_ack_batch = 8
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_signal_batch = 8
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_size = 16384
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] drain_timeout = '5s'
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] drop_connections = 0
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] event_ack_batch = 128
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rd_atomic = 16
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rdma_wr = 128
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rdma_wr_per_post = 32
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_sge = 16
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] min_rnr_timer = 1
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] record_bytes_per_peer = false
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] record_latency_per_peer = false
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] retry_cnt = 7
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] rnr_retry = 0
[2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] send_buf_cnt = 32

<!-- gh-comment-id:2735212400 --> @junzeng1 commented on GitHub (Mar 19, 2025): through I have update the parameter in config file according to the capability of E810 rdma card (like max_sge=1 and max_rd_atomic =8), but from the log of mgmtd_main, the parameters for these two are still the default value. Can any one do me a favor about how to fix this ? or whether it's the cause for the crash problem above ? ##### parameters printed out from the mgmted_main.log [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] num_event_loop = 1 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] rdma_connect_timeout = '5s' [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] read_write_rdma_in_event_thread = false [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] read_write_tcp_in_event_thread = false [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] tcp_connect_timeout = '1s' [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] wait_to_retry_send = '100ms' [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker.connect_concurrency_limiter] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_concurrency = 4 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] [server.base.groups.io_worker.ibsocket] [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_ack_batch = 8 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_signal_batch = 8 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] buf_size = 16384 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] drain_timeout = '5s' [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] drop_connections = 0 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] event_ack_batch = 128 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rd_atomic = 16 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rdma_wr = 128 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_rdma_wr_per_post = 32 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] max_sge = 16 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] min_rnr_timer = 1 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] record_bytes_per_peer = false [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] record_latency_per_peer = false [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] retry_cnt = 7 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] rnr_retry = 0 [2025-03-19T10:49:36.356504950+08:00 SvrBG1:378808 ConfigManager.cc:87 INFO] send_buf_cnt = 32
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#120
No description provided.