deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #183] monitor_collector_main start failed, throws address not mapped to object error #112

New issue
Open
opened 2026-10-26 09:45:23 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-26 09:45:23 +00:00
Owner
Copy link

Originally created by @iamazy on GitHub (Mar 17, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/183

OS: openEuler 22.10
kernel version: 5.10
arch: aarch64

error message is

[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_bond_0 port 1 ==> bond1 (Up)
[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] mlx5_bond_1 port 1 ==> bond2 (Up)
[2025-03-19T15:05:43.726121180+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_0 => bond1
[2025-03-19T15:05:43.726128590+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_1 => bond2
[2025-03-19T15:05:43.727988730+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond2.2295, addr 10.155.10.171/27, subnet 10.155.10.160/27, up true
[2025-03-19T15:05:43.728012760+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond1.2288, addr 10.153.16.171/27, subnet 10.153.16.160/27, up true
[2025-03-19T15:05:43.728027980+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond0.1318, addr 10.151.18.23/24, subnet 10.151.18.0/24, up true
[2025-03-19T15:05:43.728046960+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of eno1, addr 10.153.16.171/24, subnet 10.153.16.0/24, up true
[2025-03-19T15:05:43.731316560+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_0:1 -> bond1 not found, maybe running in container!
[2025-03-19T15:05:43.731346210+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_0:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.731444940+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_0 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:99:10:ab
[2025-03-19T15:05:43.731456520+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_0, id 0, 1 available ports
[2025-03-19T15:05:43.734180830+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_1:1 -> bond2 not found, maybe running in container!
[2025-03-19T15:05:43.734192510+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_1:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.734269090+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_1 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:9b:a:ab
[2025-03-19T15:05:43.734278690+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_1, id 1, 1 available ports
[2025-03-19T15:05:43.735039780+08:00 IBManager:927324 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] Folly log json configure: {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   "categories": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     ".": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "level": "INFO",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "inherit": true,
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "propagate": "NONE",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "handlers": [
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "normal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "err",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "fatal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "debug"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       ]
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   "handlers": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "normal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "debug": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main-debug.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "err": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "level": "ERR",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "path": "/var/log/3fs/monitor_collector_main-err.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "async": "false",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     "fatal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "type": "stream",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "level": "FATAL",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]         "stream": "stderr"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]       }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]     }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO]   }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735312910+08:00 monitor_collect:927195 OnePhaseApplication.h:87 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal","err","fatal","debug"]}},"handlers":{"normal":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"debug":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main-debug.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"err":{"type":"file","options":{"level":"ERR","path":"/var/log/3fs/monitor_collector_main-err.log","async":"false","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"fatal":{"type":"stream","options":{"level":"FATAL","stream":"stderr"}}}}
*** Aborted at 1742367943 (Unix time, try 'date -d @1742367943') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 927195 (pthread TID 0xffffb3914020) (linux TID 927195) (code: address not mapped to object), stack trace: ***
Segmentation fault (core dumped)

stack trace is

#0  0x0000000000000000 in ?? ()
#1  0x00000000003e0080 in folly::goodMallocSize (minSize=64) at /home/nfsdeplyer/3FS/third_party/folly/folly/memory/Malloc.h:390
#2  folly::fbstring_core<char>::reserveSmall (this=this@entry=0xffffffffa688, minCapacity=<optimized out>, disableSSO=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:836
#3  0x00000000003dffa4 in folly::fbstring_core<char>::expandNoinit (this=this@entry=0xffffffffa688, delta=<optimized out>, 
    expGrowth=<optimized out>, disableSSO=false) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:872
#4  0x00000000003dfe18 in folly::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, folly::fbstring_core<char> >::append (
    this=0xffffffffa688, s=0xffffffff95d0 "_ZN5folly10symbolizer13getStackTraceEPmm", n=40)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:1834
#5  0x0000000000613240 in folly::symbolizer::SymbolizePrinter::print (this=this@entry=0xffffffffa678, frame=...)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:100
#6  0x000000000061372c in folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frame=...)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:151
#7  folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frames=<optimized out>, frameCount=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:179
#8  0x00000000006105e0 in folly::symbolizer::SymbolizePrinter::println<100ul> (this=0xffffffffa678, fa=..., skip=0)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.h:80
#9  folly::symbolizer::(anonymous namespace)::getStackTraceStrImpl<100ul, bool (*)(folly::symbolizer::FrameArray<100ul>&)> (func=<optimized out>)
    at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/Symbolizer.cpp:440
#10 0x0000000000433b18 in hf3fs::net::ThreadPoolGroup::ThreadPoolGroup (this=0x8f2280, name="Svr", config=...)
    at /home/nfsdeplyer/3FS/src/common/net/ThreadPoolGroup.h:33
#11 0x000000000045b354 in hf3fs::net::Server::Server (this=0x8f2240, config=...) at /home/nfsdeplyer/3FS/src/common/net/Server.cc:13
#12 0x00000000003d1fa8 in hf3fs::monitor::MonitorCollectorServer::MonitorCollectorServer (this=0x8f2240, config=...)
    at /home/nfsdeplyer/3FS/src/monitor_collector/service/MonitorCollectorServer.cc:8
#13 0x00000000003cfe7c in std::make_unique<hf3fs::monitor::MonitorCollectorServer, hf3fs::monitor::MonitorCollectorServer::Config&> (__args=...)
    at /usr/lib/gcc/aarch64-linux-gnu/10.3.1/../../../../include/c++/10.3.1/bits/unique_ptr.h:962
#14 hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::initApplication (
    this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>)
    at /home/nfsdeplyer/3FS/src/common/app/OnePhaseApplication.h:99
#15 0x00000000003f1c7c in hf3fs::ApplicationBase::run (
    this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>, argc=1, argv=0xfffffffff5c8)
    at /home/nfsdeplyer/3FS/src/common/app/ApplicationBase.cc:65
#16 0x0000fffff71a3000 in ?? () from /usr/lib64/libc.so.6
#17 0x0000fffff71a30d8 in __libc_start_main () from /usr/lib64/libc.so.6
#18 0x000000000032e9f0 in _start ()

maybe it's a folly exception, but I don't know how to address this issue.

@SF-Zhou please take a look at this issue :)

Originally created by @iamazy on GitHub (Mar 17, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/183 OS: openEuler 22.10 kernel version: 5.10 arch: aarch64 error message is ```console [2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_bond_0 port 1 ==> bond1 (Up) [2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] mlx5_bond_1 port 1 ==> bond2 (Up) [2025-03-19T15:05:43.726121180+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_0 => bond1 [2025-03-19T15:05:43.726128590+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_1 => bond2 [2025-03-19T15:05:43.727988730+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond2.2295, addr 10.155.10.171/27, subnet 10.155.10.160/27, up true [2025-03-19T15:05:43.728012760+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond1.2288, addr 10.153.16.171/27, subnet 10.153.16.160/27, up true [2025-03-19T15:05:43.728027980+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond0.1318, addr 10.151.18.23/24, subnet 10.151.18.0/24, up true [2025-03-19T15:05:43.728046960+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of eno1, addr 10.153.16.171/24, subnet 10.153.16.0/24, up true [2025-03-19T15:05:43.731316560+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_0:1 -> bond1 not found, maybe running in container! [2025-03-19T15:05:43.731346210+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_0:1 can't set zone by IP, fallback to UNKNOWN [2025-03-19T15:05:43.731444940+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_0 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:99:10:ab [2025-03-19T15:05:43.731456520+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_0, id 0, 1 available ports [2025-03-19T15:05:43.734180830+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_1:1 -> bond2 not found, maybe running in container! [2025-03-19T15:05:43.734192510+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_1:1 can't set zone by IP, fallback to UNKNOWN [2025-03-19T15:05:43.734269090+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_1 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:9b:a:ab [2025-03-19T15:05:43.734278690+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_1, id 1, 1 available ports [2025-03-19T15:05:43.735039780+08:00 IBManager:927324 EventLoop.cc:116 INFO] EventLoop::loop() started. [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] Folly log json configure: { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "categories": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] ".": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "INFO", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "inherit": true, [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "propagate": "NONE", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "handlers": [ [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "normal", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "err", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "fatal", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "debug" [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] ] [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }, [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "handlers": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "normal": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main.log", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "true", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false" [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }, [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "debug": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main-debug.log", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "true", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false" [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }, [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "err": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "ERR", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main-err.log", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "false", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false" [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }, [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "fatal": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "stream", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": { [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "FATAL", [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "stream": "stderr" [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] } [2025-03-19T15:05:43.735312910+08:00 monitor_collect:927195 OnePhaseApplication.h:87 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal","err","fatal","debug"]}},"handlers":{"normal":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"debug":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main-debug.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"err":{"type":"file","options":{"level":"ERR","path":"/var/log/3fs/monitor_collector_main-err.log","async":"false","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"fatal":{"type":"stream","options":{"level":"FATAL","stream":"stderr"}}}} *** Aborted at 1742367943 (Unix time, try 'date -d @1742367943') *** *** Signal 11 (SIGSEGV) (0x0) received by PID 927195 (pthread TID 0xffffb3914020) (linux TID 927195) (code: address not mapped to object), stack trace: *** Segmentation fault (core dumped) ``` stack trace is ```console #0 0x0000000000000000 in ?? () #1 0x00000000003e0080 in folly::goodMallocSize (minSize=64) at /home/nfsdeplyer/3FS/third_party/folly/folly/memory/Malloc.h:390 #2 folly::fbstring_core<char>::reserveSmall (this=this@entry=0xffffffffa688, minCapacity=<optimized out>, disableSSO=<optimized out>) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:836 #3 0x00000000003dffa4 in folly::fbstring_core<char>::expandNoinit (this=this@entry=0xffffffffa688, delta=<optimized out>, expGrowth=<optimized out>, disableSSO=false) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:872 #4 0x00000000003dfe18 in folly::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, folly::fbstring_core<char> >::append ( this=0xffffffffa688, s=0xffffffff95d0 "_ZN5folly10symbolizer13getStackTraceEPmm", n=40) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:1834 #5 0x0000000000613240 in folly::symbolizer::SymbolizePrinter::print (this=this@entry=0xffffffffa678, frame=...) at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:100 #6 0x000000000061372c in folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frame=...) at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:151 #7 folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frames=<optimized out>, frameCount=<optimized out>) at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:179 #8 0x00000000006105e0 in folly::symbolizer::SymbolizePrinter::println<100ul> (this=0xffffffffa678, fa=..., skip=0) at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.h:80 #9 folly::symbolizer::(anonymous namespace)::getStackTraceStrImpl<100ul, bool (*)(folly::symbolizer::FrameArray<100ul>&)> (func=<optimized out>) at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/Symbolizer.cpp:440 #10 0x0000000000433b18 in hf3fs::net::ThreadPoolGroup::ThreadPoolGroup (this=0x8f2280, name="Svr", config=...) at /home/nfsdeplyer/3FS/src/common/net/ThreadPoolGroup.h:33 #11 0x000000000045b354 in hf3fs::net::Server::Server (this=0x8f2240, config=...) at /home/nfsdeplyer/3FS/src/common/net/Server.cc:13 #12 0x00000000003d1fa8 in hf3fs::monitor::MonitorCollectorServer::MonitorCollectorServer (this=0x8f2240, config=...) at /home/nfsdeplyer/3FS/src/monitor_collector/service/MonitorCollectorServer.cc:8 #13 0x00000000003cfe7c in std::make_unique<hf3fs::monitor::MonitorCollectorServer, hf3fs::monitor::MonitorCollectorServer::Config&> (__args=...) at /usr/lib/gcc/aarch64-linux-gnu/10.3.1/../../../../include/c++/10.3.1/bits/unique_ptr.h:962 #14 hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::initApplication ( this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>) at /home/nfsdeplyer/3FS/src/common/app/OnePhaseApplication.h:99 #15 0x00000000003f1c7c in hf3fs::ApplicationBase::run ( this=0x7aaed0 <hf3fs::OnePhaseApplication<hf3fs::monitor::MonitorCollectorServer>::instance()::app>, argc=1, argv=0xfffffffff5c8) at /home/nfsdeplyer/3FS/src/common/app/ApplicationBase.cc:65 #16 0x0000fffff71a3000 in ?? () from /usr/lib64/libc.so.6 #17 0x0000fffff71a30d8 in __libc_start_main () from /usr/lib64/libc.so.6 #18 0x000000000032e9f0 in _start () ``` maybe it's a folly exception, but I don't know how to address this issue. @SF-Zhou please take a look at this issue :)
bsakaguchi commented 2026-10-26 09:45:24 +00:00
Author
Owner
Copy link

@iamazy commented on GitHub (Mar 19, 2025):

I disable libunwind in folly and finally start monitor_collector_main success

cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON
<!-- gh-comment-id:2736458802 --> @iamazy commented on GitHub (Mar 19, 2025): I disable libunwind in folly and finally start monitor_collector_main success ```console cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON ```
bsakaguchi commented 2026-10-26 09:45:25 +00:00
Author
Owner
Copy link

@RangerCD commented on GitHub (May 15, 2025):

Hi, I've tried your solution by adding -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON to cmake, but still got that address not mapped to object error. Did you also modified other stuff that could possibly fix this issue?

<!-- gh-comment-id:2882978981 --> @RangerCD commented on GitHub (May 15, 2025): Hi, I've tried your solution by adding `-DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON` to cmake, but still got that `address not mapped to object` error. Did you also modified other stuff that could possibly fix this issue?
bsakaguchi commented 2026-10-26 09:45:25 +00:00
Author
Owner
Copy link

@ThierryZhou commented on GitHub (May 23, 2025):

2274 I tried to link with folly and jemalloc shared.

<!-- gh-comment-id:2903116108 --> @ThierryZhou commented on GitHub (May 23, 2025): [2274](https://github.com/facebook/folly/issues/2274) I tried to link with folly and jemalloc shared.
bsakaguchi commented 2026-10-26 09:45:25 +00:00
Author
Owner
Copy link

@iamazy commented on GitHub (May 23, 2025):

Hi, I've tried your solution by adding -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON to cmake, but still got that address not mapped to object error. Did you also modified other stuff that could possibly fix this issue?

I only made changes here

<!-- gh-comment-id:2903122922 --> @iamazy commented on GitHub (May 23, 2025): > Hi, I've tried your solution by adding `-DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON` to cmake, but still got that `address not mapped to object` error. Did you also modified other stuff that could possibly fix this issue? I only made changes here
bsakaguchi commented 2026-10-26 09:45:26 +00:00
Author
Owner
Copy link

@iamazy commented on GitHub (May 23, 2025):

2274 I tried to link with folly and jemalloc shared.

I am not familiar with C++/Cmake, Could you please post your compile command?

<!-- gh-comment-id:2903151840 --> @iamazy commented on GitHub (May 23, 2025): > [2274](https://github.com/facebook/folly/issues/2274) I tried to link with folly and jemalloc shared. I am not familiar with C++/Cmake, Could you please post your compile command?
bsakaguchi commented 2026-10-26 09:45:27 +00:00
Author
Owner
Copy link

@Goloker commented on GitHub (Aug 5, 2025):

I encountered a similar issue and resolved it by first executing the command export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2, and then running my command (for example: /home/3fs/build/bin/hf3fs_fuse_main --launcher_cfg /home/3fs/configs/hf3fs_fuse_main_launcher.toml).

<!-- gh-comment-id:3154512224 --> @Goloker commented on GitHub (Aug 5, 2025): I encountered a similar issue and resolved it by first executing the command `export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2`, and then running my command (for example: `/home/3fs/build/bin/hf3fs_fuse_main --launcher_cfg /home/3fs/configs/hf3fs_fuse_main_launcher.toml`).
bsakaguchi commented 2026-10-26 09:45:27 +00:00
Author
Owner
Copy link

@zhaohuxing commented on GitHub (Aug 19, 2025):

I encountered a similar issue and resolved it by first executing the command export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2, and then running my command (for example: /home/3fs/build/bin/hf3fs_fuse_main --launcher_cfg /home/3fs/configs/hf3fs_fuse_main_launcher.toml).

@Goloker I try exec this command, but failed, output:

### Init cluster
src/tcmalloc.cc:333] Attempt to free invalid pointer 0xffffb17d0000
./tests/fuse/run.sh: line 146: 55775 Aborted                 (core dumped) ${ADMIN_CLI} user-add --root --admin --token "${TOKEN}" 0 root
<!-- gh-comment-id:3199277988 --> @zhaohuxing commented on GitHub (Aug 19, 2025): > I encountered a similar issue and resolved it by first executing the command `export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2`, and then running my command (for example: `/home/3fs/build/bin/hf3fs_fuse_main --launcher_cfg /home/3fs/configs/hf3fs_fuse_main_launcher.toml`). @Goloker I try exec this command, but failed, output: ``` ### Init cluster src/tcmalloc.cc:333] Attempt to free invalid pointer 0xffffb17d0000 ./tests/fuse/run.sh: line 146: 55775 Aborted (core dumped) ${ADMIN_CLI} user-add --root --admin --token "${TOKEN}" 0 root ```
bsakaguchi commented 2026-10-26 09:45:27 +00:00
Author
Owner
Copy link

@zhaohuxing commented on GitHub (Aug 19, 2025):

Hi, I've tried your solution by adding -DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON to cmake, but still got that address not mapped to object error. Did you also modified other stuff that could possibly fix this issue?

@RangerCD some error

2025-08-19T08:37:52.953860850+00:00 mgmtd_main:94957 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/tmp/3fs-test/log/mgmtd.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}}
*** Aborted at 1755592673 (Unix time, try 'date -d @1755592673') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 94957 (pthread TID 0xffff9c743020) (linux TID 94957) (code: address not mapped to object), stack trace: ***
(error retrieving stack trace)
<!-- gh-comment-id:3199802742 --> @zhaohuxing commented on GitHub (Aug 19, 2025): > Hi, I've tried your solution by adding `-DFOLLY_USE_JEMALLOC=OFF -DFOLLY_DISABLE_LIBUNWIND=ON` to cmake, but still got that `address not mapped to object` error. Did you also modified other stuff that could possibly fix this issue? @RangerCD some error ``` 2025-08-19T08:37:52.953860850+00:00 mgmtd_main:94957 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/tmp/3fs-test/log/mgmtd.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}} *** Aborted at 1755592673 (Unix time, try 'date -d @1755592673') *** *** Signal 11 (SIGSEGV) (0x0) received by PID 94957 (pthread TID 0xffff9c743020) (linux TID 94957) (code: address not mapped to object), stack trace: *** (error retrieving stack trace) ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#112
No description provided.