deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #285] 运行单node部署测试时使用run.bash脚本storage target create步骤运行失败（TOO many open files） #172

New issue
Closed
opened 2026-10-26 09:46:34 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:46:34 +00:00
Owner
Copy link

Originally created by @SgtPepperr on GitHub (May 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/285

@yiyuanliu I met new error now

Init config for MGMTD version 1
Init config for META version 1
Init config for STORAGE version 1
Init config for FUSE version 1
### Start mgmtd
### Start meta & storage
run2.sh: line 178: 143219 Aborted                 (core dumped) "${BINARY}/storage_main" --app_cfg "${CONFIG}/storage_main_app.toml" --launcher_cfg "${CONFIG}/storage_main_launcher.toml" --cfg "${CONFIG}/storage_main.toml" > "${LOG}/storage_main.stdout" 2> "${LOG}/storage_main.stderr"
### List nodes
/root/.profile: line 10: /.cargo/env: No such file or directory
Id  Type   Status               Hostname         Pid     Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1   MGMTD  PRIMARY_MGMTD        salab-bncbeta03  143160  []    N/A                  0              250228-dev-1-999999-3c100b90
50  META   HEARTBEAT_CONNECTED  salab-bncbeta03  143220  []    2025-03-14 02:23:25  0              250228-dev-1-999999-3c100b90
### Create targets and chain table
/root/.profile: line 10: /.cargo/env: No such file or directory
Encounter error: 7004(StorageClient::RoutingError)
./src/client/cli/admin/CreateTarget.cc:69, '_result' has error
Exit with 1
fusermount3: failed to unmount /mnt/nvme2n1/3fsdir/mnt: Invalid argument

here is crashed storage server

[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "handlers": [
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "normal"
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       ]
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   },
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   "handlers": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     "normal": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "type": "file",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       "options": {
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "path": "/mnt/nvme2n1/3fsdir/log/storage.log",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "async": "true",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "rotate": "true",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "max_files": "100",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "max_file_size": "10485760",
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]         "rotate_on_open": "false"
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]       }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]     }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO]   }
[2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] }
[2025-03-14T02:23:11.458187230+00:00 storage_main:143219 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/mnt/nvme2n1/3fsdir/log/storage.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}}
*** Aborted at 1741918993 (Unix time, try 'date -d @1741918993') ***
*** Signal 6 (SIGABRT) (0x22f73) received by PID 143219 (pthread TID 0x73b6b921abc0) (linux TID 143219) (maybe from PID 143219, UID 0) (code: -6), stack trace: ***
    @ 000000000121e96f _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 000000000004251f (unknown)
    @ 00000000000969fc pthread_kill
    @ 0000000000042475 raise
    @ 00000000000287f2 abort
    @ 00000000012850a0 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71
    @ 0000000001284556 _ZN5folly18LogStreamProcessor6logNowEv
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190
    @ 00000000012847a9 _ZN5folly16LogStreamVoidifyILb1EEanERSo
                       /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222
    @ 0000000002384482 _ZN5hf3fs19TwoPhaseApplicationINS_7storage13StorageServerEE15initApplicationEv
                       /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:100
                       -> /root/wayne/3fs/3FS/src/storage/storage.cpp
    @ 0000000000fb27d6 _ZN5hf3fs15ApplicationBase3runEiPPc
                       /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65
    @ 00000000023811a3 main
                       /root/wayne/3fs/3FS/src/storage/storage.cpp:7
    @ 0000000000029d8f (unknown)
    @ 0000000000029e3f __libc_start_main
    @ 0000000000820264 _start

Originally posted by @gaowayne in #39

我遇到了和上述issue相似的问题，在运行run.sh单机脚本时create storage target时发生报错。其storage.log的报错信息如下：

[2025-05-21T14:08:12.070823506+08:00 storage_main:3099941 Components.cc:51 INFO] Start storageTargets ...
[2025-05-21T14:08:12.292459048+08:00 storage_main:3099941 StorageTargets.cc:69 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")
[2025-05-21T14:08:12.292473703+08:00 storage_main:3099941 StorageTargets.cc:197 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")
[2025-05-21T14:08:12.292478007+08:00 storage_main:3099941 Components.cc:51 ERROR] Start storageTargets failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")
[2025-05-21T14:08:12.292484947+08:00 storage_main:3099941 StorageServer.cc:37 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")
# ~/.profile: executed by Bourne-compatible login shells.
[2025-05-21T14:08:12.292489762+08:00 storage_main:3099941 Server.cc:39 ERROR] Server::beforeStart failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")
[2025-05-21T14:08:12.292496075+08:00 storage_main:3099941 TwoPhaseApplication.h:100 FATAL] Start server failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }")

我尝试了ulimit -n 调整最大打开文件数量，但仍然会有相同的报错。

此外我运行失败时还有这个log，不知道是否也存在影响：

### Create targets and chain table
/root/.profile: line 11: /.local/bin/env: No such file or directory
/root/.profile: line 12: /.cargo/env: No such file or directory
Encounter error: 7004(StorageClient::RoutingError)
./src/client/cli/admin/CreateTarget.cc:69, '_result' has error
Exit with 1
Originally created by @SgtPepperr on GitHub (May 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/285 > @yiyuanliu I met new error now > ``` > Init config for MGMTD version 1 > Init config for META version 1 > Init config for STORAGE version 1 > Init config for FUSE version 1 > ### Start mgmtd > ### Start meta & storage > run2.sh: line 178: 143219 Aborted (core dumped) "${BINARY}/storage_main" --app_cfg "${CONFIG}/storage_main_app.toml" --launcher_cfg "${CONFIG}/storage_main_launcher.toml" --cfg "${CONFIG}/storage_main.toml" > "${LOG}/storage_main.stdout" 2> "${LOG}/storage_main.stderr" > ### List nodes > /root/.profile: line 10: /.cargo/env: No such file or directory > Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion > 1 MGMTD PRIMARY_MGMTD salab-bncbeta03 143160 [] N/A 0 250228-dev-1-999999-3c100b90 > 50 META HEARTBEAT_CONNECTED salab-bncbeta03 143220 [] 2025-03-14 02:23:25 0 250228-dev-1-999999-3c100b90 > ### Create targets and chain table > /root/.profile: line 10: /.cargo/env: No such file or directory > Encounter error: 7004(StorageClient::RoutingError) > ./src/client/cli/admin/CreateTarget.cc:69, '_result' has error > Exit with 1 > fusermount3: failed to unmount /mnt/nvme2n1/3fsdir/mnt: Invalid argument > ``` > > here is crashed storage server > ``` > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "handlers": [ > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "normal" > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] ] > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] }, > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "handlers": { > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "normal": { > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "type": "file", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "options": { > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "path": "/mnt/nvme2n1/3fsdir/log/storage.log", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "async": "true", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "rotate": "true", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "max_files": "100", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "max_file_size": "10485760", > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] "rotate_on_open": "false" > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } > [2025-03-14T02:23:11.458128825+00:00 storage_main:143219 LogConfig.cc:96 INFO] } > [2025-03-14T02:23:11.458187230+00:00 storage_main:143219 Utils.cc:161 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal"]}},"handlers":{"normal":{"type":"file","options":{"path":"/mnt/nvme2n1/3fsdir/log/storage.log","async":"true","rotate":"true","max_files":"100","max_file_size":"10485760","rotate_on_open":"false"}}}} > *** Aborted at 1741918993 (Unix time, try 'date -d @1741918993') *** > *** Signal 6 (SIGABRT) (0x22f73) received by PID 143219 (pthread TID 0x73b6b921abc0) (linux TID 143219) (maybe from PID 143219, UID 0) (code: -6), stack trace: *** > @ 000000000121e96f _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv > /root/wayne/3fs/3FS/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449 > @ 000000000004251f (unknown) > @ 00000000000969fc pthread_kill > @ 0000000000042475 raise > @ 00000000000287f2 abort > @ 00000000012850a0 _ZNK5folly11LogCategory12admitMessageERKNS_10LogMessageE > /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogCategory.cpp:71 > @ 0000000001284556 _ZN5folly18LogStreamProcessor6logNowEv > /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:190 > @ 00000000012847a9 _ZN5folly16LogStreamVoidifyILb1EEanERSo > /root/wayne/3fs/3FS/third_party/folly/folly/logging/LogStreamProcessor.cpp:222 > @ 0000000002384482 _ZN5hf3fs19TwoPhaseApplicationINS_7storage13StorageServerEE15initApplicationEv > /root/wayne/3fs/3FS/src/common/app/TwoPhaseApplication.h:100 > -> /root/wayne/3fs/3FS/src/storage/storage.cpp > @ 0000000000fb27d6 _ZN5hf3fs15ApplicationBase3runEiPPc > /root/wayne/3fs/3FS/src/common/app/ApplicationBase.cc:65 > @ 00000000023811a3 main > /root/wayne/3fs/3FS/src/storage/storage.cpp:7 > @ 0000000000029d8f (unknown) > @ 0000000000029e3f __libc_start_main > @ 0000000000820264 _start > > ``` _Originally posted by @gaowayne in [#39](https://github.com/deepseek-ai/3FS/issues/39#issuecomment-2724139830)_ 我遇到了和上述issue相似的问题，在运行run.sh单机脚本时create storage target时发生报错。其storage.log的报错信息如下： ``` [2025-05-21T14:08:12.070823506+08:00 storage_main:3099941 Components.cc:51 INFO] Start storageTargets ... [2025-05-21T14:08:12.292459048+08:00 storage_main:3099941 StorageTargets.cc:69 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") [2025-05-21T14:08:12.292473703+08:00 storage_main:3099941 StorageTargets.cc:197 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") [2025-05-21T14:08:12.292478007+08:00 storage_main:3099941 Components.cc:51 ERROR] Start storageTargets failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") [2025-05-21T14:08:12.292484947+08:00 storage_main:3099941 StorageServer.cc:37 ERROR] error: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") # ~/.profile: executed by Bourne-compatible login shells. [2025-05-21T14:08:12.292489762+08:00 storage_main:3099941 Server.cc:39 ERROR] Server::beforeStart failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") [2025-05-21T14:08:12.292496075+08:00 storage_main:3099941 TwoPhaseApplication.h:100 FATAL] Start server failed: Storage::StorageStatFailed(4016) IoError("open \\"/root/lxy/3fstest/3fsdata/data/storage/data1/engine/64KiB/60\\" failed: Os { code: 24, kind: Uncategorized, message: \\"Too many open files\\" }") ``` 我尝试了ulimit -n 调整最大打开文件数量，但仍然会有相同的报错。 此外我运行失败时还有这个log，不知道是否也存在影响： ``` ### Create targets and chain table /root/.profile: line 11: /.local/bin/env: No such file or directory /root/.profile: line 12: /.cargo/env: No such file or directory Encounter error: 7004(StorageClient::RoutingError) ./src/client/cli/admin/CreateTarget.cc:69, '_result' has error Exit with 1 ```
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@yiyuanliu commented on GitHub (May 21, 2025):

从报错信息上看是 fd 数量超过限制了，请使用 ulimit -a 确认下修改是否生效。

<!-- gh-comment-id:2896811200 --> @yiyuanliu commented on GitHub (May 21, 2025): 从报错信息上看是 fd 数量超过限制了，请使用 `ulimit -a` 确认下修改是否生效。
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@SgtPepperr commented on GitHub (May 21, 2025):

(base) root@iZ2ze05aaerqo2xl3gffw6Z:~# ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2059908
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2059908
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

这里已经调整为了65535，还是存在该问题

<!-- gh-comment-id:2896824716 --> @SgtPepperr commented on GitHub (May 21, 2025): (base) root@iZ2ze05aaerqo2xl3gffw6Z:~# ulimit -a real-time non-blocking time (microseconds, -R) unlimited core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 2059908 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2059908 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 这里已经调整为了65535，还是存在该问题
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@SgtPepperr commented on GitHub (May 21, 2025):

确实是ulimit调整的问题，我这边开了两个终端，ulimit的操作是在另一个终端执行的，所以没有生效，在运行脚本的终端执行ulimit -n之后可以通过该步骤

<!-- gh-comment-id:2896851902 --> @SgtPepperr commented on GitHub (May 21, 2025): 确实是ulimit调整的问题，我这边开了两个终端，ulimit的操作是在另一个终端执行的，所以没有生效，在运行脚本的终端执行ulimit -n之后可以通过该步骤
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#172
No description provided.