deepseek-ai/3FS

Fork 0

mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00

[GH-ISSUE #335] 替换磁盘之后，对应 targets 一直 OFFLINE #205

New issue

Open

opened 2026-10-26 09:47:07 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:47:07 +00:00

Owner

Originally created by @wangzhuzhen on GitHub (Sep 1, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/335

搭建了一个16 存储个节点的 3FS ，每个存储节点使用一个 SSD 磁盘，已经使用了一段时间。

目前有一个节点 SSD 磁盘物理故障，无法恢复。因此换了个磁盘，但是 storage service 重启之后，对应节点的 storage targets 一直 OFFLINE，没有出现自动恢复的情况。请问接下来该怎么处理？

节点状态都正常

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-nodes"
Id     Type     Status               Hostname           Pid    Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
3      MGMTD    PRIMARY_MGMTD        gpu25-n204-c01-6u  18314  []    N/A                  0              250228-dev-1-999999-394583db
1      MGMTD    HEARTBEAT_CONNECTED  gpu23-n204-b17-6u  31749  []    2025-09-01 06:44:05  0              250228-dev-1-999999-394583db
2      MGMTD    HEARTBEAT_CONNECTED  gpu24-n204-b18-6u  46455  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
100    META     HEARTBEAT_CONNECTED  gpu23-n204-b17-6u  36418  []    2025-09-01 06:44:03  0              250228-dev-1-999999-394583db
200    META     HEARTBEAT_CONNECTED  gpu24-n204-b18-6u  51648  []    2025-09-01 06:44:08  0              250228-dev-1-999999-394583db
300    META     HEARTBEAT_CONNECTED  gpu25-n204-c01-6u  22979  []    2025-09-01 06:44:08  0              250228-dev-1-999999-394583db
10009  STORAGE  HEARTBEAT_CONNECTED  gpu09-n204-a13-6u  18369  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10010  STORAGE  HEARTBEAT_CONNECTED  gpu10-n204-a15-6u  62739  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10011  STORAGE  HEARTBEAT_CONNECTED  gpu11-n204-a16-6u  39859  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10012  STORAGE  HEARTBEAT_CONNECTED  gpu12-n204-a18-6u  30478  []    2025-09-01 06:44:10  0              250228-dev-1-999999-394583db
10013  STORAGE  HEARTBEAT_CONNECTED  gpu13-n204-b03-6u  47478  []    2025-09-01 06:44:11  0              250228-dev-1-999999-394583db
10014  STORAGE  HEARTBEAT_CONNECTED  gpu14-n204-b04-6u  38979  []    2025-09-01 06:44:11  0              250228-dev-1-999999-394583db
10015  STORAGE  HEARTBEAT_CONNECTED  gpu15-n204-b06-6u  7326   []    2025-09-01 06:44:11  0              250228-dev-1-999999-394583db
10016  STORAGE  HEARTBEAT_CONNECTED  gpu16-n204-b07-6u  54174  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10017  STORAGE  HEARTBEAT_CONNECTED  gpu17-n204-b08-6u  7493   []    2025-09-01 06:44:11  0              250228-dev-1-999999-394583db
10018  STORAGE  HEARTBEAT_CONNECTED  gpu18-n204-b09-6u  59328  []    2025-09-01 06:44:10  0              250228-dev-1-999999-394583db
10019  STORAGE  HEARTBEAT_CONNECTED  gpu19-n204-b11-6u  31081  []    2025-09-01 06:44:11  0              250228-dev-1-999999-394583db
10020  STORAGE  HEARTBEAT_CONNECTED  gpu20-n204-b12-6u  21612  []    2025-09-01 06:44:13  0              250228-dev-1-999999-394583db
10021  STORAGE  HEARTBEAT_CONNECTED  gpu21-n204-b14-6u  62261  []    2025-09-01 06:44:13  0              250228-dev-1-999999-394583db
10022  STORAGE  HEARTBEAT_CONNECTED  gpu22-n204-b15-6u  38910  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10023  STORAGE  HEARTBEAT_CONNECTED  gpu23-n204-b17-6u  58510  []    2025-09-01 06:44:12  0              250228-dev-1-999999-394583db
10024  STORAGE  HEARTBEAT_CONNECTED  gpu24-n204-b18-6u  64182  []    2025-09-01 06:44:10  0              250228-dev-1-999999-394583db

有OFFLINE 的 Chains(全部是同一个节点上)：

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-chains" | grep OFFLINE
900100001  1             2             SERVING(2/3)  []              101002000101(SERVING-UPTODATE)  101002200101(SERVING-UPTODATE)  101002300101(OFFLINE-OFFLINE)
900100004  1             2             SERVING(2/3)  []              101001800101(SERVING-UPTODATE)  101002100101(SERVING-UPTODATE)  101002300102(OFFLINE-OFFLINE)
900100007  1             2             SERVING(2/3)  []              101001200103(SERVING-UPTODATE)  101001400101(SERVING-UPTODATE)  101002300103(OFFLINE-OFFLINE)
900100017  1             5             SERVING(2/3)  []              101002000104(SERVING-UPTODATE)  101001600103(SERVING-UPTODATE)  101002300104(OFFLINE-OFFLINE)
900100027  1             2             SERVING(2/3)  []              101001300106(SERVING-UPTODATE)  101001400104(SERVING-UPTODATE)  101002300105(OFFLINE-OFFLINE)
900100030  1             2             SERVING(2/3)  []              101001900105(SERVING-UPTODATE)  101002200107(SERVING-UPTODATE)  101002300106(OFFLINE-OFFLINE)
900100032  1             2             SERVING(2/3)  []              101001100105(SERVING-UPTODATE)  101001700106(SERVING-UPTODATE)  101002300107(OFFLINE-OFFLINE)
900100033  1             2             SERVING(2/3)  []              101001200106(SERVING-UPTODATE)  101001500108(SERVING-UPTODATE)  101002300108(OFFLINE-OFFLINE)
900100037  1             2             SERVING(2/3)  []              101001300108(SERVING-UPTODATE)  101001800107(SERVING-UPTODATE)  101002300109(OFFLINE-OFFLINE)

对应节点上的 storage target 状态 OFFLINE

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-targets" | grep OFFLINE
101002300108  900100033  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300103  900100007  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300101  900100001  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300106  900100030  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300107  900100032  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300102  900100004  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300105  900100027  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300109  900100037  TAIL    OFFLINE      OFFLINE     N/A     N/A        0
101002300104  900100017  TAIL    OFFLINE      OFFLINE     N/A     N/A        0

正常节点上，SSD 存储目录下 /storage/data1/3fs/ 包含 targetID 目录和 engine 目录

/storage/data1/3fs# ls -l
total 0
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400101
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400102
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400103
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400104
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400105
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400106
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400107
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400108
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400109
drwxr-xr-x 14 root root 212 Aug 12 06:25 engine

而替换了磁盘的节点上，SSD 存储目录下 /storage/data1/3fs/ 只有 engine 目录

# ls -l
total 0
drwxr-xr-x 14 root root 212 Sep  1 03:26 engine

下一步需要如何操作，才能恢复这个节点上的 storage target 状态为正常的 SERVING /UPTODATE ？是要 unregister-node 然后 register-node ？或者是要 remove-target 然后 create-target ？

Originally created by @wangzhuzhen on GitHub (Sep 1, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/335 搭建了一个16 存储个节点的 3FS ，每个存储节点使用一个 SSD 磁盘，已经使用了一段时间。目前有一个节点 SSD 磁盘物理故障，无法恢复。因此换了个磁盘，但是 storage service 重启之后，对应节点的 storage targets 一直 OFFLINE，没有出现自动恢复的情况。请问接下来该怎么处理？节点状态都正常 ``` # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-nodes" Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 3 MGMTD PRIMARY_MGMTD gpu25-n204-c01-6u 18314 [] N/A 0 250228-dev-1-999999-394583db 1 MGMTD HEARTBEAT_CONNECTED gpu23-n204-b17-6u 31749 [] 2025-09-01 06:44:05 0 250228-dev-1-999999-394583db 2 MGMTD HEARTBEAT_CONNECTED gpu24-n204-b18-6u 46455 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 100 META HEARTBEAT_CONNECTED gpu23-n204-b17-6u 36418 [] 2025-09-01 06:44:03 0 250228-dev-1-999999-394583db 200 META HEARTBEAT_CONNECTED gpu24-n204-b18-6u 51648 [] 2025-09-01 06:44:08 0 250228-dev-1-999999-394583db 300 META HEARTBEAT_CONNECTED gpu25-n204-c01-6u 22979 [] 2025-09-01 06:44:08 0 250228-dev-1-999999-394583db 10009 STORAGE HEARTBEAT_CONNECTED gpu09-n204-a13-6u 18369 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10010 STORAGE HEARTBEAT_CONNECTED gpu10-n204-a15-6u 62739 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10011 STORAGE HEARTBEAT_CONNECTED gpu11-n204-a16-6u 39859 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10012 STORAGE HEARTBEAT_CONNECTED gpu12-n204-a18-6u 30478 [] 2025-09-01 06:44:10 0 250228-dev-1-999999-394583db 10013 STORAGE HEARTBEAT_CONNECTED gpu13-n204-b03-6u 47478 [] 2025-09-01 06:44:11 0 250228-dev-1-999999-394583db 10014 STORAGE HEARTBEAT_CONNECTED gpu14-n204-b04-6u 38979 [] 2025-09-01 06:44:11 0 250228-dev-1-999999-394583db 10015 STORAGE HEARTBEAT_CONNECTED gpu15-n204-b06-6u 7326 [] 2025-09-01 06:44:11 0 250228-dev-1-999999-394583db 10016 STORAGE HEARTBEAT_CONNECTED gpu16-n204-b07-6u 54174 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10017 STORAGE HEARTBEAT_CONNECTED gpu17-n204-b08-6u 7493 [] 2025-09-01 06:44:11 0 250228-dev-1-999999-394583db 10018 STORAGE HEARTBEAT_CONNECTED gpu18-n204-b09-6u 59328 [] 2025-09-01 06:44:10 0 250228-dev-1-999999-394583db 10019 STORAGE HEARTBEAT_CONNECTED gpu19-n204-b11-6u 31081 [] 2025-09-01 06:44:11 0 250228-dev-1-999999-394583db 10020 STORAGE HEARTBEAT_CONNECTED gpu20-n204-b12-6u 21612 [] 2025-09-01 06:44:13 0 250228-dev-1-999999-394583db 10021 STORAGE HEARTBEAT_CONNECTED gpu21-n204-b14-6u 62261 [] 2025-09-01 06:44:13 0 250228-dev-1-999999-394583db 10022 STORAGE HEARTBEAT_CONNECTED gpu22-n204-b15-6u 38910 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10023 STORAGE HEARTBEAT_CONNECTED gpu23-n204-b17-6u 58510 [] 2025-09-01 06:44:12 0 250228-dev-1-999999-394583db 10024 STORAGE HEARTBEAT_CONNECTED gpu24-n204-b18-6u 64182 [] 2025-09-01 06:44:10 0 250228-dev-1-999999-394583db ``` 有OFFLINE 的 Chains(全部是同一个节点上)： ``` # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-chains" | grep OFFLINE 900100001 1 2 SERVING(2/3) [] 101002000101(SERVING-UPTODATE) 101002200101(SERVING-UPTODATE) 101002300101(OFFLINE-OFFLINE) 900100004 1 2 SERVING(2/3) [] 101001800101(SERVING-UPTODATE) 101002100101(SERVING-UPTODATE) 101002300102(OFFLINE-OFFLINE) 900100007 1 2 SERVING(2/3) [] 101001200103(SERVING-UPTODATE) 101001400101(SERVING-UPTODATE) 101002300103(OFFLINE-OFFLINE) 900100017 1 5 SERVING(2/3) [] 101002000104(SERVING-UPTODATE) 101001600103(SERVING-UPTODATE) 101002300104(OFFLINE-OFFLINE) 900100027 1 2 SERVING(2/3) [] 101001300106(SERVING-UPTODATE) 101001400104(SERVING-UPTODATE) 101002300105(OFFLINE-OFFLINE) 900100030 1 2 SERVING(2/3) [] 101001900105(SERVING-UPTODATE) 101002200107(SERVING-UPTODATE) 101002300106(OFFLINE-OFFLINE) 900100032 1 2 SERVING(2/3) [] 101001100105(SERVING-UPTODATE) 101001700106(SERVING-UPTODATE) 101002300107(OFFLINE-OFFLINE) 900100033 1 2 SERVING(2/3) [] 101001200106(SERVING-UPTODATE) 101001500108(SERVING-UPTODATE) 101002300108(OFFLINE-OFFLINE) 900100037 1 2 SERVING(2/3) [] 101001300108(SERVING-UPTODATE) 101001800107(SERVING-UPTODATE) 101002300109(OFFLINE-OFFLINE) ``` 对应节点上的 storage target 状态 OFFLINE ``` # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-targets" | grep OFFLINE 101002300108 900100033 TAIL OFFLINE OFFLINE N/A N/A 0 101002300103 900100007 TAIL OFFLINE OFFLINE N/A N/A 0 101002300101 900100001 TAIL OFFLINE OFFLINE N/A N/A 0 101002300106 900100030 TAIL OFFLINE OFFLINE N/A N/A 0 101002300107 900100032 TAIL OFFLINE OFFLINE N/A N/A 0 101002300102 900100004 TAIL OFFLINE OFFLINE N/A N/A 0 101002300105 900100027 TAIL OFFLINE OFFLINE N/A N/A 0 101002300109 900100037 TAIL OFFLINE OFFLINE N/A N/A 0 101002300104 900100017 TAIL OFFLINE OFFLINE N/A N/A 0 ``` 正常节点上，SSD 存储目录下 /storage/data1/3fs/ 包含 targetID 目录和 engine 目录 ``` /storage/data1/3fs# ls -l total 0 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400101 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400102 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400103 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400104 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400105 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400106 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400107 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400108 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400109 drwxr-xr-x 14 root root 212 Aug 12 06:25 engine ``` 而替换了磁盘的节点上，SSD 存储目录下 /storage/data1/3fs/ 只有 engine 目录 ``` # ls -l total 0 drwxr-xr-x 14 root root 212 Sep 1 03:26 engine ``` 下一步需要如何操作，才能恢复这个节点上的 storage target 状态为正常的 SERVING /UPTODATE ？是要 unregister-node 然后 register-node ？或者是要 remove-target 然后 create-target ？

bsakaguchi commented

2026-10-26 09:47:09 +00:00

Author

Owner

@wangzhuzhen commented on GitHub (Sep 1, 2025):

目前通过如下方式解决了替换磁盘之后对应磁盘 targets 都 OFFLINE 的问题，但是是否是官方推荐的解决方案个人无法判断。简单来说就是在新磁盘上构建对应的 targets 配置文件。具体过程如下：

首先查看正常的节点，发现 SSD 磁盘上实际上是两类目录： targetIds 和 engine，如下所示:

/storage/data1/3fs# ls -l
total 0
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400101
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400102
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400103
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400104
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400105
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400106
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400107
drwxr-xr-x  2 root root  25 Aug 12 08:30 101002400108
drwxr-xr-x  2 root root  33 Aug 12 08:30 101002400109
drwxr-xr-x 14 root root 212 Aug 12 06:25 engine

对于异常节点，发现 SSD 磁盘上实际只有 engine目录:

/storage/data1/3fs# ls -l
total 0
drwxr-xr-x 14 root root 212 Aug 12 06:25 engine

分析正常节点的 /storage/data1/3fs/ 目录中其中每个 targetId （如 101002400101）目录下面实际只有一个名称为 target.toml 的文件

1# ls -l   /storage/data1/3fs/101002400101
total 4
-rw-r--r-- 1 root root 387 Aug 12 08:30 target.toml

target.toml 的文件的内容如下:

# cat  /storage/data1/3fs/101002400101/target.toml
only_chunk_engine = true
has_sentinel = true
kv_store_type = "LevelDB"
chunk_size_list = ["512KB","1MB","2MB","4MB","16MB","64MB"]
physical_file_count = 256
allow_existing_targets = true
allow_disk_without_uuid = false
kv_store_name = "meta"
block_device_uuid = "a28026f8-01e2-4edd-bc2c-16af4e887b6f"
target_id = 101002400101
chain_id = 900100002
path = "storage/data1/3fs/101002400101"

再对比了多个节点之后，可以看到实际上 target.toml 描述 target 关联的元数据信息(只需要块设备ID、targetID、ChainID就可以构建 target.toml 文件)，如：

块设备 uuid -----> "a28026f8-01e2-4edd-bc2c-16af4e887b6f"
target_id -----> 101002400101
chain_id ------> 900100002
path -----> "storage/data1/3fs/101002400101"

因此只需要知道块设备ID、targetID、ChainID 就可以构建 target.toml 文件。

根据前面的分析，分别获取对应信息

块设备ID 获取方式（根据设备名称 nvme4n1p1 找到对应的 uuid）:

# ls -l  /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 Sep  1 03:38 fd4f83ce-59e7-4056-8bff-128ab0f9d099 -> ../../nvme4n1p1

target_id 和 chain_id 信息 (list-targets 命令可以查看到, 如下命令输出第一列是 targetID，第二列是 chainID):

# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-targets" | grep  OFFLINE  
101002300108  900100033  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300103  900100007  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300101  900100001  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300106  900100030  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300107  900100032  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300102  900100004  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300105  900100027  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300109  900100037  TAIL    OFFLINE      OFFLINE     10023   0          0
101002300104  900100017  TAIL    OFFLINE      OFFLINE     10023   0          0

根据以上信息，构建故障节点上构建对应的 targets 目录，

cd  /storage/data1/3fs
mkdir  -p  101002300101
mkdir  -p  101002300102
mkdir  -p  101002300103
mkdir  -p  101002300104
mkdir  -p  101002300105
mkdir  -p  101002300106
mkdir  -p  101002300107
mkdir  -p  101002300108
mkdir  -p  101002300109

然后在每个目录 target 目录(如 101002300101)中创建 target.toml 文件：

block_device_uuid 都相同
target_id 和 chain_id 根据 list-targets 输出确认一一对应
path 的路径最后一段是 target_id

如 target_id 为 101002300101 的 storage 对对应的 target.toml 文件配置如下，

# cat  /storage/data1/3fs/101002300101/target.toml
only_chunk_engine = true
has_sentinel = true
kv_store_type = "LevelDB"
chunk_size_list = ["512KB","1MB","2MB","4MB","16MB","64MB"]
physical_file_count = 256
allow_existing_targets = true
allow_disk_without_uuid = false
kv_store_name = "meta"
block_device_uuid = "fd4f83ce-59e7-4056-8bff-128ab0f9d099"
target_id = 101002300101
chain_id = 900100001
path = "storage/data1/3fs/101002300101"

其他 101002300102 ~ 101002300109 几个 target 同样处理，但是操作过程务必确保各种 id 对应关联关系。

最后重启 storage service

systemctl  status  storage_main

再次查看节点信息，所有 targets 恢复正常，所有 chains 恢复正常。

@wangzhuzhen commented on GitHub (Sep 1, 2025): 目前通过如下方式解决了替换磁盘之后对应磁盘 targets 都 OFFLINE 的问题，但是是否是官方推荐的解决方案个人无法判断。简单来说就是在新磁盘上构建对应的 targets 配置文件。具体过程如下：首先查看正常的节点，发现 SSD 磁盘上实际上是两类目录： targetIds 和 engine，如下所示: ``` /storage/data1/3fs# ls -l total 0 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400101 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400102 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400103 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400104 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400105 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400106 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400107 drwxr-xr-x 2 root root 25 Aug 12 08:30 101002400108 drwxr-xr-x 2 root root 33 Aug 12 08:30 101002400109 drwxr-xr-x 14 root root 212 Aug 12 06:25 engine ``` 对于异常节点，发现 SSD 磁盘上实际只有 engine目录: ``` /storage/data1/3fs# ls -l total 0 drwxr-xr-x 14 root root 212 Aug 12 06:25 engine ``` 分析正常节点的 /storage/data1/3fs/ 目录中其中每个 targetId （如 101002400101）目录下面实际只有一个名称为 target.toml 的文件 ``` 1# ls -l /storage/data1/3fs/101002400101 total 4 -rw-r--r-- 1 root root 387 Aug 12 08:30 target.toml ``` target.toml 的文件的内容如下: ``` # cat /storage/data1/3fs/101002400101/target.toml only_chunk_engine = true has_sentinel = true kv_store_type = "LevelDB" chunk_size_list = ["512KB","1MB","2MB","4MB","16MB","64MB"] physical_file_count = 256 allow_existing_targets = true allow_disk_without_uuid = false kv_store_name = "meta" block_device_uuid = "a28026f8-01e2-4edd-bc2c-16af4e887b6f" target_id = 101002400101 chain_id = 900100002 path = "storage/data1/3fs/101002400101" ``` 再对比了多个节点之后，可以看到实际上 target.toml 描述 target 关联的元数据信息(只需要块设备ID、targetID、ChainID就可以构建 target.toml 文件)，如： * 块设备 uuid -----> "a28026f8-01e2-4edd-bc2c-16af4e887b6f" * target_id -----> 101002400101 * chain_id ------> 900100002 * path -----> "storage/data1/3fs/101002400101" 因此只需要知道块设备ID、targetID、ChainID 就可以构建 target.toml 文件。根据前面的分析，分别获取对应信息 * 块设备ID 获取方式（根据设备名称 nvme4n1p1 找到对应的 uuid）: ``` # ls -l /dev/disk/by-uuid/ total 0 lrwxrwxrwx 1 root root 15 Sep 1 03:38 fd4f83ce-59e7-4056-8bff-128ab0f9d099 -> ../../nvme4n1p1 ``` * target_id 和 chain_id 信息 (list-targets 命令可以查看到, 如下命令输出第一列是 targetID，第二列是 chainID): ``` # /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://100.70.8.24:8001","RDMA://100.70.8.23:8001","RDMA://100.70.8.25:8001"]' "list-targets" | grep OFFLINE 101002300108 900100033 TAIL OFFLINE OFFLINE 10023 0 0 101002300103 900100007 TAIL OFFLINE OFFLINE 10023 0 0 101002300101 900100001 TAIL OFFLINE OFFLINE 10023 0 0 101002300106 900100030 TAIL OFFLINE OFFLINE 10023 0 0 101002300107 900100032 TAIL OFFLINE OFFLINE 10023 0 0 101002300102 900100004 TAIL OFFLINE OFFLINE 10023 0 0 101002300105 900100027 TAIL OFFLINE OFFLINE 10023 0 0 101002300109 900100037 TAIL OFFLINE OFFLINE 10023 0 0 101002300104 900100017 TAIL OFFLINE OFFLINE 10023 0 0 ``` 根据以上信息，构建故障节点上构建对应的 targets 目录， ``` cd /storage/data1/3fs mkdir -p 101002300101 mkdir -p 101002300102 mkdir -p 101002300103 mkdir -p 101002300104 mkdir -p 101002300105 mkdir -p 101002300106 mkdir -p 101002300107 mkdir -p 101002300108 mkdir -p 101002300109 ``` 然后在每个目录 target 目录(如 101002300101)中创建 target.toml 文件： * block_device_uuid 都相同 * target_id 和 chain_id 根据 list-targets 输出确认一一对应 * path 的路径最后一段是 target_id 如 target_id 为 101002300101 的 storage 对对应的 target.toml 文件配置如下， ``` # cat /storage/data1/3fs/101002300101/target.toml only_chunk_engine = true has_sentinel = true kv_store_type = "LevelDB" chunk_size_list = ["512KB","1MB","2MB","4MB","16MB","64MB"] physical_file_count = 256 allow_existing_targets = true allow_disk_without_uuid = false kv_store_name = "meta" block_device_uuid = "fd4f83ce-59e7-4056-8bff-128ab0f9d099" target_id = 101002300101 chain_id = 900100001 path = "storage/data1/3fs/101002300101" ``` 其他 101002300102 ~ 101002300109 几个 target 同样处理，但是操作过程务必确保各种 id 对应关联关系。最后重启 storage service ``` systemctl status storage_main ``` 再次查看节点信息，所有 targets 恢复正常，所有 chains 恢复正常。

bsakaguchi commented

2026-10-26 09:47:09 +00:00

Author

Owner

@kaede11 commented on GitHub (Sep 30, 2025):

换盘后，执行create-target/update-chain操作了吗？

@kaede11 commented on GitHub (Sep 30, 2025): 换盘后，执行create-target/update-chain操作了吗？

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#205

No description provided.

Rows
Columns