mirror of
https://github.com/deepseek-ai/3FS.git
synced 2026-11-05 00:47:47 +00:00
[GH-ISSUE #232] 3FS fio test Input/output error #141
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/3FS#141
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @qiuxinyidian on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/232
搭建集群完毕后，执行fio dd touch test error
`
[2025-04-07T16:22:36.964508280+08:00 SvrBG1:59618 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0xfff822540628, timeout: 5s, peer address: RDMA://xx.xx.xx.10:8500
[2025-04-07T16:22:36.964532630+08:00 SvrBG1:59618 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), querySpaceInfo request: 0xfff822540628, usercall: #82662, timeout: 5s, latency: 5220ms, host: node10, address: RDMA://xx.xx.xx.10:8500
[2025-04-07T16:22:36.964581410+08:00 SvrBG0:59617 FileHelper.cc:171 ERROR] FileHelper statFs: failed to querySpaceInfo of {"app":{"nodeId":10001,"hostname":"node10","pid":74783,"serviceGroups":[{"services":["StorageSerde"],"endpoints":["RDMA://1xx.xx.xx.10:8500"]},{"services":["Core"],"endpoints":["TCP://xx.xx.xx。10:9500"]}],"releaseVersion":{"majorVersion":1,"minorVersion":0,"patchVersion":0,"commitHashShort":246715887,"buildTimeInSeconds":1074721076543489,"buildPipelineId":999999},"podname":""},"type":"STORAGE","status":"HEARTBEAT_CONNECTED","lastHeartbeatTs":"2025-04-07 12:43:06","tags":[],"configVersion":7,"configStatus":"NORMAL"}, error RPC::Timeout(2005)
[2025-04-07T16:25:41.434299960+08:00 SvrBG0:59617 MgmtdClient.cc:183 ERROR] MgmtdClient: ListClientSessions failed: RPC::Timeout(2005)
[2025-04-07T16:25:41.434355940+08:00 SvrBG0:59617 MgmtdClient.cc:203 ERROR] [MgmtdClientOp ListClientSessions No.475453] failed: MgmtdClient::PrimaryMgmtdNotFound(6000) src/client/mgmtd/MgmtdClient.cc:186, '_result' has error. latency: 3day 6h 10min 41s 179ms 664us 896ns
[2025-04-07T16:25:41.434374170+08:00 SvrBG1:59618 SessionManager.cc:48 ERROR] Failed to list active clients, error MgmtdClient::PrimaryMgmtdNotFound(6000) src/client/mgmtd/MgmtdClient.cc:186, '_result' has error
[2025-04-07T16:25:56.480824840+08:00 SvrProc1:59614 ChainAllocator.h:107 ERROR] Failed to allocate for layout {"tableId":1,"tableVersion":0,"chunkSize":1048576,"stripeSize":16,"chains":{"type":"Empty","value":"empty"}}, chain table 1 have only 6 chains.
如果chains不够，搭建完集群后如何修改呢？
`
@qiuxinyidian commented on GitHub (Apr 7, 2025):
最小集群搭建有什么要求吗？比如对磁盘？
@qiuxinyidian commented on GitHub (Apr 7, 2025):
hf3fs_fuse_main:15740 MetaClient.cc:404 INFO] Op create{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"514ac38c-e75a-46c7-a591-0dfe3a6aba86","hostname":"node167"},"forward":0,"uuid":"fbf75471-5ff7-4b98-a943-d6f5a37a7842","path":{"parent":"0x0000000000007000","path":"123.txt"},"session":{"client":{"uuid":"514ac38c-e75a-46c7-a591-0dfe3a6aba86","hostname":"node167"},"session":"86eb13b4-d5a3-415c-a566-29e9e38371d9"},"flags":"8841","perm":420,"layout":null,"removeChunksBatchSize":32,"dynStripe":true}, result Meta::InvalidFileLayout(3013) try to allocate 16 chains from ChainTableId(1), found 12
@dev-wuyi commented on GitHub (Apr 8, 2025):
你好问题解决了吗，我也遇到了类似的问题。Failed to allocate for layout {"tableId":1,"tableVersion":0,"chunkSize":1048576,"stripeSize":16,"chains":{"type":"Empty","value":"empty"}}, chain table 1 have only 4 chains， 我的配置是 python ~/3fs/deploy/data_placement/src/model/data_placement.py
-ql -relax -type CR --num_nodes 2 --replication_factor 2 --min_targets_per_disk 2
@dev-wuyi commented on GitHub (Apr 8, 2025):
~/3fs/deploy/data_placement/src/setup/gen_chain_table.py
--chain_table_type CR --node_id_begin 10001 --node_id_end 10002
--num_disks_per_node 2 --num_targets_per_disk 2
--target_id_prefix 1 --chain_id_prefix 9
--incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle
@dev-wuyi commented on GitHub (Apr 8, 2025):
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16" 在执行集群初始化的时候，这里初始化成16了。 所以在后面进行命令配置时，应该改一下参数，保证/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://xxx]' "list-chain-tables" 输出的chaincount大于16
@dev-wuyi commented on GitHub (Apr 8, 2025):
chain_count = (num_nodes * num_disks_per_node * num_targets_per_disk) // group_size
@qiuxinyidian commented on GitHub (Apr 8, 2025):
解决了，简单来讲停服务，删数据foundationdb中所有的数据，初始化，重新注册
@qiuxinyidian commented on GitHub (Apr 8, 2025):
2（nodes）x 2（disks per node）x 2（target）/2（rep） =4
@qiuxinyidian commented on GitHub (Apr 8, 2025):
stripesize 小于等于 target总数才行，至于是否还有其他关系还没来及看代码。
@dev-wuyi commented on GitHub (Apr 9, 2025):
我也解决了，我是重新执行 python ~/3fs/deploy/data_placement/src/model/data_placement.py
-ql -relax -type CR --num_nodes 2 --replication_factor 2 --min_targets_per_disk 12，增大targets，保证chaincount大于stripesize 即可。
@Navacore commented on GitHub (Apr 20, 2025):
I don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.I
don't know what they're accusing me of, please stop sending messages.
El mar, 8 de abr. de 2025 5:54 a. m., dev-wuyi @.***>
escribió:
@leonail1 commented on GitHub (Aug 21, 2025):
我试了，成功了，我把 min_targets_per_dist 改成 16