deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #193] 3fs write data Input/output error #119

New issue
Open
opened 2026-10-26 09:45:30 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:45:30 +00:00
Owner
Copy link

Originally created by @cloudp1216 on GitHub (Mar 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/193

root@3fs:/3fs/stage# cp /etc/fstab .
cp: failed to close './fstab': Input/output error

root@3fs:/var/log/3fs# tail -f hf3fs_fuse_main-err.log 
[2025-03-19T09:46:44.854210120+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 1s 600ms}, retry true
[2025-03-19T09:46:53.125981138+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 3s 200ms}, retry true
[2025-03-19T09:47:04.593119903+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 6s 400ms}, retry true
[2025-03-19T09:47:19.690132036+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:47:34.793061291+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:47:49.896841950+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:48:04.992859500+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:48:20.093259122+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:48:35.172193761+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:48:50.227867018+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true
[2025-03-19T09:49:05.293091134+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true

root@3fs:/var/log/3fs# tail -f hf3fs_meta_main-err.log
[2025-03-19T09:48:35.130991880+08:00 SvrProc1:13349 StorageClientImpl.cc:1173 ERROR] Give up retrying 2/16 ops 0x7febc6e493e0 after #2 retries, elapsed time: 5004ms, max retry time: 5000ms, user: Uid(0)/Gid(0), usercall: #657
[2025-03-19T09:48:35.131095346+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] #1/16 Failed queryLastChunk operation 0x7febc6ff4e00 with id RequestId(0), routing target: ChainId(900600002)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #657, status: StorageClient::NotAvailable(7005)
[2025-03-19T09:48:35.131132985+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] #2/16 Failed queryLastChunk operation 0x7febc6ff4ed8 with id RequestId(0), routing target: ChainId(900600001)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #657, status: StorageClient::NotAvailable(7005)
[2025-03-19T09:48:35.131158815+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] Finally collected 2/16 failed queryLastChunk operations 0x7febc6e493e0 issued by user Uid(0)/Gid(0), usercall: #657
[2025-03-19T09:48:35.131214497+08:00 SvrProc1:13349 FileOperation.cc:137 ERROR] Failed to quertLastChunk from chain ChainId(900600002), err StorageClient::NotAvailable(7005)
[2025-03-19T09:48:50.183682702+08:00 SvrProc0:13348 StorageClientImpl.cc:1173 ERROR] Give up retrying 2/16 ops 0x7feb9991afe0 after #2 retries, elapsed time: 5002ms, max retry time: 5000ms, user: Uid(0)/Gid(0), usercall: #667
[2025-03-19T09:48:50.183772583+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] #1/16 Failed queryLastChunk operation 0x7feb9990e000 with id RequestId(0), routing target: ChainId(900600002)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #667, status: StorageClient::NotAvailable(7005)
[2025-03-19T09:48:50.183832383+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] #2/16 Failed queryLastChunk operation 0x7feb9990e0d8 with id RequestId(0), routing target: ChainId(900600001)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #667, status: StorageClient::NotAvailable(7005)
[2025-03-19T09:48:50.183857453+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] Finally collected 2/16 failed queryLastChunk operations 0x7feb9991afe0 issued by user Uid(0)/Gid(0), usercall: #667
[2025-03-19T09:48:50.183911273+08:00 SvrProc0:13348 FileOperation.cc:137 ERROR] Failed to quertLastChunk from chain ChainId(900600002), err StorageClient::NotAvailable(7005)

root@3fs:/var/log/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["IPoIB://10.252.0.100:8000"]' "list-chains"
ChainId    ReferencedBy  ChainVersion  Status       PreferredOrder  Target                          Target                          Target
900100001  1             6             SERVING      []              101000100101(SERVING-UPTODATE)  101000300101(SERVING-UPTODATE)  101000200101(SERVING-UPTODATE)
900100002  1             6             SERVING      []              101000100102(SERVING-UPTODATE)  101000300102(SERVING-UPTODATE)  101000200102(SERVING-UPTODATE)
900100003  1             6             SERVING      []              101000100103(SERVING-UPTODATE)  101000300103(SERVING-UPTODATE)  101000200103(SERVING-UPTODATE)
900100004  1             6             SERVING      []              101000100104(SERVING-UPTODATE)  101000300104(SERVING-UPTODATE)  101000200104(SERVING-UPTODATE)
900100005  1             6             SERVING      []              101000100105(SERVING-UPTODATE)  101000300105(SERVING-UPTODATE)  101000200105(SERVING-UPTODATE)
900100006  1             6             SERVING      []              101000100106(SERVING-UPTODATE)  101000300106(SERVING-UPTODATE)  101000200106(SERVING-UPTODATE)
900200001  1             6             SERVING      []              101000100201(SERVING-UPTODATE)  101000300201(SERVING-UPTODATE)  101000200201(SERVING-UPTODATE)
900200002  1             6             SERVING      []              101000100202(SERVING-UPTODATE)  101000300202(SERVING-UPTODATE)  101000200202(SERVING-UPTODATE)
900200003  1             6             SERVING      []              101000100203(SERVING-UPTODATE)  101000300203(SERVING-UPTODATE)  101000200203(SERVING-UPTODATE)
900200004  1             6             SERVING      []              101000100204(SERVING-UPTODATE)  101000300204(SERVING-UPTODATE)  101000200204(SERVING-UPTODATE)
900200005  1             6             SERVING      []              101000100205(SERVING-UPTODATE)  101000300205(SERVING-UPTODATE)  101000200205(SERVING-UPTODATE)
900200006  1             6             SERVING      []              101000100206(SERVING-UPTODATE)  101000300206(SERVING-UPTODATE)  101000200206(SERVING-UPTODATE)
900300001  1             6             SERVING      []              101000100301(SERVING-UPTODATE)  101000300301(SERVING-UPTODATE)  101000200301(SERVING-UPTODATE)
900300002  1             6             SERVING      []              101000100302(SERVING-UPTODATE)  101000300302(SERVING-UPTODATE)  101000200302(SERVING-UPTODATE)
900300003  1             6             SERVING      []              101000100303(SERVING-UPTODATE)  101000300303(SERVING-UPTODATE)  101000200303(SERVING-UPTODATE)
900300004  1             6             SERVING      []              101000100304(SERVING-UPTODATE)  101000300304(SERVING-UPTODATE)  101000200304(SERVING-UPTODATE)
900300005  1             6             SERVING      []              101000100305(SERVING-UPTODATE)  101000300305(SERVING-UPTODATE)  101000200305(SERVING-UPTODATE)
900300006  1             6             SERVING      []              101000100306(SERVING-UPTODATE)  101000300306(SERVING-UPTODATE)  101000200306(SERVING-UPTODATE)
900400001  1             6             SERVING      []              101000100401(SERVING-UPTODATE)  101000300401(SERVING-UPTODATE)  101000200401(SERVING-UPTODATE)
900400002  1             6             SERVING      []              101000100402(SERVING-UPTODATE)  101000300402(SERVING-UPTODATE)  101000200402(SERVING-UPTODATE)
900400003  1             6             SERVING      []              101000100403(SERVING-UPTODATE)  101000300403(SERVING-UPTODATE)  101000200403(SERVING-UPTODATE)
900400004  1             6             SERVING      []              101000100404(SERVING-UPTODATE)  101000300404(SERVING-UPTODATE)  101000200404(SERVING-UPTODATE)
900400005  1             6             SERVING      []              101000100405(SERVING-UPTODATE)  101000300405(SERVING-UPTODATE)  101000200405(SERVING-UPTODATE)
900400006  1             6             SERVING      []              101000100406(SERVING-UPTODATE)  101000300406(SERVING-UPTODATE)  101000200406(SERVING-UPTODATE)
900500001  1             6             SERVING      []              101000100501(SERVING-UPTODATE)  101000300501(SERVING-UPTODATE)  101000200501(SERVING-UPTODATE)
900500002  1             6             SERVING      []              101000100502(SERVING-UPTODATE)  101000300502(SERVING-UPTODATE)  101000200502(SERVING-UPTODATE)
900500003  1             6             SERVING      []              101000100503(SERVING-UPTODATE)  101000300503(SERVING-UPTODATE)  101000200503(SERVING-UPTODATE)
900500004  1             6             SERVING      []              101000100504(SERVING-UPTODATE)  101000300504(SERVING-UPTODATE)  101000200504(SERVING-UPTODATE)
900500005  1             6             SERVING      []              101000100505(SERVING-UPTODATE)  101000300505(SERVING-UPTODATE)  101000200505(SERVING-UPTODATE)
900500006  1             6             SERVING      []              101000100506(SERVING-UPTODATE)  101000300506(SERVING-UPTODATE)  101000200506(SERVING-UPTODATE)
900600001  1             9             UNAVAILABLE  []              101000200601(LASTSRV-OFFLINE)   101000100601(OFFLINE-OFFLINE)   101000300601(OFFLINE-OFFLINE)
900600002  1             9             UNAVAILABLE  []              101000200602(LASTSRV-OFFLINE)   101000100602(OFFLINE-OFFLINE)   101000300602(OFFLINE-OFFLINE)
900600003  1             9             UNAVAILABLE  []              101000200603(LASTSRV-OFFLINE)   101000100603(OFFLINE-OFFLINE)   101000300603(OFFLINE-OFFLINE)
900600004  1             9             UNAVAILABLE  []              101000200604(LASTSRV-OFFLINE)   101000100604(OFFLINE-OFFLINE)   101000300604(OFFLINE-OFFLINE)
900600005  1             9             UNAVAILABLE  []              101000200605(LASTSRV-OFFLINE)   101000100605(OFFLINE-OFFLINE)   101000300605(OFFLINE-OFFLINE)
900600006  1             9             UNAVAILABLE  []              101000200606(LASTSRV-OFFLINE)   101000100606(OFFLINE-OFFLINE)   101000300606(OFFLINE-OFFLINE)
900700001  1             6             SERVING      []              101000100701(SERVING-UPTODATE)  101000300701(SERVING-UPTODATE)  101000200701(SERVING-UPTODATE)
900700002  1             6             SERVING      []              101000100702(SERVING-UPTODATE)  101000300702(SERVING-UPTODATE)  101000200702(SERVING-UPTODATE)
900700003  1             6             SERVING      []              101000100703(SERVING-UPTODATE)  101000300703(SERVING-UPTODATE)  101000200703(SERVING-UPTODATE)
900700004  1             6             SERVING      []              101000100704(SERVING-UPTODATE)  101000300704(SERVING-UPTODATE)  101000200704(SERVING-UPTODATE)
900700005  1             6             SERVING      []              101000100705(SERVING-UPTODATE)  101000300705(SERVING-UPTODATE)  101000200705(SERVING-UPTODATE)
900700006  1             6             SERVING      []              101000100706(SERVING-UPTODATE)  101000300706(SERVING-UPTODATE)  101000200706(SERVING-UPTODATE)
900800001  1             6             SERVING      []              101000100801(SERVING-UPTODATE)  101000300801(SERVING-UPTODATE)  101000200801(SERVING-UPTODATE)
900800002  1             6             SERVING      []              101000100802(SERVING-UPTODATE)  101000300802(SERVING-UPTODATE)  101000200802(SERVING-UPTODATE)
900800003  1             6             SERVING      []              101000100803(SERVING-UPTODATE)  101000300803(SERVING-UPTODATE)  101000200803(SERVING-UPTODATE)
900800004  1             6             SERVING      []              101000100804(SERVING-UPTODATE)  101000300804(SERVING-UPTODATE)  101000200804(SERVING-UPTODATE)
900800005  1             6             SERVING      []              101000100805(SERVING-UPTODATE)  101000300805(SERVING-UPTODATE)  101000200805(SERVING-UPTODATE)
900800006  1             6             SERVING      []              101000100806(SERVING-UPTODATE)  101000300806(SERVING-UPTODATE)  101000200806(SERVING-UPTODATE)

root@3fs:/var/log/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["IPoIB://10.252.0.100:8000"]' "list-nodes"
Id     Type     Status               Hostname  Pid    Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1      MGMTD    PRIMARY_MGMTD        3fs       13590  []    N/A                  1(UPTODATE)    250228-dev-1-999999-3a30c53b
100    META     HEARTBEAT_CONNECTED  3fs       13295  []    2025-03-19 09:52:56  2(UPTODATE)    250228-dev-1-999999-3a30c53b
10001  STORAGE  HEARTBEAT_CONNECTED  3fs       18095  []    2025-03-19 09:52:59  0(UPTODATE)    250228-dev-1-999999-3a30c53b
10002  STORAGE  HEARTBEAT_CONNECTED  3fs       18032  []    2025-03-19 09:53:02  0(UPTODATE)    250228-dev-1-999999-3a30c53b
10003  STORAGE  HEARTBEAT_CONNECTED  3fs       18033  []    2025-03-19 09:53:02  0(UPTODATE)    250228-dev-1-999999-3a30c53b
Originally created by @cloudp1216 on GitHub (Mar 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/193 ```shell root@3fs:/3fs/stage# cp /etc/fstab . cp: failed to close './fstab': Input/output error ``` ```shell root@3fs:/var/log/3fs# tail -f hf3fs_fuse_main-err.log [2025-03-19T09:46:44.854210120+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 1s 600ms}, retry true [2025-03-19T09:46:53.125981138+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 3s 200ms}, retry true [2025-03-19T09:47:04.593119903+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 6s 400ms}, retry true [2025-03-19T09:47:19.690132036+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:47:34.793061291+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:47:49.896841950+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:48:04.992859500+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:48:20.093259122+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:48:35.172193761+08:00 CliBG0:19420 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:48:50.227867018+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true [2025-03-19T09:49:05.293091134+08:00 CliBG1:19421 MetaClient.cc:234 ERROR] MetaClient failed to run close task {{"user":"(uid 0, gid 0, groups ())","client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"forward":0,"uuid":"00000000-0000-0000-0000-000000000000","inode":"0x000000000001f800","session":{"client":{"uuid":"e1537c48-038b-4f96-a16b-d8e2c9d8d02e","hostname":"3fs"},"session":"fa0e430f-cee3-435f-977a-72f3fce30c13"},"updateLength":true,"atime":null,"mtime":"2025-03-19 09:45:13","lengthHint":null}, backoff 10s}, retry true ``` ```shell root@3fs:/var/log/3fs# tail -f hf3fs_meta_main-err.log [2025-03-19T09:48:35.130991880+08:00 SvrProc1:13349 StorageClientImpl.cc:1173 ERROR] Give up retrying 2/16 ops 0x7febc6e493e0 after #2 retries, elapsed time: 5004ms, max retry time: 5000ms, user: Uid(0)/Gid(0), usercall: #657 [2025-03-19T09:48:35.131095346+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] #1/16 Failed queryLastChunk operation 0x7febc6ff4e00 with id RequestId(0), routing target: ChainId(900600002)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #657, status: StorageClient::NotAvailable(7005) [2025-03-19T09:48:35.131132985+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] #2/16 Failed queryLastChunk operation 0x7febc6ff4ed8 with id RequestId(0), routing target: ChainId(900600001)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #657, status: StorageClient::NotAvailable(7005) [2025-03-19T09:48:35.131158815+08:00 SvrProc1:13349 StorageClientImpl.cc:2056 ERROR] Finally collected 2/16 failed queryLastChunk operations 0x7febc6e493e0 issued by user Uid(0)/Gid(0), usercall: #657 [2025-03-19T09:48:35.131214497+08:00 SvrProc1:13349 FileOperation.cc:137 ERROR] Failed to quertLastChunk from chain ChainId(900600002), err StorageClient::NotAvailable(7005) [2025-03-19T09:48:50.183682702+08:00 SvrProc0:13348 StorageClientImpl.cc:1173 ERROR] Give up retrying 2/16 ops 0x7feb9991afe0 after #2 retries, elapsed time: 5002ms, max retry time: 5000ms, user: Uid(0)/Gid(0), usercall: #667 [2025-03-19T09:48:50.183772583+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] #1/16 Failed queryLastChunk operation 0x7feb9990e000 with id RequestId(0), routing target: ChainId(900600002)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #667, status: StorageClient::NotAvailable(7005) [2025-03-19T09:48:50.183832383+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] #2/16 Failed queryLastChunk operation 0x7feb9990e0d8 with id RequestId(0), routing target: ChainId(900600001)@ChainVersion(9)@RoutingInfoVersion(23):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-F8000000-00000000), user: Uid(0)/Gid(0), usercall: #667, status: StorageClient::NotAvailable(7005) [2025-03-19T09:48:50.183857453+08:00 SvrProc0:13348 StorageClientImpl.cc:2056 ERROR] Finally collected 2/16 failed queryLastChunk operations 0x7feb9991afe0 issued by user Uid(0)/Gid(0), usercall: #667 [2025-03-19T09:48:50.183911273+08:00 SvrProc0:13348 FileOperation.cc:137 ERROR] Failed to quertLastChunk from chain ChainId(900600002), err StorageClient::NotAvailable(7005) ``` ```shell root@3fs:/var/log/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["IPoIB://10.252.0.100:8000"]' "list-chains" ChainId ReferencedBy ChainVersion Status PreferredOrder Target Target Target 900100001 1 6 SERVING [] 101000100101(SERVING-UPTODATE) 101000300101(SERVING-UPTODATE) 101000200101(SERVING-UPTODATE) 900100002 1 6 SERVING [] 101000100102(SERVING-UPTODATE) 101000300102(SERVING-UPTODATE) 101000200102(SERVING-UPTODATE) 900100003 1 6 SERVING [] 101000100103(SERVING-UPTODATE) 101000300103(SERVING-UPTODATE) 101000200103(SERVING-UPTODATE) 900100004 1 6 SERVING [] 101000100104(SERVING-UPTODATE) 101000300104(SERVING-UPTODATE) 101000200104(SERVING-UPTODATE) 900100005 1 6 SERVING [] 101000100105(SERVING-UPTODATE) 101000300105(SERVING-UPTODATE) 101000200105(SERVING-UPTODATE) 900100006 1 6 SERVING [] 101000100106(SERVING-UPTODATE) 101000300106(SERVING-UPTODATE) 101000200106(SERVING-UPTODATE) 900200001 1 6 SERVING [] 101000100201(SERVING-UPTODATE) 101000300201(SERVING-UPTODATE) 101000200201(SERVING-UPTODATE) 900200002 1 6 SERVING [] 101000100202(SERVING-UPTODATE) 101000300202(SERVING-UPTODATE) 101000200202(SERVING-UPTODATE) 900200003 1 6 SERVING [] 101000100203(SERVING-UPTODATE) 101000300203(SERVING-UPTODATE) 101000200203(SERVING-UPTODATE) 900200004 1 6 SERVING [] 101000100204(SERVING-UPTODATE) 101000300204(SERVING-UPTODATE) 101000200204(SERVING-UPTODATE) 900200005 1 6 SERVING [] 101000100205(SERVING-UPTODATE) 101000300205(SERVING-UPTODATE) 101000200205(SERVING-UPTODATE) 900200006 1 6 SERVING [] 101000100206(SERVING-UPTODATE) 101000300206(SERVING-UPTODATE) 101000200206(SERVING-UPTODATE) 900300001 1 6 SERVING [] 101000100301(SERVING-UPTODATE) 101000300301(SERVING-UPTODATE) 101000200301(SERVING-UPTODATE) 900300002 1 6 SERVING [] 101000100302(SERVING-UPTODATE) 101000300302(SERVING-UPTODATE) 101000200302(SERVING-UPTODATE) 900300003 1 6 SERVING [] 101000100303(SERVING-UPTODATE) 101000300303(SERVING-UPTODATE) 101000200303(SERVING-UPTODATE) 900300004 1 6 SERVING [] 101000100304(SERVING-UPTODATE) 101000300304(SERVING-UPTODATE) 101000200304(SERVING-UPTODATE) 900300005 1 6 SERVING [] 101000100305(SERVING-UPTODATE) 101000300305(SERVING-UPTODATE) 101000200305(SERVING-UPTODATE) 900300006 1 6 SERVING [] 101000100306(SERVING-UPTODATE) 101000300306(SERVING-UPTODATE) 101000200306(SERVING-UPTODATE) 900400001 1 6 SERVING [] 101000100401(SERVING-UPTODATE) 101000300401(SERVING-UPTODATE) 101000200401(SERVING-UPTODATE) 900400002 1 6 SERVING [] 101000100402(SERVING-UPTODATE) 101000300402(SERVING-UPTODATE) 101000200402(SERVING-UPTODATE) 900400003 1 6 SERVING [] 101000100403(SERVING-UPTODATE) 101000300403(SERVING-UPTODATE) 101000200403(SERVING-UPTODATE) 900400004 1 6 SERVING [] 101000100404(SERVING-UPTODATE) 101000300404(SERVING-UPTODATE) 101000200404(SERVING-UPTODATE) 900400005 1 6 SERVING [] 101000100405(SERVING-UPTODATE) 101000300405(SERVING-UPTODATE) 101000200405(SERVING-UPTODATE) 900400006 1 6 SERVING [] 101000100406(SERVING-UPTODATE) 101000300406(SERVING-UPTODATE) 101000200406(SERVING-UPTODATE) 900500001 1 6 SERVING [] 101000100501(SERVING-UPTODATE) 101000300501(SERVING-UPTODATE) 101000200501(SERVING-UPTODATE) 900500002 1 6 SERVING [] 101000100502(SERVING-UPTODATE) 101000300502(SERVING-UPTODATE) 101000200502(SERVING-UPTODATE) 900500003 1 6 SERVING [] 101000100503(SERVING-UPTODATE) 101000300503(SERVING-UPTODATE) 101000200503(SERVING-UPTODATE) 900500004 1 6 SERVING [] 101000100504(SERVING-UPTODATE) 101000300504(SERVING-UPTODATE) 101000200504(SERVING-UPTODATE) 900500005 1 6 SERVING [] 101000100505(SERVING-UPTODATE) 101000300505(SERVING-UPTODATE) 101000200505(SERVING-UPTODATE) 900500006 1 6 SERVING [] 101000100506(SERVING-UPTODATE) 101000300506(SERVING-UPTODATE) 101000200506(SERVING-UPTODATE) 900600001 1 9 UNAVAILABLE [] 101000200601(LASTSRV-OFFLINE) 101000100601(OFFLINE-OFFLINE) 101000300601(OFFLINE-OFFLINE) 900600002 1 9 UNAVAILABLE [] 101000200602(LASTSRV-OFFLINE) 101000100602(OFFLINE-OFFLINE) 101000300602(OFFLINE-OFFLINE) 900600003 1 9 UNAVAILABLE [] 101000200603(LASTSRV-OFFLINE) 101000100603(OFFLINE-OFFLINE) 101000300603(OFFLINE-OFFLINE) 900600004 1 9 UNAVAILABLE [] 101000200604(LASTSRV-OFFLINE) 101000100604(OFFLINE-OFFLINE) 101000300604(OFFLINE-OFFLINE) 900600005 1 9 UNAVAILABLE [] 101000200605(LASTSRV-OFFLINE) 101000100605(OFFLINE-OFFLINE) 101000300605(OFFLINE-OFFLINE) 900600006 1 9 UNAVAILABLE [] 101000200606(LASTSRV-OFFLINE) 101000100606(OFFLINE-OFFLINE) 101000300606(OFFLINE-OFFLINE) 900700001 1 6 SERVING [] 101000100701(SERVING-UPTODATE) 101000300701(SERVING-UPTODATE) 101000200701(SERVING-UPTODATE) 900700002 1 6 SERVING [] 101000100702(SERVING-UPTODATE) 101000300702(SERVING-UPTODATE) 101000200702(SERVING-UPTODATE) 900700003 1 6 SERVING [] 101000100703(SERVING-UPTODATE) 101000300703(SERVING-UPTODATE) 101000200703(SERVING-UPTODATE) 900700004 1 6 SERVING [] 101000100704(SERVING-UPTODATE) 101000300704(SERVING-UPTODATE) 101000200704(SERVING-UPTODATE) 900700005 1 6 SERVING [] 101000100705(SERVING-UPTODATE) 101000300705(SERVING-UPTODATE) 101000200705(SERVING-UPTODATE) 900700006 1 6 SERVING [] 101000100706(SERVING-UPTODATE) 101000300706(SERVING-UPTODATE) 101000200706(SERVING-UPTODATE) 900800001 1 6 SERVING [] 101000100801(SERVING-UPTODATE) 101000300801(SERVING-UPTODATE) 101000200801(SERVING-UPTODATE) 900800002 1 6 SERVING [] 101000100802(SERVING-UPTODATE) 101000300802(SERVING-UPTODATE) 101000200802(SERVING-UPTODATE) 900800003 1 6 SERVING [] 101000100803(SERVING-UPTODATE) 101000300803(SERVING-UPTODATE) 101000200803(SERVING-UPTODATE) 900800004 1 6 SERVING [] 101000100804(SERVING-UPTODATE) 101000300804(SERVING-UPTODATE) 101000200804(SERVING-UPTODATE) 900800005 1 6 SERVING [] 101000100805(SERVING-UPTODATE) 101000300805(SERVING-UPTODATE) 101000200805(SERVING-UPTODATE) 900800006 1 6 SERVING [] 101000100806(SERVING-UPTODATE) 101000300806(SERVING-UPTODATE) 101000200806(SERVING-UPTODATE) ``` ```shell root@3fs:/var/log/3fs# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["IPoIB://10.252.0.100:8000"]' "list-nodes" Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD 3fs 13590 [] N/A 1(UPTODATE) 250228-dev-1-999999-3a30c53b 100 META HEARTBEAT_CONNECTED 3fs 13295 [] 2025-03-19 09:52:56 2(UPTODATE) 250228-dev-1-999999-3a30c53b 10001 STORAGE HEARTBEAT_CONNECTED 3fs 18095 [] 2025-03-19 09:52:59 0(UPTODATE) 250228-dev-1-999999-3a30c53b 10002 STORAGE HEARTBEAT_CONNECTED 3fs 18032 [] 2025-03-19 09:53:02 0(UPTODATE) 250228-dev-1-999999-3a30c53b 10003 STORAGE HEARTBEAT_CONNECTED 3fs 18033 [] 2025-03-19 09:53:02 0(UPTODATE) 250228-dev-1-999999-3a30c53b ```
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@qiuxinyidian commented on GitHub (Apr 7, 2025):

老哥，你这边有什么进展吗？我这边单机搭建fio dd touch 都报Input/output error

<!-- gh-comment-id:2782550611 --> @qiuxinyidian commented on GitHub (Apr 7, 2025): 老哥，你这边有什么进展吗？我这边单机搭建fio dd touch 都报Input/output error
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@yaoeco commented on GitHub (May 24, 2025):

@qiuxinyidian 老哥你解决了吗，我碰到了一样的问题= =

<!-- gh-comment-id:2906933352 --> @yaoeco commented on GitHub (May 24, 2025): @qiuxinyidian 老哥你解决了吗，我碰到了一样的问题= =
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@qiuxinyidian commented on GitHub (Jun 3, 2025):

@qiuxinyidian 老哥你解决了吗，我碰到了一样的问题= =

done

<!-- gh-comment-id:2933567074 --> @qiuxinyidian commented on GitHub (Jun 3, 2025): > [@qiuxinyidian](https://github.com/qiuxinyidian) 老哥你解决了吗，我碰到了一样的问题= = done
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@yaoeco commented on GitHub (Jun 3, 2025):

@qiuxinyidian 老哥你解决了吗，我碰到了一样的问题= =

done

6

<!-- gh-comment-id:2933612784 --> @yaoeco commented on GitHub (Jun 3, 2025): > > [@qiuxinyidian](https://github.com/qiuxinyidian) 老哥你解决了吗，我碰到了一样的问题= = > > done 6
bsakaguchi commented 2026-10-26 09:45:32 +00:00
Author
Owner
Copy link

@xlcbingo1999 commented on GitHub (Jul 22, 2025):

@qiuxinyidian @yaoeco 想问下是如何解决这批chains的呢？

900600001  1             9             UNAVAILABLE  []              101000200601(LASTSRV-OFFLINE)   101000100601(OFFLINE-OFFLINE)   101000300601(OFFLINE-OFFLINE)
900600002  1             9             UNAVAILABLE  []              101000200602(LASTSRV-OFFLINE)   101000100602(OFFLINE-OFFLINE)   101000300602(OFFLINE-OFFLINE)
900600003  1             9             UNAVAILABLE  []              101000200603(LASTSRV-OFFLINE)   101000100603(OFFLINE-OFFLINE)   101000300603(OFFLINE-OFFLINE)
900600004  1             9             UNAVAILABLE  []              101000200604(LASTSRV-OFFLINE)   101000100604(OFFLINE-OFFLINE)   101000300604(OFFLINE-OFFLINE)
900600005  1             9             UNAVAILABLE  []              101000200605(LASTSRV-OFFLINE)   101000100605(OFFLINE-OFFLINE)   101000300605(OFFLINE-OFFLINE)
900600006  1             9             UNAVAILABLE  []              101000200606(LASTSRV-OFFLINE)   101000100606(OFFLINE-OFFLINE)   101000300606(OFFLINE-OFFLINE)
<!-- gh-comment-id:3102268102 --> @xlcbingo1999 commented on GitHub (Jul 22, 2025): @qiuxinyidian @yaoeco 想问下是如何解决这批chains的呢？ ``` 900600001 1 9 UNAVAILABLE [] 101000200601(LASTSRV-OFFLINE) 101000100601(OFFLINE-OFFLINE) 101000300601(OFFLINE-OFFLINE) 900600002 1 9 UNAVAILABLE [] 101000200602(LASTSRV-OFFLINE) 101000100602(OFFLINE-OFFLINE) 101000300602(OFFLINE-OFFLINE) 900600003 1 9 UNAVAILABLE [] 101000200603(LASTSRV-OFFLINE) 101000100603(OFFLINE-OFFLINE) 101000300603(OFFLINE-OFFLINE) 900600004 1 9 UNAVAILABLE [] 101000200604(LASTSRV-OFFLINE) 101000100604(OFFLINE-OFFLINE) 101000300604(OFFLINE-OFFLINE) 900600005 1 9 UNAVAILABLE [] 101000200605(LASTSRV-OFFLINE) 101000100605(OFFLINE-OFFLINE) 101000300605(OFFLINE-OFFLINE) 900600006 1 9 UNAVAILABLE [] 101000200606(LASTSRV-OFFLINE) 101000100606(OFFLINE-OFFLINE) 101000300606(OFFLINE-OFFLINE) ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#119
No description provided.