deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #360] Unable to switch to the second disk storage #218

New issue
Closed
opened 2026-10-26 09:47:16 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:47:16 +00:00
Owner
Copy link

Originally created by @leihuang-sketch on GitHub (Nov 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/360

Four nodes were used, with two 3.5T hard drives per node
Why can't I switch to the second hard drive after the first hard drive is full? bug？

Physical storage status of 4 nodes

# node 10001
# fdb
/dev/vda2       985G  404G  541G  43% /
# 3fs data0
/dev/nvme2n1    3.5T  3.5T   67G  99% /3fs/data0
# 3fs data
/dev/nvme3n1    3.5T   33G  3.5T   1% /3fs/data1

# node 10002
/dev/vda2       985G  380G  565G  41% /
/dev/nvme2n1    3.5T  2.7T  839G  77% /3fs/data0
/dev/nvme3n1    3.5T   33G  3.5T   1% /3fs/data1

# node 10003
/dev/vda2       985G  436G  509G  47% /
/dev/nvme2n1    3.5T  3.5T   68G  99% /3fs/data0
/dev/nvme3n1    3.5T   33G  3.5T   1% /3fs/data1

# node 10004
/dev/vda2       985G  386G  559G  41% /
/dev/nvme2n1    3.5T  2.7T  839G  77% /3fs/data0
/dev/nvme3n1    3.5T   33G  3.5T   1% /3fs/data1

According to the physical hard drive status, it was found that there were two disks that were indeed full, but the system did not switch to the second disk for writing

Virtual directory status

hf3fs.stage      28T   13T   16T  44% /3fs/stage

fuse log

[2025-11-21T08:06:10.156056458+00:00 CliBG5:  122 StorageClientImpl.cc:1292 INFO] Waiting 29642ms before #3 retrying 26/32 remaining ops 0x7f71e84a2a30, user: Uid(0)/Gid(0), usercall: #449710
[2025-11-21T08:06:15.959308633+00:00 CliBG5:  122 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.324] start execution. from:
[2025-11-21T08:06:15.959540229+00:00 CliBG5:  122 MgmtdClient.cc:191 INFO] MgmtdClient: RefreshRoutingInfo finished
[2025-11-21T08:06:15.959547545+00:00 CliBG5:  122 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.324] succeeded. latency: 0ns
[2025-11-21T08:06:15.963704357+00:00 CliBG5:  122 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.325] start execution. from:
[2025-11-21T08:06:15.963982661+00:00 CliBG5:  122 MgmtdClient.cc:191 INFO] MgmtdClient: ExtendClientSession finished
[2025-11-21T08:06:15.963986306+00:00 CliBG5:  122 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.325] succeeded. latency: 0ns
[2025-11-21T08:06:25.959103677+00:00 CliBG0:  117 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.326] start execution. from:
[2025-11-21T08:06:25.959357838+00:00 CliBG0:  117 MgmtdClient.cc:191 INFO] MgmtdClient: RefreshRoutingInfo finished
[2025-11-21T08:06:25.959368754+00:00 CliBG0:  117 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.326] succeeded. latency: 27083day 15h 35min 25s 639ms 434us 360ns
[2025-11-21T08:06:25.967631856+00:00 CliBG0:  117 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.327] start execution. from:
[2025-11-21T08:06:25.967828214+00:00 CliBG0:  117 MgmtdClient.cc:191 INFO] MgmtdClient: ExtendClientSession finished
[2025-11-21T08:06:25.967833499+00:00 CliBG0:  117 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.327] succeeded. latency: 27083day 15h 35min 23s 937ms 456us 134ns
[2025-11-21T08:06:30.022998094+00:00 CliBG3:  120 StorageClientImpl.cc:1173 ERROR] Give up retrying 8/11 ops 0x7f71f130eb20 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449685
[2025-11-21T08:06:30.023001629+00:00 CliBG2:  119 StorageClientImpl.cc:1173 ERROR] Give up retrying 28/32 ops 0x7f71fd573f30 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449683
[2025-11-21T08:06:30.022993719+00:00 CliBG0:  117 StorageClientImpl.cc:1173 ERROR] Give up retrying 7/10 ops 0x7f71f44a46a0 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449684
[2025-11-21T08:06:30.022993527+00:00 CliBG5:  122 StorageClientImpl.cc:1173 ERROR] Give up retrying 3/5 ops 0x7f720c8e1f70 after #3 retries, elapsed time: 60004ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449682
[2025-11-21T08:06:30.023051985+00:00 CliBG2:  119 StorageClientImpl.cc:1794 ERROR] #1/32 Failed batchWrite operation 0x7f71fc464810 with id RequestId(7792569), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608957), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::NoSpace(7021) NoSpace
[2025-11-21T08:06:30.023058689+00:00 CliBG5:  122 StorageClientImpl.cc:1794 ERROR] #2/5 Failed batchWrite operation 0x7f720d550990 with id RequestId(7792565), routing target: ChainId(900100013)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200106)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C65), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::ResourceBusy(7012)
[2025-11-21T08:06:30.023065214+00:00 CliBG2:  119 StorageClientImpl.cc:1794 ERROR] #2/32 Failed batchWrite operation 0x7f71fc4648c0 with id RequestId(7792570), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608958), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::ResourceBusy(7012)
[2025-11-21T08:06:30.023069558+00:00 CliBG5:  122 StorageClientImpl.cc:1794 ERROR] #3/5 Failed batchWrite operation 0x7f720d550a40 with id RequestId(7792566), routing target: ChainId(900100008)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200103)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C66), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023073592+00:00 CliBG0:  117 StorageClientImpl.cc:1794 ERROR] #2/10 Failed batchWrite operation 0x7f71f44bf6b0 with id RequestId(7792602), routing target: ChainId(900100006)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100104)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A00), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::ResourceBusy(7012)
[2025-11-21T08:06:30.023085113+00:00 CliBG0:  117 StorageClientImpl.cc:1794 ERROR] #3/10 Failed batchWrite operation 0x7f71f44bf760 with id RequestId(7792603), routing target: ChainId(900100012)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100106)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A01), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023087777+00:00 CliBG2:  119 StorageClientImpl.cc:1794 ERROR] #3/32 Failed batchWrite operation 0x7f71fc464970 with id RequestId(7792571), routing target: ChainId(900100015)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100108)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608959), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023094387+00:00 CliBG3:  120 StorageClientImpl.cc:1794 ERROR] #1/11 Failed batchWrite operation 0x7f71f1349250 with id RequestId(7792611), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A07), user: Uid(0)/Gid(0), usercall: #449685, status: StorageClient::NoSpace(7021) NoSpace
[2025-11-21T08:06:30.023104727+00:00 CliBG0:  117 StorageClientImpl.cc:1794 ERROR] #4/10 Failed batchWrite operation 0x7f71f44bf810 with id RequestId(7792604), routing target: ChainId(900100009)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300106)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A02), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::NoSpace(7021) NoSpace
[2025-11-21T08:06:30.023114346+00:00 CliBG0:  117 StorageClientImpl.cc:1794 ERROR] #5/10 Failed batchWrite operation 0x7f71f44bf8c0 with id RequestId(7792605), routing target: ChainId(900100002)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100101)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A03), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023116645+00:00 CliBG5:  122 StorageClientImpl.cc:1794 ERROR] #4/5 Failed batchWrite operation 0x7f720d550af0 with id RequestId(7792567), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C67), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::NoSpace(7021) NoSpace
[2025-11-21T08:06:30.023117125+00:00 CliBG2:  119 StorageClientImpl.cc:1794 ERROR] #4/32 Failed batchWrite operation 0x7f71fc464a20 with id RequestId(7792572), routing target: ChainId(900100014)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100107)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-0060895A), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023123738+00:00 CliBG3:  120 StorageClientImpl.cc:1794 ERROR] #2/11 Failed batchWrite operation 0x7f71f1349300 with id RequestId(7792612), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A08), user: Uid(0)/Gid(0), usercall: #449685, status: StorageClient::ResourceBusy(7012)
[2025-11-21T08:06:30.023125532+00:00 CliBG5:  122 StorageClientImpl.cc:1794 ERROR] #5/5 Failed batchWrite operation 0x7f720d550ba0 with id RequestId(7792568), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C68), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::ResourceBusy(7012)
[2025-11-21T08:06:30.023126550+00:00 CliBG0:  117 StorageClientImpl.cc:1794 ERROR] #6/10 Failed batchWrite operation 0x7f71f44bf970 with id RequestId(7792606), routing target: ChainId(900100002)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100101)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A03), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008)
[2025-11-21T08:06:30.023129829+00:00 CliBG2:  119 StorageClientImpl.cc:1794 ERROR] #5/32 Failed batchWrite operation 0x7f71fc464ad0 with id RequestId(7792573), routing target: ChainId(900100011)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200104)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-0060895B), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::ResourceBusy(7012)

storage log

[2025-11-21T08:06:10.857294434+00:00 UpdatePool6:  218 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B6D"},"rdmabuf":{"addr":94803882991616,"size":1048576,"rkeys":[{"rkey":2238976,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":448327086},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792698,"channel":{"id":936,"seqnum":7840314}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:10.857306169+00:00 UpdatePool5:  217 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":90112,"length":958464,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100006,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B70"},"rdmabuf":{"addr":94804130455552,"size":958464,"rkeys":[{"rkey":2246400,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3028245883},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792702,"channel":{"id":436,"seqnum":7840318}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:10.857327121+00:00 UpdatePool6:  218 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B6D"},"rdmabuf":{"addr":140108645851136,"size":1048576,"rkeys":[{"rkey":2401280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":448327086},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792698,"channel":{"id":936,"seqnum":7840314}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21100ms
[2025-11-21T08:06:10.857475213+00:00 UpdatePool5:  217 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":90112,"length":958464,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100006,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B70"},"rdmabuf":{"addr":140108649086976,"size":958464,"rkeys":[{"rkey":2401280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3028245883},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792702,"channel":{"id":436,"seqnum":7840318}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21100ms
[2025-11-21T08:06:10.981177910+00:00 UpdatePool4:  216 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DBD"},"rdmabuf":{"addr":94804080123904,"size":1048576,"rkeys":[{"rkey":2245120,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":454467712},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793169,"channel":{"id":99,"seqnum":7840783}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:10.981179960+00:00 UpdatePool7:  219 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C7D"},"rdmabuf":{"addr":94804075929600,"size":1048576,"rkeys":[{"rkey":2245120,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1074616378},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792652,"channel":{"id":440,"seqnum":7840266}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:10.981204895+00:00 UpdatePool4:  216 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DBD"},"rdmabuf":{"addr":140107204042752,"size":1048576,"rkeys":[{"rkey":2413824,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":454467712},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793169,"channel":{"id":99,"seqnum":7840783}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21057ms
[2025-11-21T08:06:10.981234050+00:00 UpdatePool7:  219 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C7D"},"rdmabuf":{"addr":140107995750400,"size":1048576,"rkeys":[{"rkey":2407168,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1074616378},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792652,"channel":{"id":440,"seqnum":7840266}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21081ms
[2025-11-21T08:06:11.026092646+00:00 UpdatePool7:  219 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608ABD"},"rdmabuf":{"addr":94804105289728,"size":1048576,"rkeys":[{"rkey":2245888,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1181320652},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792772,"channel":{"id":701,"seqnum":7840393}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:11.026128584+00:00 UpdatePool7:  219 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608ABD"},"rdmabuf":{"addr":140109060833280,"size":1048576,"rkeys":[{"rkey":2396928,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1181320652},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792772,"channel":{"id":701,"seqnum":7840393}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21112ms
[2025-11-21T08:06:11.084551903+00:00 UpdatePool1:  213 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DDD"},"rdmabuf":{"addr":94803954294784,"size":1048576,"rkeys":[{"rkey":2241280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3486255691},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793208,"channel":{"id":896,"seqnum":7840823}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
[2025-11-21T08:06:11.084585836+00:00 UpdatePool1:  213 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DDD"},"rdmabuf":{"addr":140107237597184,"size":1048576,"rkeys":[{"rkey":2413824,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3486255691},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793208,"channel":{"id":896,"seqnum":7840823}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21102ms
[2025-11-21T08:06:11.184924274+00:00 UpdatePool5:  217 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100007,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C88"},"rdmabuf":{"addr":94804155621376,"size":1048576,"rkeys":[{"rkey":2247424,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3282862733},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792788,"channel":{"id":860,"seqnum":7840438}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}
Originally created by @leihuang-sketch on GitHub (Nov 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/360 Four nodes were used, with two 3.5T hard drives per node Why can't I switch to the second hard drive after the first hard drive is full? bug？ Physical storage status of 4 nodes ``` # node 10001 # fdb /dev/vda2 985G 404G 541G 43% / # 3fs data0 /dev/nvme2n1 3.5T 3.5T 67G 99% /3fs/data0 # 3fs data /dev/nvme3n1 3.5T 33G 3.5T 1% /3fs/data1 # node 10002 /dev/vda2 985G 380G 565G 41% / /dev/nvme2n1 3.5T 2.7T 839G 77% /3fs/data0 /dev/nvme3n1 3.5T 33G 3.5T 1% /3fs/data1 # node 10003 /dev/vda2 985G 436G 509G 47% / /dev/nvme2n1 3.5T 3.5T 68G 99% /3fs/data0 /dev/nvme3n1 3.5T 33G 3.5T 1% /3fs/data1 # node 10004 /dev/vda2 985G 386G 559G 41% / /dev/nvme2n1 3.5T 2.7T 839G 77% /3fs/data0 /dev/nvme3n1 3.5T 33G 3.5T 1% /3fs/data1 ``` According to the physical hard drive status, it was found that there were two disks that were indeed full, but the system did not switch to the second disk for writing Virtual directory status ``` hf3fs.stage 28T 13T 16T 44% /3fs/stage ``` fuse log ``` [2025-11-21T08:06:10.156056458+00:00 CliBG5: 122 StorageClientImpl.cc:1292 INFO] Waiting 29642ms before #3 retrying 26/32 remaining ops 0x7f71e84a2a30, user: Uid(0)/Gid(0), usercall: #449710 [2025-11-21T08:06:15.959308633+00:00 CliBG5: 122 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.324] start execution. from: [2025-11-21T08:06:15.959540229+00:00 CliBG5: 122 MgmtdClient.cc:191 INFO] MgmtdClient: RefreshRoutingInfo finished [2025-11-21T08:06:15.959547545+00:00 CliBG5: 122 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.324] succeeded. latency: 0ns [2025-11-21T08:06:15.963704357+00:00 CliBG5: 122 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.325] start execution. from: [2025-11-21T08:06:15.963982661+00:00 CliBG5: 122 MgmtdClient.cc:191 INFO] MgmtdClient: ExtendClientSession finished [2025-11-21T08:06:15.963986306+00:00 CliBG5: 122 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.325] succeeded. latency: 0ns [2025-11-21T08:06:25.959103677+00:00 CliBG0: 117 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.326] start execution. from: [2025-11-21T08:06:25.959357838+00:00 CliBG0: 117 MgmtdClient.cc:191 INFO] MgmtdClient: RefreshRoutingInfo finished [2025-11-21T08:06:25.959368754+00:00 CliBG0: 117 MgmtdClient.cc:398 INFO] [MgmtdClientOp RefreshRoutingInfo force=false No.326] succeeded. latency: 27083day 15h 35min 25s 639ms 434us 360ns [2025-11-21T08:06:25.967631856+00:00 CliBG0: 117 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.327] start execution. from: [2025-11-21T08:06:25.967828214+00:00 CliBG0: 117 MgmtdClient.cc:191 INFO] MgmtdClient: ExtendClientSession finished [2025-11-21T08:06:25.967833499+00:00 CliBG0: 117 MgmtdClient.cc:211 INFO] [MgmtdClientOp ExtendClientSession No.327] succeeded. latency: 27083day 15h 35min 23s 937ms 456us 134ns [2025-11-21T08:06:30.022998094+00:00 CliBG3: 120 StorageClientImpl.cc:1173 ERROR] Give up retrying 8/11 ops 0x7f71f130eb20 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449685 [2025-11-21T08:06:30.023001629+00:00 CliBG2: 119 StorageClientImpl.cc:1173 ERROR] Give up retrying 28/32 ops 0x7f71fd573f30 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449683 [2025-11-21T08:06:30.022993719+00:00 CliBG0: 117 StorageClientImpl.cc:1173 ERROR] Give up retrying 7/10 ops 0x7f71f44a46a0 after #3 retries, elapsed time: 60002ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449684 [2025-11-21T08:06:30.022993527+00:00 CliBG5: 122 StorageClientImpl.cc:1173 ERROR] Give up retrying 3/5 ops 0x7f720c8e1f70 after #3 retries, elapsed time: 60004ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #449682 [2025-11-21T08:06:30.023051985+00:00 CliBG2: 119 StorageClientImpl.cc:1794 ERROR] #1/32 Failed batchWrite operation 0x7f71fc464810 with id RequestId(7792569), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608957), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::NoSpace(7021) NoSpace [2025-11-21T08:06:30.023058689+00:00 CliBG5: 122 StorageClientImpl.cc:1794 ERROR] #2/5 Failed batchWrite operation 0x7f720d550990 with id RequestId(7792565), routing target: ChainId(900100013)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200106)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C65), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::ResourceBusy(7012) [2025-11-21T08:06:30.023065214+00:00 CliBG2: 119 StorageClientImpl.cc:1794 ERROR] #2/32 Failed batchWrite operation 0x7f71fc4648c0 with id RequestId(7792570), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608958), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::ResourceBusy(7012) [2025-11-21T08:06:30.023069558+00:00 CliBG5: 122 StorageClientImpl.cc:1794 ERROR] #3/5 Failed batchWrite operation 0x7f720d550a40 with id RequestId(7792566), routing target: ChainId(900100008)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200103)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C66), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023073592+00:00 CliBG0: 117 StorageClientImpl.cc:1794 ERROR] #2/10 Failed batchWrite operation 0x7f71f44bf6b0 with id RequestId(7792602), routing target: ChainId(900100006)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100104)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A00), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::ResourceBusy(7012) [2025-11-21T08:06:30.023085113+00:00 CliBG0: 117 StorageClientImpl.cc:1794 ERROR] #3/10 Failed batchWrite operation 0x7f71f44bf760 with id RequestId(7792603), routing target: ChainId(900100012)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100106)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A01), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023087777+00:00 CliBG2: 119 StorageClientImpl.cc:1794 ERROR] #3/32 Failed batchWrite operation 0x7f71fc464970 with id RequestId(7792571), routing target: ChainId(900100015)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100108)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608959), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023094387+00:00 CliBG3: 120 StorageClientImpl.cc:1794 ERROR] #1/11 Failed batchWrite operation 0x7f71f1349250 with id RequestId(7792611), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A07), user: Uid(0)/Gid(0), usercall: #449685, status: StorageClient::NoSpace(7021) NoSpace [2025-11-21T08:06:30.023104727+00:00 CliBG0: 117 StorageClientImpl.cc:1794 ERROR] #4/10 Failed batchWrite operation 0x7f71f44bf810 with id RequestId(7792604), routing target: ChainId(900100009)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300106)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A02), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::NoSpace(7021) NoSpace [2025-11-21T08:06:30.023114346+00:00 CliBG0: 117 StorageClientImpl.cc:1794 ERROR] #5/10 Failed batchWrite operation 0x7f71f44bf8c0 with id RequestId(7792605), routing target: ChainId(900100002)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100101)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A03), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023116645+00:00 CliBG5: 122 StorageClientImpl.cc:1794 ERROR] #4/5 Failed batchWrite operation 0x7f720d550af0 with id RequestId(7792567), routing target: ChainId(900100001)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000300101)@NodeId(10003):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C67), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::NoSpace(7021) NoSpace [2025-11-21T08:06:30.023117125+00:00 CliBG2: 119 StorageClientImpl.cc:1794 ERROR] #4/32 Failed batchWrite operation 0x7f71fc464a20 with id RequestId(7792572), routing target: ChainId(900100014)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100107)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-0060895A), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023123738+00:00 CliBG3: 120 StorageClientImpl.cc:1794 ERROR] #2/11 Failed batchWrite operation 0x7f71f1349300 with id RequestId(7792612), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A08), user: Uid(0)/Gid(0), usercall: #449685, status: StorageClient::ResourceBusy(7012) [2025-11-21T08:06:30.023125532+00:00 CliBG5: 122 StorageClientImpl.cc:1794 ERROR] #5/5 Failed batchWrite operation 0x7f720d550ba0 with id RequestId(7792568), routing target: ChainId(900100007)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100105)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608C68), user: Uid(0)/Gid(0), usercall: #449682, status: StorageClient::ResourceBusy(7012) [2025-11-21T08:06:30.023126550+00:00 CliBG0: 117 StorageClientImpl.cc:1794 ERROR] #6/10 Failed batchWrite operation 0x7f71f44bf970 with id RequestId(7792606), routing target: ChainId(900100002)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000100101)@NodeId(10001):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-00608A03), user: Uid(0)/Gid(0), usercall: #449684, status: StorageClient::Timeout(7008) [2025-11-21T08:06:30.023129829+00:00 CliBG2: 119 StorageClientImpl.cc:1794 ERROR] #5/32 Failed batchWrite operation 0x7f71fc464ad0 with id RequestId(7792573), routing target: ChainId(900100011)@ChainVersion(1)@RoutingInfoVersion(15):TargetId(101000200104)@NodeId(10002):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000001-A8000000-0060895B), user: Uid(0)/Gid(0), usercall: #449683, status: StorageClient::ResourceBusy(7012) ``` storage log ``` [2025-11-21T08:06:10.857294434+00:00 UpdatePool6: 218 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B6D"},"rdmabuf":{"addr":94803882991616,"size":1048576,"rkeys":[{"rkey":2238976,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":448327086},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792698,"channel":{"id":936,"seqnum":7840314}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:10.857306169+00:00 UpdatePool5: 217 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":90112,"length":958464,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100006,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B70"},"rdmabuf":{"addr":94804130455552,"size":958464,"rkeys":[{"rkey":2246400,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3028245883},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792702,"channel":{"id":436,"seqnum":7840318}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:10.857327121+00:00 UpdatePool6: 218 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B6D"},"rdmabuf":{"addr":140108645851136,"size":1048576,"rkeys":[{"rkey":2401280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":448327086},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792698,"channel":{"id":936,"seqnum":7840314}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21100ms [2025-11-21T08:06:10.857475213+00:00 UpdatePool5: 217 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":90112,"length":958464,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100006,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608B70"},"rdmabuf":{"addr":140108649086976,"size":958464,"rkeys":[{"rkey":2401280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3028245883},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792702,"channel":{"id":436,"seqnum":7840318}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21100ms [2025-11-21T08:06:10.981177910+00:00 UpdatePool4: 216 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DBD"},"rdmabuf":{"addr":94804080123904,"size":1048576,"rkeys":[{"rkey":2245120,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":454467712},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793169,"channel":{"id":99,"seqnum":7840783}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:10.981179960+00:00 UpdatePool7: 219 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C7D"},"rdmabuf":{"addr":94804075929600,"size":1048576,"rkeys":[{"rkey":2245120,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1074616378},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792652,"channel":{"id":440,"seqnum":7840266}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:10.981204895+00:00 UpdatePool4: 216 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DBD"},"rdmabuf":{"addr":140107204042752,"size":1048576,"rkeys":[{"rkey":2413824,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":454467712},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793169,"channel":{"id":99,"seqnum":7840783}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21057ms [2025-11-21T08:06:10.981234050+00:00 UpdatePool7: 219 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C7D"},"rdmabuf":{"addr":140107995750400,"size":1048576,"rkeys":[{"rkey":2407168,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1074616378},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792652,"channel":{"id":440,"seqnum":7840266}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21081ms [2025-11-21T08:06:11.026092646+00:00 UpdatePool7: 219 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608ABD"},"rdmabuf":{"addr":94804105289728,"size":1048576,"rkeys":[{"rkey":2245888,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1181320652},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792772,"channel":{"id":701,"seqnum":7840393}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:11.026128584+00:00 UpdatePool7: 219 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608ABD"},"rdmabuf":{"addr":140109060833280,"size":1048576,"rkeys":[{"rkey":2396928,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":1181320652},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792772,"channel":{"id":701,"seqnum":7840393}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21112ms [2025-11-21T08:06:11.084551903+00:00 UpdatePool1: 213 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DDD"},"rdmabuf":{"addr":94803954294784,"size":1048576,"rkeys":[{"rkey":2241280,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3486255691},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793208,"channel":{"id":896,"seqnum":7840823}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} [2025-11-21T08:06:11.084585836+00:00 UpdatePool1: 213 ReliableForwarding.cc:87 WARNING] forwarding wait and retry, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608DDD"},"rdmabuf":{"addr":140107237597184,"size":1048576,"rkeys":[{"rkey":2413824,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3486255691},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7793208,"channel":{"id":896,"seqnum":7840823}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, error length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}, elapsed 21102ms [2025-11-21T08:06:11.184924274+00:00 UpdatePool5: 217 ReliableForwarding.cc:261 ERROR] forward failed, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100007,"chainVer":1},"chunkId":"00000000-00000001-A8000000-00608C88"},"rdmabuf":{"addr":94804155621376,"size":1048576,"rkeys":[{"rkey":2247424,"devId":0}]},"updateVer":1,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3282862733},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"19f02f08-4034-4d91-b60f-f0b940891f37","hostname":"host-10-97-65-155"},"requestId":7792788,"channel":{"id":860,"seqnum":7840438}},"retryCount":21,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{StorageClient::NoSpace(7021) NoSpace} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)} ```
bsakaguchi commented 2026-10-26 09:47:19 +00:00
Author
Owner
Copy link

@leihuang-sketch commented on GitHub (Nov 26, 2025):

According to @kangyifei suggestion, adjust stripsize=chainsize to solve the problem. Thank you very much.
The fundamental reason for this problem is that the stripsize is much smaller than the chainsize, which leads to uneven physical disk storage in high concurrency write situations. This means that only certain chains are used, and these chains fill up a certain disk while other disks still have space. This is also the essence of the CRAQ algorithm.

<!-- gh-comment-id:3581231029 --> @leihuang-sketch commented on GitHub (Nov 26, 2025): According to @kangyifei suggestion, adjust `stripsize=chainsize` to solve the problem. Thank you very much. The fundamental reason for this problem is that the `stripsize` is much smaller than the `chainsize`, which leads to uneven physical disk storage in high concurrency write situations. This means that only certain chains are used, and these chains fill up a certain disk while other disks still have space. This is also the essence of the CRAQ algorithm.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#218
No description provided.