deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #140] RPC timeout during a stressful write test #81

New issue
Open
opened 2026-10-26 09:44:44 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-26 09:44:44 +00:00
Owner
Copy link

Originally created by @201608ybs on GitHub (Mar 10, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/140

Hi, we deploy 3FS on Alibaba Cloud ECS 8th generation instance with its Elastic RDMA Interface, following this document.
The configuration of our cluster is as following:

Additionally, the meta node also serves as a FUSE client.

We start 128 fio write jobs on each FUSE client, which write to 3FS simultaneously.

test_dir="/3fs/stage/`hostname`-fio"
mkdir "/3fs/stage/`hostname`-fio"

fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=300 -name=2depth_128file_4M_direct_write_bw -directory=${test_dir}

The fio process reports the following error:

fio: io_u error on file /3fs/stage/3fs-fuse-fio/2depth_128file_4M_direct_write_bw.50.0: Input/output error: write offset=8388608, buflen=4194304
fio: pid=35612, err=5/file:io_u.c:1845, func=io_u error, error=Input/output error

We found the following errors in /var/log/3fs/hf3fs_fuse_main-err.log:

[2025-03-10T17:35:35.101867099+08:00 hf3fs_fuse_main:36660 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f583452e6d0, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101869946+08:00 hf3fs_fuse_main:36847 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f56ec109ac0, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101884872+08:00 hf3fs_fuse_main:36798 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f5758720bc0, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101893372+08:00 hf3fs_fuse_main:36806 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f572c7205d0, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101898173+08:00 hf3fs_fuse_main:36827 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f572871f370, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101886611+08:00 hf3fs_fuse_main:36794 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f576474da60, timeout: 10s, peer address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101914629+08:00 hf3fs_fuse_main:36660 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f583452e6d0, usercall: #238477, timeout: 10s, latency: 10000ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.102039496+08:00 hf3fs_fuse_main:36794 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f576474da60, usercall: #264383, timeout: 10s, latency: 3ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.101979856+08:00 hf3fs_fuse_main:36798 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f5758720bc0, usercall: #264391, timeout: 10s, latency: 0ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.102064488+08:00 hf3fs_fuse_main:36794 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #264383, write key: {"vChainId":{"chainId":900200019,"chainVer":1},"chunkId":"00000000-00000000-18400000-00000034"}
[2025-03-10T17:35:35.101993967+08:00 hf3fs_fuse_main:36806 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f572c7205d0, usercall: #252659, timeout: 10s, latency: 4414ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.102006687+08:00 hf3fs_fuse_main:36827 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f572871f370, usercall: #264378, timeout: 10s, latency: 5ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000
[2025-03-10T17:35:35.102070023+08:00 hf3fs_fuse_main:36660 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #238477, write key: {"vChainId":{"chainId":900600018,"chainVer":1},"chunkId":"00000000-00000000-19DF0000-00000043"}
[2025-03-10T17:35:35.102086198+08:00 hf3fs_fuse_main:36806 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #252659, write key: {"vChainId":{"chainId":900600018,"chainVer":1},"chunkId":"00000000-00000000-19DF0000-00000043"}
[2025-03-10T17:35:35.102082654+08:00 hf3fs_fuse_main:36798 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #264391, write key: {"vChainId":{"chainId":900300018,"chainVer":1},"chunkId":"00000000-00000000-18E10000-0000003F"}

The storage node also reports errors in /var/log/3fs/storage_main-err.log.

[2025-03-10T17:34:25.198116083+08:00 UpdatePool1:34409 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565280c29188, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198116209+08:00 UpdatePool0:34408 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527d7cd588, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198121261+08:00 UpdatePool7:34415 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527b27b588, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198116423+08:00 UpdatePool5:34413 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565279628788, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198118587+08:00 UpdatePool3:34411 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527d7cf888, timeout: 100ms, peer address: RDMA://10.99.0.4:8000
[2025-03-10T17:34:25.198183483+08:00 UpdatePool1:34409 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900600002,"chainVer":1},"chunkId":"00000000-00000000-18B20000-0000002D"},"rdmabuf":{"addr":94903668514816,"size":1048576,"rkeys":[{"rkey":2972672,"devId":0}]},"updateVer":7,"updateType":"WRITE","checksum":{"type":"CRC32C","value":2107099607},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"7684d6be-96f3-4fa5-bad8-31d0aa3cf139","hostname":"3fs-meta"},"requestId":82377,"channel":{"id":3,"seqnum":82385}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005)
[2025-03-10T17:34:25.198115877+08:00 UpdatePool6:34414 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565280a60388, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198116402+08:00 UpdatePool2:34410 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527c010088, timeout: 100ms, peer address: RDMA://10.99.0.4:8000
[2025-03-10T17:34:25.198131506+08:00 UpdatePool4:34412 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527c116788, timeout: 100ms, peer address: RDMA://10.99.0.5:8000
[2025-03-10T17:34:25.198186195+08:00 UpdatePool0:34408 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100002,"chainVer":1},"chunkId":"00000000-00000000-183E0000-00000023"},"rdmabuf":{"addr":94903660126208,"size":1048576,"rkeys":[{"rkey":2972416,"devId":0}]},"updateVer":7,"updateType":"WRITE","checksum":{"type":"CRC32C","value":4168365105},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"5b48d7e5-368f-483e-9ab0-20ebaddf93ea","hostname":"shenkai-3fs-client-bm003"},"requestId":81717,"channel":{"id":9,"seqnum":81725}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005)

We believe that this error is related to the reliable forwarding timeout and have adjusted different timeout parameters in the storage_main.toml. Finally, we resolved this error by increasing the retry_first_timeout to 500ms (default is 100ms). So, what does the retry_first_timeout parameter indicate for 3FS?
sed -i "s/retry_first_wait = '100ms'/retry_first_wait = '500ms'/g" /opt/3fs/etc/storage_main.toml

Originally created by @201608ybs on GitHub (Mar 10, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/140 Hi, we deploy 3FS on Alibaba Cloud ECS 8th generation instance with its Elastic RDMA Interface, following [this document](https://mp.weixin.qq.com/s/Ri9-sjsaoQ3LUE2-dTAN_g). The configuration of our cluster is as following: - Meta Node: 1x [ecs.g8i.48xlarge](https://help.aliyun.com/document_detail/25378.html#g8i) - FUSE Client: 4x [ecs.ebmg8i.48xlarge](https://help.aliyun.com/document_detail/25378.html#ebmg8i) - Storage Node: 5x [ecs.i4.4xlarge](https://help.aliyun.com/document_detail/25378.html#i4) Additionally, the meta node also serves as a FUSE client. We start 128 fio write jobs on each FUSE client, which write to 3FS simultaneously. ``` test_dir="/3fs/stage/`hostname`-fio" mkdir "/3fs/stage/`hostname`-fio" fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=300 -name=2depth_128file_4M_direct_write_bw -directory=${test_dir} ``` The fio process reports the following error: ``` fio: io_u error on file /3fs/stage/3fs-fuse-fio/2depth_128file_4M_direct_write_bw.50.0: Input/output error: write offset=8388608, buflen=4194304 fio: pid=35612, err=5/file:io_u.c:1845, func=io_u error, error=Input/output error ``` We found the following errors in `/var/log/3fs/hf3fs_fuse_main-err.log`: ``` [2025-03-10T17:35:35.101867099+08:00 hf3fs_fuse_main:36660 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f583452e6d0, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101869946+08:00 hf3fs_fuse_main:36847 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f56ec109ac0, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101884872+08:00 hf3fs_fuse_main:36798 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f5758720bc0, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101893372+08:00 hf3fs_fuse_main:36806 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f572c7205d0, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101898173+08:00 hf3fs_fuse_main:36827 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f572871f370, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101886611+08:00 hf3fs_fuse_main:36794 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x7f576474da60, timeout: 10s, peer address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101914629+08:00 hf3fs_fuse_main:36660 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f583452e6d0, usercall: #238477, timeout: 10s, latency: 10000ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.102039496+08:00 hf3fs_fuse_main:36794 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f576474da60, usercall: #264383, timeout: 10s, latency: 3ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.101979856+08:00 hf3fs_fuse_main:36798 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f5758720bc0, usercall: #264391, timeout: 10s, latency: 0ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.102064488+08:00 hf3fs_fuse_main:36794 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #264383, write key: {"vChainId":{"chainId":900200019,"chainVer":1},"chunkId":"00000000-00000000-18400000-00000034"} [2025-03-10T17:35:35.101993967+08:00 hf3fs_fuse_main:36806 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f572c7205d0, usercall: #252659, timeout: 10s, latency: 4414ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.102006687+08:00 hf3fs_fuse_main:36827 StorageClientImpl.cc:876 ERROR] Call messenger error: RPC::Timeout(2005), batchWrite request: 0x7f572871f370, usercall: #264378, timeout: 10s, latency: 5ms, host: 3fs-storage002, address: RDMA://10.99.0.3:8000 [2025-03-10T17:35:35.102070023+08:00 hf3fs_fuse_main:36660 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #238477, write key: {"vChainId":{"chainId":900600018,"chainVer":1},"chunkId":"00000000-00000000-19DF0000-00000043"} [2025-03-10T17:35:35.102086198+08:00 hf3fs_fuse_main:36806 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #252659, write key: {"vChainId":{"chainId":900600018,"chainVer":1},"chunkId":"00000000-00000000-19DF0000-00000043"} [2025-03-10T17:35:35.102082654+08:00 hf3fs_fuse_main:36798 StorageClientImpl.cc:1917 ERROR] Cannot communicate with node NodeId(10002), error: RPC::Timeout(2005), usercall: #264391, write key: {"vChainId":{"chainId":900300018,"chainVer":1},"chunkId":"00000000-00000000-18E10000-0000003F"} ``` The storage node also reports errors in `/var/log/3fs/storage_main-err.log`. ``` [2025-03-10T17:34:25.198116083+08:00 UpdatePool1:34409 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565280c29188, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198116209+08:00 UpdatePool0:34408 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527d7cd588, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198121261+08:00 UpdatePool7:34415 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527b27b588, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198116423+08:00 UpdatePool5:34413 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565279628788, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198118587+08:00 UpdatePool3:34411 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527d7cf888, timeout: 100ms, peer address: RDMA://10.99.0.4:8000 [2025-03-10T17:34:25.198183483+08:00 UpdatePool1:34409 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900600002,"chainVer":1},"chunkId":"00000000-00000000-18B20000-0000002D"},"rdmabuf":{"addr":94903668514816,"size":1048576,"rkeys":[{"rkey":2972672,"devId":0}]},"updateVer":7,"updateType":"WRITE","checksum":{"type":"CRC32C","value":2107099607},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"7684d6be-96f3-4fa5-bad8-31d0aa3cf139","hostname":"3fs-meta"},"requestId":82377,"channel":{"id":3,"seqnum":82385}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005) [2025-03-10T17:34:25.198115877+08:00 UpdatePool6:34414 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x565280a60388, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198116402+08:00 UpdatePool2:34410 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527c010088, timeout: 100ms, peer address: RDMA://10.99.0.4:8000 [2025-03-10T17:34:25.198131506+08:00 UpdatePool4:34412 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x56527c116788, timeout: 100ms, peer address: RDMA://10.99.0.5:8000 [2025-03-10T17:34:25.198186195+08:00 UpdatePool0:34408 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100002,"chainVer":1},"chunkId":"00000000-00000000-183E0000-00000023"},"rdmabuf":{"addr":94903660126208,"size":1048576,"rkeys":[{"rkey":2972416,"devId":0}]},"updateVer":7,"updateType":"WRITE","checksum":{"type":"CRC32C","value":4168365105},"inlinebuf":"std::vector<uint8_t>(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"5b48d7e5-368f-483e-9ab0-20ebaddf93ea","hostname":"shenkai-3fs-client-bm003"},"requestId":81717,"channel":{"id":9,"seqnum":81725}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005) ``` We believe that this error is related to the reliable forwarding timeout and have adjusted different timeout parameters in the storage_main.toml. Finally, we resolved this error by increasing the `retry_first_timeout` to 500ms (default is 100ms). So, what does the `retry_first_timeout` parameter indicate for 3FS? `sed -i "s/retry_first_wait = '100ms'/retry_first_wait = '500ms'/g" /opt/3fs/etc/storage_main.toml`
bsakaguchi commented 2026-10-26 09:44:45 +00:00
Author
Owner
Copy link

@zartbot commented on GitHub (Mar 10, 2025):

Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.

<!-- gh-comment-id:2710534001 --> @zartbot commented on GitHub (Mar 10, 2025): Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.
bsakaguchi commented 2026-10-26 09:44:45 +00:00
Author
Owner
Copy link

@RainSnowIceRain commented on GitHub (Mar 11, 2025):

Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.

Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue?

<!-- gh-comment-id:2712274110 --> @RainSnowIceRain commented on GitHub (Mar 11, 2025): > Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings. Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue?
bsakaguchi commented 2026-10-26 09:44:45 +00:00
Author
Owner
Copy link

@201608ybs commented on GitHub (Mar 11, 2025):

Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.

Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue?

Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment:

  1. We modify the ALIGN_SIZE to 4096 in src/storage/chunk_engine/src/utils/aligned.rs as mentioned in this issue, which should be equal to your disk sector size.
  2. We modify the max_sge parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in this document.
  3. We set the ulimit.max_open_files to unlimited on the meta node.
  4. We increase the retry_first_timeout to 500ms in the storage_main.toml on all storage nodes to solve the RPC timeout problem mentioned above.
<!-- gh-comment-id:2712727172 --> @201608ybs commented on GitHub (Mar 11, 2025): > > Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings. > > Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue? Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment: 1. We modify the `ALIGN_SIZE` to 4096 in `src/storage/chunk_engine/src/utils/aligned.rs` as mentioned in [this issue](https://github.com/deepseek-ai/3FS/issues/99), which should be equal to your disk sector size. 2. We modify the `max_sge` parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in[ this document](https://mp.weixin.qq.com/s/Ri9-sjsaoQ3LUE2-dTAN_g). 3. We set the ulimit.max_open_files to unlimited on the meta node. 4. We increase the `retry_first_timeout` to 500ms in the `storage_main.toml` on all storage nodes to solve the RPC timeout problem mentioned above.
bsakaguchi commented 2026-10-26 09:44:45 +00:00
Author
Owner
Copy link

@RainSnowIceRain commented on GitHub (Mar 11, 2025):

Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.

Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue?

Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment:

  1. We modify the ALIGN_SIZE to 4096 in src/storage/chunk_engine/src/utils/aligned.rs as mentioned in this issue, which should be equal to your disk sector size.
  2. We modify the max_sge parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in this document.
  3. We set the ulimit.max_open_files to unlimited on the meta node.
  4. We increase the retry_first_timeout to 500ms in the storage_main.toml on all storage nodes to solve the RPC timeout problem mentioned above.

@201608ybs Excellent! Thank you very much! We have also referred to the deployment blog on Alibaba Cloud. We are currently encountering a performance issue:

  1. We deployed a three-node cluster, where one node serves as the client and metadata server, while the other two nodes function as storage nodes. We configured the system with a dual-replication strategy, set the chunk to 1MB, and the stripe to 16.
  2. We utilized the USRBIO FIO with the following parameters: -numjobs=128 -fallocate=none -iodepth=256 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -direct=1 -rw=read -bs=1M --group_reporting -size=500M -mountpoint=/3fs/stage.

However, neither the CPU, DRAM, NIC, nor the SSDs on the client/metadata node and storage nodes reached their maximum utilization. The 100Gbps NIC only achieved an actual throughput of approximately 40Gbps.

Is this issue attributable to suboptimal cluster deployment methods, inadequate 3FS configurations, or improper fio settings?

<!-- gh-comment-id:2712791426 --> @RainSnowIceRain commented on GitHub (Mar 11, 2025): > > > Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings. > > > > > > Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue? > > Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment: > > 1. We modify the `ALIGN_SIZE` to 4096 in `src/storage/chunk_engine/src/utils/aligned.rs` as mentioned in [this issue](https://github.com/deepseek-ai/3FS/issues/99), which should be equal to your disk sector size. > 2. We modify the `max_sge` parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in[ this document](https://mp.weixin.qq.com/s/Ri9-sjsaoQ3LUE2-dTAN_g). > 3. We set the ulimit.max_open_files to unlimited on the meta node. > 4. We increase the `retry_first_timeout` to 500ms in the `storage_main.toml` on all storage nodes to solve the RPC timeout problem mentioned above. @201608ybs Excellent! Thank you very much! We have also referred to the deployment blog on Alibaba Cloud. We are currently encountering a performance issue: 1. We deployed a three-node cluster, where one node serves as the client and metadata server, while the other two nodes function as storage nodes. We configured the system with a dual-replication strategy, set the chunk to 1MB, and the stripe to 16. 2. We utilized the USRBIO FIO with the following parameters: -numjobs=128 -fallocate=none -iodepth=256 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -direct=1 -rw=read -bs=1M --group_reporting -size=500M -mountpoint=/3fs/stage. However, neither the CPU, DRAM, NIC, nor the SSDs on the client/metadata node and storage nodes reached their maximum utilization. The 100Gbps NIC only achieved an actual throughput of approximately 40Gbps. Is this issue attributable to suboptimal cluster deployment methods, inadequate 3FS configurations, or improper fio settings?
bsakaguchi commented 2026-10-26 09:44:46 +00:00
Author
Owner
Copy link

@201608ybs commented on GitHub (Mar 11, 2025):

Excellent! Thank you very much! We have also referred to the deployment

Oh, I observe similar problem using FUSE clients. However, as the number of Fuse Clients increases, the network traffic of the storage nodes also increases. I am uncertain about the root cause of this issue now. Maybe you can consult with the 3FS developers.

<!-- gh-comment-id:2713265214 --> @201608ybs commented on GitHub (Mar 11, 2025): > Excellent! Thank you very much! We have also referred to the deployment Oh, I observe similar problem using FUSE clients. However, as the number of Fuse Clients increases, the network traffic of the storage nodes also increases. I am uncertain about the root cause of this issue now. Maybe you can consult with the 3FS developers.
bsakaguchi commented 2026-10-26 09:44:46 +00:00
Author
Owner
Copy link

@zartbot commented on GitHub (Mar 16, 2025):

Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings.

Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue?

Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment:

  1. We modify the ALIGN_SIZE to 4096 in src/storage/chunk_engine/src/utils/aligned.rs as mentioned in this issue, which should be equal to your disk sector size.
  2. We modify the max_sge parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in this document.
  3. We set the ulimit.max_open_files to unlimited on the meta node.
  4. We increase the retry_first_timeout to 500ms in the storage_main.toml on all storage nodes to solve the RPC timeout problem mentioned above.

@201608ybs Excellent! Thank you very much! We have also referred to the deployment blog on Alibaba Cloud. We are currently encountering a performance issue:

  1. We deployed a three-node cluster, where one node serves as the client and metadata server, while the other two nodes function as storage nodes. We configured the system with a dual-replication strategy, set the chunk to 1MB, and the stripe to 16.
  2. We utilized the USRBIO FIO with the following parameters: -numjobs=128 -fallocate=none -iodepth=256 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -direct=1 -rw=read -bs=1M --group_reporting -size=500M -mountpoint=/3fs/stage.

However, neither the CPU, DRAM, NIC, nor the SSDs on the client/metadata node and storage nodes reached their maximum utilization. The 100Gbps NIC only achieved an actual throughput of approximately 40Gbps.

Is this issue attributable to suboptimal cluster deployment methods, inadequate 3FS configurations, or improper fio settings?

which type of instance are you used ? 40Gbps throughput seems to be storage BW policy limited on some type of instance if you did not use local disk.

<!-- gh-comment-id:2727112737 --> @zartbot commented on GitHub (Mar 16, 2025): > > > > Additionally, during the write stress test, we discovered that the meta server encounters issues when reading the fdb.cluster file due to system ulimit restrictions. This results in the file failing to open and causing errors. These problems can be mitigated by modifying the relevant settings. > > > > > > > > > Under a concurrency load of 256 writes, the entire cluster experiences a complete stall. Could you please specify which parameters in which configuration files should be adjusted to address this issue? > > > > > > Hi, we conduct our test on Alibaba Cloud and make following modifications in our environment: > > > > 1. We modify the `ALIGN_SIZE` to 4096 in `src/storage/chunk_engine/src/utils/aligned.rs` as mentioned in [this issue](https://github.com/deepseek-ai/3FS/issues/99), which should be equal to your disk sector size. > > 2. We modify the `max_sge` parameter to 1 in all configuration files(*.toml) on all nodes as mentioned in[ this document](https://mp.weixin.qq.com/s/Ri9-sjsaoQ3LUE2-dTAN_g). > > 3. We set the ulimit.max_open_files to unlimited on the meta node. > > 4. We increase the `retry_first_timeout` to 500ms in the `storage_main.toml` on all storage nodes to solve the RPC timeout problem mentioned above. > > [@201608ybs](https://github.com/201608ybs) Excellent! Thank you very much! We have also referred to the deployment blog on Alibaba Cloud. We are currently encountering a performance issue: > > 1. We deployed a three-node cluster, where one node serves as the client and metadata server, while the other two nodes function as storage nodes. We configured the system with a dual-replication strategy, set the chunk to 1MB, and the stripe to 16. > 2. We utilized the USRBIO FIO with the following parameters: -numjobs=128 -fallocate=none -iodepth=256 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -direct=1 -rw=read -bs=1M --group_reporting -size=500M -mountpoint=/3fs/stage. > > However, neither the CPU, DRAM, NIC, nor the SSDs on the client/metadata node and storage nodes reached their maximum utilization. The 100Gbps NIC only achieved an actual throughput of approximately 40Gbps. > > Is this issue attributable to suboptimal cluster deployment methods, inadequate 3FS configurations, or improper fio settings? which type of instance are you used ? 40Gbps throughput seems to be storage BW policy limited on some type of instance if you did not use local disk.
bsakaguchi commented 2026-10-26 09:44:46 +00:00
Author
Owner
Copy link

@RainSnowIceRain commented on GitHub (Mar 16, 2025):

@zartbot The client, meta, and storage nodes are all equipped with local NVMe SSDs. In our two-storage-node cluster with a two-replica storage configuration, FIO tests revealed that the NIC bandwidth utilization on storage nodes was only 40%. Does this indicate potential latent issues in the 3FS two-replica setup?

<!-- gh-comment-id:2727190837 --> @RainSnowIceRain commented on GitHub (Mar 16, 2025): @zartbot The client, meta, and storage nodes are all equipped with local NVMe SSDs. In our two-storage-node cluster with a two-replica storage configuration, FIO tests revealed that the NIC bandwidth utilization on storage nodes was only 40%. Does this indicate potential latent issues in the 3FS two-replica setup?
bsakaguchi commented 2026-10-26 09:44:46 +00:00
Author
Owner
Copy link

@201608ybs commented on GitHub (Mar 18, 2025):

@zartbot The client, meta, and storage nodes are all equipped with local NVMe SSDs. In our two-storage-node cluster with a two-replica storage configuration, FIO tests revealed that the NIC bandwidth utilization on storage nodes was only 40%. Does this indicate potential latent issues in the 3FS two-replica setup?

Hi, have you checked the NIC bandwidth of the FUSE client node? You can refer to the official document to find out the detailed information, such as the NIC bandwidth and IOPS, for your instance type. I have tried to reproduce your problem with similar configurations (2 storage nodes with 2 replicas). The detailed configuration of my cluster is as follows:

The output of 3FS list-nodes command:

Id     Type     Status               Hostname  Pid     Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
1      MGMTD    PRIMARY_MGMTD        meta      244661  []    N/A                  1(UPTODATE)    250228-dev-1-999999-8307bce1
100    META     HEARTBEAT_CONNECTED  meta      244725  []    2025-03-18 11:04:51  1(UPTODATE)    250228-dev-1-999999-8307bce1
10001  STORAGE  HEARTBEAT_CONNECTED  storage1  352741  []    2025-03-18 11:04:52  2(UPTODATE)    250228-dev-1-999999-8307bce1
10002  STORAGE  HEARTBEAT_CONNECTED  storage2  337346  []    2025-03-18 11:04:51  2(UPTODATE)    250228-dev-1-999999-8307bce1

The output of 3FS list-targets command:

TargetId      ChainId    Role  PublicState  LocalState  NodeId  DiskIndex  UsedSize
101000100101  900100001  HEAD  SERVING      UPTODATE    10001   0          0
101000200101  900100001  TAIL  SERVING      UPTODATE    10002   0          0
101000100201  900200001  HEAD  SERVING      UPTODATE    10001   1          3762290688
101000200201  900200001  TAIL  SERVING      UPTODATE    10002   1          3762290688
101000100801  900800001  HEAD  SERVING      UPTODATE    10001   7          3736076288
101000200801  900800001  TAIL  SERVING      UPTODATE    10002   7          3736076288
101000100601  900600001  HEAD  SERVING      UPTODATE    10001   5          3751804928
101000200601  900600001  TAIL  SERVING      UPTODATE    10002   5          3751804928
101000100301  900300001  HEAD  SERVING      UPTODATE    10001   2          3762290688
101000200301  900300001  TAIL  SERVING      UPTODATE    10002   2          3762290688
101000100701  900700001  HEAD  SERVING      UPTODATE    10001   6          3745513472
101000200701  900700001  TAIL  SERVING      UPTODATE    10002   6          3745513472
101000100401  900400001  HEAD  SERVING      UPTODATE    10001   3          3711959040
101000200401  900400001  TAIL  SERVING      UPTODATE    10002   3          3711959040
101000100501  900500001  HEAD  SERVING      UPTODATE    10001   4          3743416320
101000200501  900500001  TAIL  SERVING      UPTODATE    10002   4          3743416320

The output of 3FS list-chains command:

ChainId    ReferencedBy  ChainVersion  Status   PreferredOrder  Target                          Target
900100001  1             1             SERVING  []              101000100101(SERVING-UPTODATE)  101000200101(SERVING-UPTODATE)
900200001  1             1             SERVING  []              101000100201(SERVING-UPTODATE)  101000200201(SERVING-UPTODATE)
900300001  1             1             SERVING  []              101000100301(SERVING-UPTODATE)  101000200301(SERVING-UPTODATE)
900400001  1             1             SERVING  []              101000100401(SERVING-UPTODATE)  101000200401(SERVING-UPTODATE)
900500001  1             1             SERVING  []              101000100501(SERVING-UPTODATE)  101000200501(SERVING-UPTODATE)
900600001  1             1             SERVING  []              101000100601(SERVING-UPTODATE)  101000200601(SERVING-UPTODATE)
900700001  1             1             SERVING  []              101000100701(SERVING-UPTODATE)  101000200701(SERVING-UPTODATE)
900800001  1             1             SERVING  []              101000100801(SERVING-UPTODATE)  101000200801(SERVING-UPTODATE)

In the first attempt, I got a read bandwidth of 3.8 GB/s in the FIO read test with 16 jobs. Then, I found that the bandwidth of the FUSE client node (ecs.g8i.16xlarge) is only 32 Gbps.

/3fs/stage/test/test1: (groupid=0, jobs=16): err= 0: pid=275608: Tue Mar 18 11:06:32 2025
  read: IOPS=3743, BW=3744MiB/s (3926MB/s)(129GiB/35248msec)
    clat (usec): min=810, max=21786, avg=4270.83, stdev=1441.41
     lat (usec): min=810, max=21787, avg=4270.88, stdev=1441.41
    clat percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 2040], 10.00th=[ 2343], 20.00th=[ 2835],
     | 30.00th=[ 3326], 40.00th=[ 3785], 50.00th=[ 4293], 60.00th=[ 4686],
     | 70.00th=[ 5145], 80.00th=[ 5604], 90.00th=[ 6259], 95.00th=[ 6652],
     | 99.00th=[ 7373], 99.50th=[ 7570], 99.90th=[ 8225], 99.95th=[ 8717],
     | 99.99th=[10159]
   bw (  MiB/s): min= 3460, max= 4082, per=100.00%, avg=3745.53, stdev= 7.37, samples=1120
   iops        : min= 3460, max= 4082, avg=3745.46, stdev= 7.37, samples=1120
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=4.31%, 4=39.04%, 10=56.60%, 20=0.01%, 50=0.01%
  cpu          : usr=0.12%, sys=0.21%, ctx=601563, majf=0, minf=183
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=131966,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3744MiB/s (3926MB/s), 3744MiB/s-3744MiB/s (3926MB/s-3926MB/s), io=129GiB (138GB), run=35248-35248msec

After I increased the NIC bandwidth of the FUSE client node (ecs.g8i.48xlarge), the read bandwidth increased from 3.8 GB/s to 9.9 GB/s.

Jobs: 16 (f=16): [R(16)][100.0%][r=9893MiB/s][r=9893 IOPS][eta 00m:00s]
/3fs/stage/test/test1: (groupid=0, jobs=16): err= 0: pid=16780: Tue Mar 18 11:16:06 2025
  read: IOPS=9901, BW=9902MiB/s (10.4GB/s)(580GiB/60003msec)
    clat (usec): min=547, max=13781, avg=1615.14, stdev=646.53
     lat (usec): min=547, max=13781, avg=1615.18, stdev=646.53
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  717], 10.00th=[  799], 20.00th=[ 1012],
     | 30.00th=[ 1188], 40.00th=[ 1352], 50.00th=[ 1598], 60.00th=[ 1795],
     | 70.00th=[ 1926], 80.00th=[ 2114], 90.00th=[ 2376], 95.00th=[ 2868],
     | 99.00th=[ 3359], 99.50th=[ 3490], 99.90th=[ 4113], 99.95th=[ 4686],
     | 99.99th=[ 5932]
   bw (  MiB/s): min= 9286, max=10536, per=100.00%, avg=9903.46, stdev=14.11, samples=1904
   iops        : min= 9286, max=10536, avg=9903.45, stdev=14.12, samples=1904
  lat (usec)   : 750=7.05%, 1000=12.42%
  lat (msec)   : 2=55.48%, 4=24.93%, 10=0.11%, 20=0.01%
  cpu          : usr=0.16%, sys=0.29%, ctx=1184299, majf=0, minf=160
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=594143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=9902MiB/s (10.4GB/s), 9902MiB/s-9902MiB/s (10.4GB/s-10.4GB/s), io=580GiB (623GB), run=60003-60003msec

The fio test command is as following:
/root/fio/fio -numjobs=16 -fallocate=none -iodepth=1 -ioengine=external:/root/hf3fs_usrbio.so -direct=1 -rw=read -bs=1MB --group_reporting -size=100MB -time_based -runtime=60 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -name=/3fs/stage/test/test1 -mountpoint=/3fs/stage -ior_depth=1

<!-- gh-comment-id:2731537095 --> @201608ybs commented on GitHub (Mar 18, 2025): > [@zartbot](https://github.com/zartbot) The client, meta, and storage nodes are all equipped with local NVMe SSDs. In our two-storage-node cluster with a two-replica storage configuration, FIO tests revealed that the NIC bandwidth utilization on storage nodes was only 40%. Does this indicate potential latent issues in the 3FS two-replica setup? Hi, have you checked the NIC bandwidth of the FUSE client node? You can refer to the [official document](https://help.aliyun.com/zh/ecs/user-guide/overview-of-instance-families) to find out the detailed information, such as the NIC bandwidth and IOPS, for your instance type. I have tried to reproduce your problem with similar configurations (2 storage nodes with 2 replicas). The detailed configuration of my cluster is as follows: - Meta Node: 1x [ecs.g8i.4xlarge](https://help.aliyun.com/document_detail/25378.html#g8i) - Storage Node: 2x [ecs.i4.32xlarge](https://help.aliyun.com/document_detail/25378.html#i4) The output of 3FS `list-nodes` command: ``` Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion 1 MGMTD PRIMARY_MGMTD meta 244661 [] N/A 1(UPTODATE) 250228-dev-1-999999-8307bce1 100 META HEARTBEAT_CONNECTED meta 244725 [] 2025-03-18 11:04:51 1(UPTODATE) 250228-dev-1-999999-8307bce1 10001 STORAGE HEARTBEAT_CONNECTED storage1 352741 [] 2025-03-18 11:04:52 2(UPTODATE) 250228-dev-1-999999-8307bce1 10002 STORAGE HEARTBEAT_CONNECTED storage2 337346 [] 2025-03-18 11:04:51 2(UPTODATE) 250228-dev-1-999999-8307bce1 ``` The output of 3FS `list-targets` command: ``` TargetId ChainId Role PublicState LocalState NodeId DiskIndex UsedSize 101000100101 900100001 HEAD SERVING UPTODATE 10001 0 0 101000200101 900100001 TAIL SERVING UPTODATE 10002 0 0 101000100201 900200001 HEAD SERVING UPTODATE 10001 1 3762290688 101000200201 900200001 TAIL SERVING UPTODATE 10002 1 3762290688 101000100801 900800001 HEAD SERVING UPTODATE 10001 7 3736076288 101000200801 900800001 TAIL SERVING UPTODATE 10002 7 3736076288 101000100601 900600001 HEAD SERVING UPTODATE 10001 5 3751804928 101000200601 900600001 TAIL SERVING UPTODATE 10002 5 3751804928 101000100301 900300001 HEAD SERVING UPTODATE 10001 2 3762290688 101000200301 900300001 TAIL SERVING UPTODATE 10002 2 3762290688 101000100701 900700001 HEAD SERVING UPTODATE 10001 6 3745513472 101000200701 900700001 TAIL SERVING UPTODATE 10002 6 3745513472 101000100401 900400001 HEAD SERVING UPTODATE 10001 3 3711959040 101000200401 900400001 TAIL SERVING UPTODATE 10002 3 3711959040 101000100501 900500001 HEAD SERVING UPTODATE 10001 4 3743416320 101000200501 900500001 TAIL SERVING UPTODATE 10002 4 3743416320 ``` The output of 3FS `list-chains` command: ``` ChainId ReferencedBy ChainVersion Status PreferredOrder Target Target 900100001 1 1 SERVING [] 101000100101(SERVING-UPTODATE) 101000200101(SERVING-UPTODATE) 900200001 1 1 SERVING [] 101000100201(SERVING-UPTODATE) 101000200201(SERVING-UPTODATE) 900300001 1 1 SERVING [] 101000100301(SERVING-UPTODATE) 101000200301(SERVING-UPTODATE) 900400001 1 1 SERVING [] 101000100401(SERVING-UPTODATE) 101000200401(SERVING-UPTODATE) 900500001 1 1 SERVING [] 101000100501(SERVING-UPTODATE) 101000200501(SERVING-UPTODATE) 900600001 1 1 SERVING [] 101000100601(SERVING-UPTODATE) 101000200601(SERVING-UPTODATE) 900700001 1 1 SERVING [] 101000100701(SERVING-UPTODATE) 101000200701(SERVING-UPTODATE) 900800001 1 1 SERVING [] 101000100801(SERVING-UPTODATE) 101000200801(SERVING-UPTODATE) ``` In the first attempt, I got a read bandwidth of 3.8 GB/s in the FIO read test with 16 jobs. Then, I found that the bandwidth of the FUSE client node ([ecs.g8i.16xlarge](https://help.aliyun.com/document_detail/25378.html#g8i)) is only 32 Gbps. ``` /3fs/stage/test/test1: (groupid=0, jobs=16): err= 0: pid=275608: Tue Mar 18 11:06:32 2025 read: IOPS=3743, BW=3744MiB/s (3926MB/s)(129GiB/35248msec) clat (usec): min=810, max=21786, avg=4270.83, stdev=1441.41 lat (usec): min=810, max=21787, avg=4270.88, stdev=1441.41 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 2040], 10.00th=[ 2343], 20.00th=[ 2835], | 30.00th=[ 3326], 40.00th=[ 3785], 50.00th=[ 4293], 60.00th=[ 4686], | 70.00th=[ 5145], 80.00th=[ 5604], 90.00th=[ 6259], 95.00th=[ 6652], | 99.00th=[ 7373], 99.50th=[ 7570], 99.90th=[ 8225], 99.95th=[ 8717], | 99.99th=[10159] bw ( MiB/s): min= 3460, max= 4082, per=100.00%, avg=3745.53, stdev= 7.37, samples=1120 iops : min= 3460, max= 4082, avg=3745.46, stdev= 7.37, samples=1120 lat (usec) : 1000=0.03% lat (msec) : 2=4.31%, 4=39.04%, 10=56.60%, 20=0.01%, 50=0.01% cpu : usr=0.12%, sys=0.21%, ctx=601563, majf=0, minf=183 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=131966,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3744MiB/s (3926MB/s), 3744MiB/s-3744MiB/s (3926MB/s-3926MB/s), io=129GiB (138GB), run=35248-35248msec ``` After I increased the NIC bandwidth of the FUSE client node ([ecs.g8i.48xlarge](https://help.aliyun.com/document_detail/25378.html#g8i)), the read bandwidth increased from 3.8 GB/s to 9.9 GB/s. ``` Jobs: 16 (f=16): [R(16)][100.0%][r=9893MiB/s][r=9893 IOPS][eta 00m:00s] /3fs/stage/test/test1: (groupid=0, jobs=16): err= 0: pid=16780: Tue Mar 18 11:16:06 2025 read: IOPS=9901, BW=9902MiB/s (10.4GB/s)(580GiB/60003msec) clat (usec): min=547, max=13781, avg=1615.14, stdev=646.53 lat (usec): min=547, max=13781, avg=1615.18, stdev=646.53 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 717], 10.00th=[ 799], 20.00th=[ 1012], | 30.00th=[ 1188], 40.00th=[ 1352], 50.00th=[ 1598], 60.00th=[ 1795], | 70.00th=[ 1926], 80.00th=[ 2114], 90.00th=[ 2376], 95.00th=[ 2868], | 99.00th=[ 3359], 99.50th=[ 3490], 99.90th=[ 4113], 99.95th=[ 4686], | 99.99th=[ 5932] bw ( MiB/s): min= 9286, max=10536, per=100.00%, avg=9903.46, stdev=14.11, samples=1904 iops : min= 9286, max=10536, avg=9903.45, stdev=14.12, samples=1904 lat (usec) : 750=7.05%, 1000=12.42% lat (msec) : 2=55.48%, 4=24.93%, 10=0.11%, 20=0.01% cpu : usr=0.16%, sys=0.29%, ctx=1184299, majf=0, minf=160 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=594143,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9902MiB/s (10.4GB/s), 9902MiB/s-9902MiB/s (10.4GB/s-10.4GB/s), io=580GiB (623GB), run=60003-60003msec ``` The fio test command is as following: `/root/fio/fio -numjobs=16 -fallocate=none -iodepth=1 -ioengine=external:/root/hf3fs_usrbio.so -direct=1 -rw=read -bs=1MB --group_reporting -size=100MB -time_based -runtime=60 -iodepth_batch_submit=256 -iodepth_batch_complete_min=256 -iodepth_batch_complete_max=256 -name=/3fs/stage/test/test1 -mountpoint=/3fs/stage -ior_depth=1`
bsakaguchi commented 2026-10-26 09:44:47 +00:00
Author
Owner
Copy link

@RainSnowIceRain commented on GitHub (Mar 19, 2025):

@201608ybs Thank you very much! In fact, we are using physical machines. Under a read workload with 64 threads and a block size of 1 MB, the bandwidth measured by FIO is 3.3 GB/s, while the NIC bandwidth is 43 Gb/s.
When we increase the concurrency from 64 to 128 and the block size from 1 MB to 4 MB, the NIC bandwidth increases to 55 Gb/s, but the FIO bandwidth decreases to 2.9 GB/s.

Our test workload is as follows:
fio -thread -verify=0 -numjobs=128 -ioengine=external:/root/software/3fs/benchmarks/fio_usrbio/hf3fs_usrbio.so -direct=1 -rw=read -bs=4M --group_reporting=1 -size=100M -time_based=1 -ramp_time=0 -runtime=120 -name=direct_read_bw -directory=/3fs/stage -mountpoint=/3fs/stage

We are not sure whether a similar phenomenon would occur on cloud-based virtual machines?

<!-- gh-comment-id:2735945368 --> @RainSnowIceRain commented on GitHub (Mar 19, 2025): @201608ybs Thank you very much! In fact, we are using physical machines. Under a read workload with 64 threads and a block size of 1 MB, the bandwidth measured by FIO is 3.3 GB/s, while the NIC bandwidth is 43 Gb/s. When we increase the concurrency from 64 to 128 and the block size from 1 MB to 4 MB, the NIC bandwidth increases to 55 Gb/s, but the FIO bandwidth decreases to 2.9 GB/s. Our test workload is as follows: fio -thread -verify=0 -numjobs=128 -ioengine=external:/root/software/3fs/benchmarks/fio_usrbio/hf3fs_usrbio.so -direct=1 -rw=read -bs=4M --group_reporting=1 -size=100M -time_based=1 -ramp_time=0 -runtime=120 -name=direct_read_bw -directory=/3fs/stage -mountpoint=/3fs/stage We are not sure whether a similar phenomenon would occur on cloud-based virtual machines?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#81
No description provided.