mirror of
https://github.com/deepseek-ai/3FS.git
synced 2026-11-05 00:47:47 +00:00
[GH-ISSUE #410] Performance Issue with 6 storage nodes #237
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/3FS#237
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @MTCXin on GitHub (Apr 20, 2026).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/410
I built 3FS mostly following README, and the only Services Side difference is that I made 6 storage nodes (5 in README) , 1*400Gb NIC & 5 NVMe on each (model Samsung 1733 15.36T) (16 in README),
----Performance----
Using one of the storage node as client (2*EPYC 9554, 1.5T mem) (which is also meta node), I got following benchmark results:
( fio -rw=randread -bs=4K -iodepth=1024 -iodepth_batch_submit=1024 -iodepth_batch_complete_min=1024 -iodepth_batch_complete_max=1024 -size=1G -ioengine=hf3fs_usrbio.so -direct=1 -numjobs=128 -iodepth=1)
4K randread: 810K IOPS
( fio -rw=randread -bs=1M -size=1G -ioengine=hf3fs_usrbio.so -direct=1 -numjobs=32 -iodepth=1)
1M randread: 23.2 GiB/s
( fio -rw=randread -bs=4M -size=1G -ioengine=hf3fs_usrbio.so -direct=1 -numjobs=32 -iodepth=1)
4M randread: 32.5 GiB/s
---Performance Updates---
After modifying fuse_main.toml. 4K rand read reach to 2M IOPS,
with -iodepth=4 -iodepth_batch_submit=4 -iodepth_batch_complete_min=4 -iodepth_batch_complete_max=4, 1M rand read also reach to NIC bottleneck of ~42GiB/s.
The concern comes to rand write:
( fio -rw=randwrite -bs=1M -iodepth=8 -iodepth_batch_submit=8 -iodepth_batch_complete_min=8 -iodepth_batch_complete_max= 8 -size=1G -ioengine=hf3fs_usrbio.so -direct=1 -numjobs=32)
1M randwrite: 7.6 GiB/s (3min avg)
There's a phenominan that it hit to 11GiB/s most of time, but the writing speed would drop for a while periodically, is this somewhat 3FS feature?
Also I found no scalabilty in my system, 2 clients rand write: 7.2GiB/s (in total), 3 clients rand write: 6.6GiB/s (in total)
----Concerns----
Following readme, setting -num_targets_per_disk 6 didn't work, I followed Gemini's guide to change it to 10, could this be the potential problem?
I noticed that during fio test, there're 2 cores always heavily used, could this related to [client.io_worker.transport_pool] or [client.thread_pool] in client config? should I do some modification?
I'm looking forward to your response and ready to provide any further details