deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #245] 2 * 100Gbps RDMA network cards, single client write throughput can only reach 6GiB/s ??? #151

New issue
Open
opened 2026-10-26 09:46:01 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-26 09:46:01 +00:00
Owner
Copy link

Originally created by @zk-kiger on GitHub (Apr 17, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/245

Deploy 6 storage nodes(24 * nvmeSSD per node), all with 2 * 100Gbps network cards, network card model
0000:17:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
0000:17:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

Client fio can only run at 6GiB/s, the fio command is as follows:

/root/3fs/fio --name=500M_1M_bigio_write -numjobs=100 -fallocate=none \
-ioengine=external:/usr/lib64/hf3fs_usrbio.so -direct=1 -rw=write -bs=1MB \
--group_reporting -filesize=500MB --nrfiles=10 -iodepth=1 \
--directory=/3fs/stage/500M_1M_bigio_write -mountpoint=/3fs/stage -ior_depth=1

Image

Does this capability meet 3FS expectations? Are there other optimization parameters that are not enabled?

Originally created by @zk-kiger on GitHub (Apr 17, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/245 Deploy 6 storage nodes(24 * nvmeSSD per node), all with 2 * 100Gbps network cards, network card model 0000:17:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 0000:17:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] Client fio can only run at 6GiB/s, the fio command is as follows: ``` /root/3fs/fio --name=500M_1M_bigio_write -numjobs=100 -fallocate=none \ -ioengine=external:/usr/lib64/hf3fs_usrbio.so -direct=1 -rw=write -bs=1MB \ --group_reporting -filesize=500MB --nrfiles=10 -iodepth=1 \ --directory=/3fs/stage/500M_1M_bigio_write -mountpoint=/3fs/stage -ior_depth=1 ``` ![Image](https://github.com/user-attachments/assets/851d89b3-49d9-48da-a171-06f4e535a214) Does this capability meet 3FS expectations? Are there other optimization parameters that are not enabled?
bsakaguchi commented 2026-10-26 09:46:02 +00:00
Author
Owner
Copy link

@feeyman commented on GitHub (Apr 23, 2025):

Deploy 6 storage nodes(24 * nvmeSSD per node), all with 2 * 100Gbps network cards, network card model 0000:17:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 0000:17:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

Client fio can only run at 6GiB/s, the fio command is as follows:

/root/3fs/fio --name=500M_1M_bigio_write -numjobs=100 -fallocate=none \
-ioengine=external:/usr/lib64/hf3fs_usrbio.so -direct=1 -rw=write -bs=1MB \
--group_reporting -filesize=500MB --nrfiles=10 -iodepth=1 \
--directory=/3fs/stage/500M_1M_bigio_write -mountpoint=/3fs/stage -ior_depth=1

Image

Does this capability meet 3FS expectations? Are there other optimization parameters that are not enabled?

@zk-kiger Hello, may I ask if your issue has been resolved? By the way, what is the performance test for sequential reading?

<!-- gh-comment-id:2823420930 --> @feeyman commented on GitHub (Apr 23, 2025): > Deploy 6 storage nodes(24 * nvmeSSD per node), all with 2 * 100Gbps network cards, network card model 0000:17:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 0000:17:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] > > Client fio can only run at 6GiB/s, the fio command is as follows: > > ``` > /root/3fs/fio --name=500M_1M_bigio_write -numjobs=100 -fallocate=none \ > -ioengine=external:/usr/lib64/hf3fs_usrbio.so -direct=1 -rw=write -bs=1MB \ > --group_reporting -filesize=500MB --nrfiles=10 -iodepth=1 \ > --directory=/3fs/stage/500M_1M_bigio_write -mountpoint=/3fs/stage -ior_depth=1 > ``` > > ![Image](https://github.com/user-attachments/assets/851d89b3-49d9-48da-a171-06f4e535a214) > > Does this capability meet 3FS expectations? Are there other optimization parameters that are not enabled? @zk-kiger Hello, may I ask if your issue has been resolved? By the way, what is the performance test for sequential reading?
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@zk-kiger commented on GitHub (Apr 23, 2025):

not yet, single client sequential read can reach 18GiB/s

<!-- gh-comment-id:2823936889 --> @zk-kiger commented on GitHub (Apr 23, 2025): not yet, single client sequential read can reach 18GiB/s
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@RangerCD commented on GitHub (Apr 25, 2025):

I built a 3FS cluster at about 1/6 of your scale(23 SSDs total), and the performance I achieved is also roughly 1/6 of yours(~900MiB/s). I believe my observed performance aligns with expectations and 3FS's design principles, since I'm using QLC SSDs with poor direct write speed. Could you share the SSD model you used in your setup?

<!-- gh-comment-id:2829271047 --> @RangerCD commented on GitHub (Apr 25, 2025): I built a 3FS cluster at about 1/6 of your scale(23 SSDs total), and the performance I achieved is also roughly 1/6 of yours(~900MiB/s). I believe my observed performance aligns with expectations and 3FS's design principles, since I'm using QLC SSDs with poor direct write speed. Could you share the SSD model you used in your setup?
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@zk-kiger commented on GitHub (Apr 25, 2025):

Intel SSDPF2KX076T1 Hard Drive 7.68TB SSD PCIe 4.0 x4 NVMe U.2 15mm TLC

<!-- gh-comment-id:2829277221 --> @zk-kiger commented on GitHub (Apr 25, 2025): Intel SSDPF2KX076T1 Hard Drive 7.68TB SSD PCIe 4.0 x4 NVMe U.2 15mm TLC
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@RangerCD commented on GitHub (Apr 29, 2025):

With this SSD, your cluster should achieve approximately 60GiB/s in write performance.

<!-- gh-comment-id:2837299303 --> @RangerCD commented on GitHub (Apr 29, 2025): With this SSD, your cluster should achieve approximately 60GiB/s in write performance.
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@Funny-Boy-Max commented on GitHub (Apr 29, 2025):

还没有，单客户端顺序读可以达到18GiB/s

佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况

<!-- gh-comment-id:2837794390 --> @Funny-Boy-Max commented on GitHub (Apr 29, 2025): > 还没有，单客户端顺序读可以达到18GiB/s 佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况
bsakaguchi commented 2026-10-26 09:46:03 +00:00
Author
Owner
Copy link

@bugwz commented on GitHub (May 9, 2025):

佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况

@Funny-Boy-Max iftop的流量不统计 ib 流量，为此你可以直接统计对应网卡设备的数据信息来监控对应 ib 的网络流量，给你提供一个脚本。

#!/bin/bash
DEVICE="mlx5_bond_0"   # 替换为你的 RDMA 设备名（通过 `ibstat` 查看）
PORT=1            # 替换为端口号（通常是 1）
INTERVAL=1        # 刷新间隔（秒）

# 初始化计数器
prev_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data)
prev_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data)

while true; do
  sleep $INTERVAL
  # 读取当前计数器
  curr_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data)
  curr_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data)
  
  # 计算差值并转换为速率（单位：MB/s）
  rcv_bytes=$(( (curr_rcv - prev_rcv) * 4 ))  # RDMA 计数器单位为 4 字节
  xmit_bytes=$(( (curr_xmit - prev_xmit) * 4 ))
  rcv_rate=$(echo "scale=2; $rcv_bytes / $INTERVAL / 1000000" | bc)  # 转换为 MB/s
  xmit_rate=$(echo "scale=2; $xmit_bytes / $INTERVAL / 1000000" | bc)
  
  # 输出结果
  echo -e "RX: ${rcv_rate} MB/s \t TX: ${xmit_rate} MB/s"
  
  # 更新计数器
  prev_rcv=$curr_rcv
  prev_xmit=$curr_xmit
done
<!-- gh-comment-id:2865941761 --> @bugwz commented on GitHub (May 9, 2025): > 佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况 @Funny-Boy-Max iftop的流量不统计 ib 流量，为此你可以直接统计对应网卡设备的数据信息来监控对应 ib 的网络流量，给你提供一个脚本。 ```bash #!/bin/bash DEVICE="mlx5_bond_0" # 替换为你的 RDMA 设备名（通过 `ibstat` 查看） PORT=1 # 替换为端口号（通常是 1） INTERVAL=1 # 刷新间隔（秒） # 初始化计数器 prev_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data) prev_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data) while true; do sleep $INTERVAL # 读取当前计数器 curr_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data) curr_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data) # 计算差值并转换为速率（单位：MB/s） rcv_bytes=$(( (curr_rcv - prev_rcv) * 4 )) # RDMA 计数器单位为 4 字节 xmit_bytes=$(( (curr_xmit - prev_xmit) * 4 )) rcv_rate=$(echo "scale=2; $rcv_bytes / $INTERVAL / 1000000" | bc) # 转换为 MB/s xmit_rate=$(echo "scale=2; $xmit_bytes / $INTERVAL / 1000000" | bc) # 输出结果 echo -e "RX: ${rcv_rate} MB/s \t TX: ${xmit_rate} MB/s" # 更新计数器 prev_rcv=$curr_rcv prev_xmit=$curr_xmit done ```
bsakaguchi commented 2026-10-26 09:46:04 +00:00
Author
Owner
Copy link

@Funny-Boy-Max commented on GitHub (May 12, 2025):

佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况

@Funny-Boy-Max iftop的流量不统计 ib 流量，为此你可以直接统计对应网卡设备的数据信息来监控对应 ib 的网络流量，给你提供一个脚本。

#!/bin/bash
DEVICE="mlx5_bond_0" # 替换为你的 RDMA 设备名（通过 ibstat 查看）
PORT=1 # 替换为端口号（通常是 1）
INTERVAL=1 # 刷新间隔（秒）

初始化计数器

prev_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data)
prev_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data)

while true; do
sleep $INTERVAL

读取当前计数器

curr_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data)
curr_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data)

计算差值并转换为速率（单位：MB/s）

rcv_bytes=$(( (curr_rcv - prev_rcv) * 4 )) # RDMA 计数器单位为 4 字节
xmit_bytes=$(( (curr_xmit - prev_xmit) * 4 ))
rcv_rate=$(echo "scale=2; $rcv_bytes / $INTERVAL / 1000000" | bc) # 转换为 MB/s
xmit_rate=$(echo "scale=2; $xmit_bytes / $INTERVAL / 1000000" | bc)

输出结果

echo -e "RX: ${rcv_rate} MB/s \t TX: ${xmit_rate} MB/s"

更新计数器

prev_rcv=$curr_rcv
prev_xmit=$curr_xmit
done

兄弟靠谱

<!-- gh-comment-id:2870485762 --> @Funny-Boy-Max commented on GitHub (May 12, 2025): > > 佬，请问你的测试命令是什么，我这边4个存储节点，1个元数据节点，存储节点内2个SSD，我这边测试下来是8467MB/s，但是我无法监测RDMA网络流量，这是为什么，具体如何能够监测到读写测试中，RDMA的使用带框情况 > > [@Funny-Boy-Max](https://github.com/Funny-Boy-Max) iftop的流量不统计 ib 流量，为此你可以直接统计对应网卡设备的数据信息来监控对应 ib 的网络流量，给你提供一个脚本。 > > #!/bin/bash > DEVICE="mlx5_bond_0" # 替换为你的 RDMA 设备名（通过 `ibstat` 查看） > PORT=1 # 替换为端口号（通常是 1） > INTERVAL=1 # 刷新间隔（秒） > > # 初始化计数器 > prev_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data) > prev_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data) > > while true; do > sleep $INTERVAL > # 读取当前计数器 > curr_rcv=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_rcv_data) > curr_xmit=$(cat /sys/class/infiniband/$DEVICE/ports/$PORT/counters/port_xmit_data) > > # 计算差值并转换为速率（单位：MB/s） > rcv_bytes=$(( (curr_rcv - prev_rcv) * 4 )) # RDMA 计数器单位为 4 字节 > xmit_bytes=$(( (curr_xmit - prev_xmit) * 4 )) > rcv_rate=$(echo "scale=2; $rcv_bytes / $INTERVAL / 1000000" | bc) # 转换为 MB/s > xmit_rate=$(echo "scale=2; $xmit_bytes / $INTERVAL / 1000000" | bc) > > # 输出结果 > echo -e "RX: ${rcv_rate} MB/s \t TX: ${xmit_rate} MB/s" > > # 更新计数器 > prev_rcv=$curr_rcv > prev_xmit=$curr_xmit > done 兄弟靠谱
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#151
No description provided.