deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #236] FIO顺序读性能只有（6.7G/s），但是网卡却显满载(12G/s)，请问有遇到过这个问题的吗？ #144

New issue
Closed
opened 2026-10-26 09:45:54 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:45:54 +00:00
Owner
Copy link

Originally created by @feeyman on GitHub (Apr 10, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/236

大家好，遇到一个问题，FIO测试时，结果和网卡流量统计不一致；
FIO结果只有6.7G/s，但是IB的流量已经满负荷了（12G/s），请问有遇到过这个问题的吗？

Hello everyone, I have encountered a problem:
The network card traffic of the metadata node is fully loaded, but the FIO tool shows only half of the full load.
What is the reason for this? What kind of traffic is occupying the network card without clearing it? Because the network card traffic of the storage node is only half of its full load.

The 3fs cluster structure is as follows:
Image

The traffic display of IB network card is as follows:
Image

Looking forward to guidance, thank you.

Originally created by @feeyman on GitHub (Apr 10, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/236 大家好，遇到一个问题，FIO测试时，结果和网卡流量统计不一致； FIO结果只有6.7G/s，但是IB的流量已经满负荷了（12G/s），请问有遇到过这个问题的吗？ Hello everyone, I have encountered a problem: The network card traffic of the metadata node is **fully loaded**, but the FIO tool shows only **half of the full load.** What is the reason for this? **What kind of traffic is occupying the network card without clearing it?** Because the network card traffic of the storage node is only half of its full load. **The 3fs cluster structure is as follows:** ![Image](https://github.com/user-attachments/assets/4435f388-9a76-4d5f-bbbc-9f1d2dc92cba) **The traffic display of IB network card is as follows:** ![Image](https://github.com/user-attachments/assets/aa5d4008-ae18-4e05-98b8-63653a56fada) **Looking forward to guidance, thank you.**
bsakaguchi commented 2026-10-26 09:45:54 +00:00
Author
Owner
Copy link

@bugwz commented on GitHub (Apr 11, 2025):

给你几个排查建议：

  1. 排查实时带宽显示数值是否正常：
    • 配合 ib_send_bw 工具压测带宽，对比你的带宽实时显示数值与 ib_send_bw 的数值是否接近：
# 服务器
ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1

# 客户端
ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP
  1. 排查是否还有其他 fio 压测进程同时处于运行状态；
<!-- gh-comment-id:2795713036 --> @bugwz commented on GitHub (Apr 11, 2025): 给你几个排查建议： 1. 排查实时带宽显示数值是否正常： + 配合 `ib_send_bw` 工具压测带宽，对比你的带宽实时显示数值与 `ib_send_bw` 的数值是否接近： ```bash # 服务器 ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1 # 客户端 ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP ``` 2. 排查是否还有其他 fio 压测进程同时处于运行状态；
bsakaguchi commented 2026-10-26 09:45:55 +00:00
Author
Owner
Copy link

@Icedroid commented on GitHub (Apr 11, 2025):

给你几个排查建议：

  1. 排查实时带宽显示数值是否正常：

    • 配合 ib_send_bw 工具压测带宽，对比你的带宽实时显示数值与 ib_send_bw 的数值是否接近：

服务器

ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1

客户端

ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP
2. 排查是否还有其他 fio 压测进程同时处于运行状态；

我用这个压测试带宽是2789.15 MB/s，单网卡是25Gbps，说明打满单网卡，那我是两张25G卡Bond，是RDMA网卡对Bond卡不生效？
还有个问题想请问楼主，RDMA 实时带宽是用哪个命令查看的？

<!-- gh-comment-id:2795754032 --> @Icedroid commented on GitHub (Apr 11, 2025): > 给你几个排查建议： > > 1. 排查实时带宽显示数值是否正常： > > * 配合 `ib_send_bw` 工具压测带宽，对比你的带宽实时显示数值与 `ib_send_bw` 的数值是否接近： > > # 服务器 > ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1 > > # 客户端 > ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP > 2. 排查是否还有其他 fio 压测进程同时处于运行状态； 我用这个压测试带宽是2789.15 MB/s，单网卡是25Gbps，说明打满单网卡，那我是两张25G卡Bond，是RDMA网卡对Bond卡不生效？ 还有个问题想请问楼主，RDMA 实时带宽是用哪个命令查看的？
bsakaguchi commented 2026-10-26 09:45:55 +00:00
Author
Owner
Copy link

@feeyman commented on GitHub (Apr 11, 2025):

给你几个排查建议：

  1. 排查实时带宽显示数值是否正常：

    • 配合 ib_send_bw 工具压测带宽，对比你的带宽实时显示数值与 ib_send_bw 的数值是否接近：

服务器

ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1

客户端

ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP
2. 排查是否还有其他 fio 压测进程同时处于运行状态；

多谢建议，试了一下，233--->>234，统计结果还是有误差，不确定是不是ibtop统计的问题。

Image

<!-- gh-comment-id:2796229876 --> @feeyman commented on GitHub (Apr 11, 2025): > 给你几个排查建议： > > 1. 排查实时带宽显示数值是否正常： > > * 配合 `ib_send_bw` 工具压测带宽，对比你的带宽实时显示数值与 `ib_send_bw` 的数值是否接近： > > # 服务器 > ib_send_bw -d mlx5_0 -a -n 1000000 -c RC -i 1 > > # 客户端 > ib_send_bw -a -n 1000000 -c RC -d mlx5_0 -i 1 $SERVER_IP > 2. 排查是否还有其他 fio 压测进程同时处于运行状态； 多谢建议，试了一下，233--->>234，统计结果还是有误差，不确定是不是ibtop统计的问题。 ![Image](https://github.com/user-attachments/assets/a687dca6-2660-473c-a142-3eb5a0d3bbd6)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#144
No description provided.