deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #281] Low write throughput despite very normal IB Network and Disk I/O #170

New issue
Closed
opened 2026-10-26 09:46:32 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:46:32 +00:00
Owner
Copy link

Originally created by @SimonCqk on GitHub (May 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/281

I am experiencing significantly lower-than-expected write throughput for 3FS, while the IB network and r/w throught for each single ssd is normal, utilization of components are far from bottleneck, but end-to-end write throughput is low.

This is likely environment-specific. I have deployed and verified 3fs using the same method in other environments, and I haven't encountered this issue.

Verified point-to-point IB throughput using ib_write_bw and tested individual SSD performance with fio. No significant abnormalities were detected in either.

The latency analysis revealed a significant disparity: Client-side write operation latency is considerably higher than the corresponding latency observed at the storage layer. Specific metrics identified with high latency include:

  • Instances of removeChunks operations.
  • fuse.piov.submit latency within the FUSE p_iov path (noted that overall latency encompasses submit and complete, and submit appears to be the dominant factor contributing to high overall latency).

Image

Image

Originally created by @SimonCqk on GitHub (May 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/281 I am experiencing significantly lower-than-expected write throughput for 3FS, while the IB network and r/w throught for each single ssd is normal, utilization of components are far from bottleneck, but end-to-end write throughput is low. > **This is likely environment-specific. I have deployed and verified 3fs using the same method in other environments, and I haven't encountered this issue.** Verified point-to-point IB throughput using ib_write_bw and tested individual SSD performance with fio. No significant abnormalities were detected in either. The latency analysis revealed a significant disparity: Client-side write operation latency is considerably higher than the corresponding latency observed at the storage layer. Specific metrics identified with high latency include: - Instances of `removeChunks` operations. - `fuse.piov.submit` latency within the FUSE p_iov path (noted that `overall` latency encompasses `submit and complete`, and submit appears to be the dominant factor contributing to high overall latency). ![Image](https://github.com/user-attachments/assets/2c340a30-ea4f-4474-aa7c-e0d0941dc454) ![Image](https://github.com/user-attachments/assets/3bb99025-2ce5-4d68-8cfe-29158e6fd1a7)
bsakaguchi commented 2026-10-26 09:46:34 +00:00
Author
Owner
Copy link

@bugwz commented on GitHub (May 20, 2025):

Recommend some 3fs Grafana dashboards for more intuitive troubleshooting.

https://grafana.com/grafana/dashboards/?search=3fs

<!-- gh-comment-id:2892611885 --> @bugwz commented on GitHub (May 20, 2025): Recommend some 3fs Grafana dashboards for more intuitive troubleshooting. https://grafana.com/grafana/dashboards/?search=3fs
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#170
No description provided.