deepseek-ai/3FS

Fork 0

mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00

[GH-ISSUE #319] Can we use GPU Direct Storage in 3FS？ #198

New issue

Open

opened 2026-10-26 09:46:58 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-26 09:46:58 +00:00

Owner

Originally created by @SgtPepperr on GitHub (Aug 1, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/319

I noticed that the native interface of 3fs provides using iov as the buffer for data writing or reading. I'd like to ask if 3fs exposes any interfaces or functionalities that can utilize GDS (GPU Direct Storage), or if there are any methods to integrate 3fs with GDS to achieve direct data transfer from persistent storage to GPU memory? Thanks for any help！

Originally created by @SgtPepperr on GitHub (Aug 1, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/319 I noticed that the native interface of 3fs provides using iov as the buffer for data writing or reading. I'd like to ask if 3fs exposes any interfaces or functionalities that can utilize GDS (GPU Direct Storage), or if there are any methods to integrate 3fs with GDS to achieve direct data transfer from persistent storage to GPU memory? Thanks for any help！

bsakaguchi commented

2026-10-26 09:47:00 +00:00

Author

Owner

@SimonCqk commented on GitHub (Aug 11, 2025):

Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :)

@SimonCqk commented on GitHub (Aug 11, 2025): Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :)

bsakaguchi commented

2026-10-26 09:47:00 +00:00

Author

Owner

@SgtPepperr commented on GitHub (Aug 11, 2025):

Great, I'm really looking forward to your follow-up PR. Currently, we also need the 3fs+GDS functionality to further accelerate performance in certain scenarios.

@SgtPepperr commented on GitHub (Aug 11, 2025): Great, I'm really looking forward to your follow-up PR. Currently, we also need the 3fs+GDS functionality to further accelerate performance in certain scenarios.

bsakaguchi commented

2026-10-26 09:47:00 +00:00

Author

Owner

@SimonCqk commented on GitHub (Aug 11, 2025):

Great, I'm really looking forward to your follow-up PR. Currently, we also need the 3fs+GDS functionality to further accelerate performance in certain scenarios.

Apologies!! I misread the context. To clarify, we have implemented GDR, but GDS has not been implemented yet. In the current 3FS file layout design, file content is split into multiple distributed chunks, and the single-node IOBuffer class is already highly optimized. Data read from an SSD goes directly to the NIC's registered memory space, eliminating an extra IO buffer copy. So, it seems the end-to-end benefit of GDS might be quite limited (IMO) ?

@SimonCqk commented on GitHub (Aug 11, 2025): > Great, I'm really looking forward to your follow-up PR. Currently, we also need the 3fs+GDS functionality to further accelerate performance in certain scenarios. Apologies!! I misread the context. To clarify, we have implemented GDR, but GDS has not been implemented yet. In the current 3FS file layout design, file content is split into multiple distributed chunks, and the single-node IOBuffer class is already highly optimized. Data read from an SSD goes directly to the NIC's registered memory space, eliminating an extra IO buffer copy. So, it seems the end-to-end benefit of GDS might be quite limited (IMO) ?

bsakaguchi commented

2026-10-26 09:47:01 +00:00

Author

Owner

@kangyifei commented on GitHub (Aug 19, 2025):

Based on my understanding, when GDS is used with remote filesystems such as NFS over RDMA or NVMe over RDMA, it is essentially a NIC-to-GPU DMA。So supporting GDR in 3fs should effectively implement GDS.

@kangyifei commented on GitHub (Aug 19, 2025): Based on my understanding, when GDS is used with remote filesystems such as NFS over RDMA or NVMe over RDMA, it is essentially a NIC-to-GPU DMA。So supporting GDR in 3fs should effectively implement GDS.

bsakaguchi commented

2026-10-26 09:47:01 +00:00

Author

Owner

@ChenxingZhou777 commented on GitHub (Feb 12, 2026):

Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :)

I’d like to know whether you’ve open-sourced your work.

@ChenxingZhou777 commented on GitHub (Feb 12, 2026): > Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :) I’d like to know whether you’ve open-sourced your work.

bsakaguchi commented

2026-10-26 09:47:01 +00:00

Author

Owner

@SimonCqk commented on GitHub (Feb 12, 2026):

Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :)

I’d like to know whether you’ve open-sourced your work.

You can track this topic in #395 , the plan is to align with the maintainers first. Once we're on the same page, I'll submit the design doc and code :)

@SimonCqk commented on GitHub (Feb 12, 2026): > > Yes, it's possible to enable GDR with some proper extensions to USRBIO. In fact, we've already implemented a preliminary version and can contribute it to the community later on :) > > I’d like to know whether you’ve open-sourced your work. You can track this topic in #395 , the plan is to align with the maintainers first. Once we're on the same page, I'll submit the design doc and code :)

bsakaguchi referenced this issue

2026-10-26 09:48:38 +00:00

[PR #198] [CLOSED] Optimize main CMakeLists.txt #321