[GH-ISSUE #395] [Feature Request] Any plan for official GDR support to enable zero-copy GPU IO ? #230

New issue

Open

opened 2026-10-26 09:47:28 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:47:28 +00:00

Owner

Originally created by @SimonCqk on GitHub (Feb 4, 2026).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/395

As we deploy 3FS in production to accelerate both training and inference, currently, data loading involves a CPU bounce buffer, which consumes PCIe bandwidth and CPU cycles. Enabling GDR support would allow for direct DMA transfers between the storage backend and GPU memory, significantly improving throughput and reducing latency (also reduce cpu cycles).

Before move forward, I'd ask is there an official roadmap or ongoing internal effort to support GDR/GDS in 3FS? I have already developed a working PoC that implements end-to-end GDR support. The implementation touches the following key components:

usrbio : Extended the iov/ior structures to support device pointers and memory registration awareness, also introduced a new create API (and/or extended flag support) to facilitate proper GPU memory handle registration during file operations.
backend impl: Added the necessary logic in both the FUSE and Storage layers to handle the RDMA operations directly to GPU addresses.

If the community are open to this proposal, I would submit a design doc and break down implementations into individual PRs.

Looking forward to your feedback.

Originally created by @SimonCqk on GitHub (Feb 4, 2026). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/395 As we deploy 3FS in production to accelerate both training and inference, currently, data loading involves a CPU bounce buffer, which consumes PCIe bandwidth and CPU cycles. Enabling GDR support would allow for direct DMA transfers between the storage backend and GPU memory, significantly improving throughput and reducing latency (also reduce cpu cycles). Before move forward, I'd ask is there an official roadmap or ongoing internal effort to support GDR/GDS in 3FS? I have already developed a working PoC that implements end-to-end GDR support. The implementation touches the following key components: 1. `usrbio` : Extended the iov/ior structures to support device pointers and memory registration awareness, also introduced a new create API (and/or extended flag support) to facilitate proper GPU memory handle registration during file operations. 2. backend impl: Added the necessary logic in both the FUSE and Storage layers to handle the RDMA operations directly to GPU addresses. If the community are open to this proposal, I would submit a design doc and break down implementations into individual PRs. Looking forward to your feedback.

bsakaguchi commented

2026-10-26 09:47:30 +00:00

Author

Owner

@ChenxingZhou777 commented on GitHub (Feb 12, 2026):

When you mention “memory registration awareness,” what is the concrete mechanism you use to pass the GPU memory handle to the FUSE daemon?
Is it done by extending the existing iov/ior structures, by overloading something like symlinks, or via a new control/side channel?

@ChenxingZhou777 commented on GitHub (Feb 12, 2026): When you mention “memory registration awareness,” what is the concrete mechanism you use to pass the GPU memory handle to the FUSE daemon? Is it done by extending the existing iov/ior structures, by overloading something like symlinks, or via a new control/side channel?

bsakaguchi commented

2026-10-26 09:47:30 +00:00

Author

Owner

@SimonCqk commented on GitHub (Feb 27, 2026):

When you mention “memory registration awareness,” what is the concrete mechanism you use to pass the GPU memory handle to the FUSE daemon? Is it done by extending the existing iov/ior structures, by overloading something like symlinks, or via a new control/side channel?

extending iov/ior structures to minimize upgrade overhead, we should aim for minimal API surface changes for our usrbio users.

@SimonCqk commented on GitHub (Feb 27, 2026): > When you mention “memory registration awareness,” what is the concrete mechanism you use to pass the GPU memory handle to the FUSE daemon? Is it done by extending the existing iov/ior structures, by overloading something like symlinks, or via a new control/side channel? extending iov/ior structures to minimize upgrade overhead, we should aim for minimal API surface changes for our usrbio users.

bsakaguchi commented

2026-10-26 09:47:30 +00:00

Author

Owner

@ChenxingZhou777 commented on GitHub (Mar 4, 2026):

How does your implementation handle this cross-process GPU memory registration? Specifically, which mechanism do you use to let the FUSE daemon obtain a valid MR for the user app's GPU memory?

@ChenxingZhou777 commented on GitHub (Mar 4, 2026): How does your implementation handle this cross-process GPU memory registration? Specifically, which mechanism do you use to let the FUSE daemon obtain a valid MR for the user app's GPU memory?

bsakaguchi commented

2026-10-26 09:47:30 +00:00

Author

Owner

@SimonCqk commented on GitHub (Mar 17, 2026):

@ChenxingZhou777 for more details, please refer #402 , feel free to chime in with any thoughts !

@SimonCqk commented on GitHub (Mar 17, 2026): @ChenxingZhou777 for more details, please refer #402 , feel free to chime in with any thoughts !

bsakaguchi referenced this issue

2026-10-26 09:48:48 +00:00

[PR #230] [MERGED] fix targetId parse in set-preferred-target-order #336