[PR #407] feat(gdr): GPU Direct RDMA (GDR) support for direct storage-to-GPU transfers. #420

New issue

Open

opened 2026-10-26 09:50:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:50:06 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/3FS/pull/407
Author: @SimonCqk
Created: 4/2/2026
Status: 🔄 Open

Base: main ← Head: features/gdr-integration

📝 Commits (4)

81ee9c6 feat(gdr): Add GPU Direct RDMA (GDR) support for direct storage-to-GPU transfers
36c7c67 Enable GDR main path
0ea5a9c Refine GDR ownership and data path
ce56194 Simplify GDR memory registration path

📊 Changes

54 files changed (+5475 additions, -242 deletions)

View changed files

📝 .github/workflows/build.yml (+26 -1)
📝 CMakeLists.txt (+1 -0)
📝 cmake/CLangFormat.cmake (+2 -2)
➕ cmake/Cuda.cmake (+89 -0)
➕ docs/gdr-integration.md (+637 -0)
📝 src/client/storage/CMakeLists.txt (+4 -0)
📝 src/client/storage/StorageClient.cc (+116 -0)
📝 src/client/storage/StorageClient.h (+90 -7)
📝 src/client/storage/StorageClientImpl.cc (+90 -20)
📝 src/client/storage/StorageClientImpl.h (+4 -0)
📝 src/common/CMakeLists.txt (+10 -0)
➕ src/common/cuda/CudaMemory.cc (+254 -0)
➕ src/common/cuda/CudaMemory.h (+50 -0)
📝 src/common/net/ib/RDMABuf.cc (+41 -2)
📝 src/common/net/ib/RDMABuf.h (+88 -16)
📝 src/fbs/storage/Common.h (+1 -0)
📝 src/fuse/CMakeLists.txt (+3 -0)
📝 src/fuse/FuseClients.cc (+27 -38)
📝 src/fuse/FuseClients.h (+11 -0)
📝 src/fuse/FuseOps.cc (+14 -15)

...and 34 more files

📄 Description

See design doc plan: #402

Changes:

Add GPU Direct RDMA (GDR) support to 3FS usrbio, enabling direct data transfers between storage and GPU vRAM without host memory bounce buffers.

API

New _device API variants: hf3fs_iovcreate_device, hf3fs_iovopen_device, hf3fs_iovwrap_device
iovcreate_device always available with transparent host fallback; iovopen_device/iovwrap_device
require HF3FS_GDR_ENABLED
hf3fs_iovsync(iov, direction) for GPU↔NIC coherency fencing
Runtime capability queries: hf3fs_gdr_available(), hf3fs_gdr_device_count(), hf3fs_iov_mem_type()
Existing host API unchanged and fully backward compatible

RDMA Layer

RDMABufAccelerator for GPU memory RDMA registration via nvidia_peermem
RDMABufUnified as polymorphic host/GPU RDMA buffer abstraction
AcceleratorMemoryRegionCache for MR caching with per-device eviction
AcceleratorMemoryBridge for cross-process CUDA IPC import/export
IBSocket::RDMAReqBatch::add() and CallContext::RDMATransmission::add() overloads for
RDMABufUnified

StorageClient

IOBuffer wraps RDMABufUnified instead of plain RDMABuf
Inline data transfer automatically disabled for GPU buffers
Client-side CPU checksum automatically skipped for GPU buffers
registerGpuIOBuffer() with fallback to RDMABuf::createFromUserBuffer

Fuse Daemon

IovTable parses .gdr.d{N} symlink keys and gdr://v1/device/{id}/size/{size}/ipc/{hex} URIs
GpuShmBuf/GpuShmBufForIO for daemon-side imported GPU buffer management
IoBufForIO = std::variant<ShmBufForIO, GpuShmBufForIO> with std::visit dispatch in
IoRing::process()
lookupBufs two-phase locking: shmLock then gpuShmLock (never nested)

Build & Infra

cmake/Cuda.cmake for CUDA toolkit detection and HF3FS_GDR_ENABLED flag
CI workflow updated for optional GDR build matrix

Docs

docs/gdr-integration.md: user-facing guide with prerequisites, API reference, 3 complete usage
examples, runtime behavior, and known limitations

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/3FS/pull/407 **Author:** [@SimonCqk](https://github.com/SimonCqk) **Created:** 4/2/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `features/gdr-integration` --- ### 📝 Commits (4) - [`81ee9c6`](https://github.com/deepseek-ai/3FS/commit/81ee9c6f45073a397ce65254472522870ea469e3) feat(gdr): Add GPU Direct RDMA (GDR) support for direct storage-to-GPU transfers - [`36c7c67`](https://github.com/deepseek-ai/3FS/commit/36c7c67ffb973ab641ea942d4b09681457319960) Enable GDR main path - [`0ea5a9c`](https://github.com/deepseek-ai/3FS/commit/0ea5a9c29c8eb5a7c035d007a31838a8e9dbdf76) Refine GDR ownership and data path - [`ce56194`](https://github.com/deepseek-ai/3FS/commit/ce561949808290828156a30b81cd425863b99d09) Simplify GDR memory registration path ### 📊 Changes **54 files changed** (+5475 additions, -242 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/build.yml` (+26 -1) 📝 `CMakeLists.txt` (+1 -0) 📝 `cmake/CLangFormat.cmake` (+2 -2) ➕ `cmake/Cuda.cmake` (+89 -0) ➕ `docs/gdr-integration.md` (+637 -0) 📝 `src/client/storage/CMakeLists.txt` (+4 -0) 📝 `src/client/storage/StorageClient.cc` (+116 -0) 📝 `src/client/storage/StorageClient.h` (+90 -7) 📝 `src/client/storage/StorageClientImpl.cc` (+90 -20) 📝 `src/client/storage/StorageClientImpl.h` (+4 -0) 📝 `src/common/CMakeLists.txt` (+10 -0) ➕ `src/common/cuda/CudaMemory.cc` (+254 -0) ➕ `src/common/cuda/CudaMemory.h` (+50 -0) 📝 `src/common/net/ib/RDMABuf.cc` (+41 -2) 📝 `src/common/net/ib/RDMABuf.h` (+88 -16) 📝 `src/fbs/storage/Common.h` (+1 -0) 📝 `src/fuse/CMakeLists.txt` (+3 -0) 📝 `src/fuse/FuseClients.cc` (+27 -38) 📝 `src/fuse/FuseClients.h` (+11 -0) 📝 `src/fuse/FuseOps.cc` (+14 -15) _...and 34 more files_ </details> ### 📄 Description See design doc plan: #402 Changes: Add GPU Direct RDMA (GDR) support to 3FS usrbio, enabling direct data transfers between storage and GPU vRAM without host memory bounce buffers. ### API - New `_device` API variants: `hf3fs_iovcreate_device`, `hf3fs_iovopen_device`, `hf3fs_iovwrap_device` - `iovcreate_device` always available with transparent host fallback; `iovopen_device`/`iovwrap_device` require `HF3FS_GDR_ENABLED` - `hf3fs_iovsync(iov, direction)` for GPU↔NIC coherency fencing - Runtime capability queries: `hf3fs_gdr_available()`, `hf3fs_gdr_device_count()`, `hf3fs_iov_mem_type()` - Existing host API unchanged and fully backward compatible ### RDMA Layer - `RDMABufAccelerator` for GPU memory RDMA registration via `nvidia_peermem` - `RDMABufUnified` as polymorphic host/GPU RDMA buffer abstraction - `AcceleratorMemoryRegionCache` for MR caching with per-device eviction - `AcceleratorMemoryBridge` for cross-process CUDA IPC import/export - `IBSocket::RDMAReqBatch::add()` and `CallContext::RDMATransmission::add()` overloads for `RDMABufUnified` ### StorageClient - `IOBuffer` wraps `RDMABufUnified` instead of plain `RDMABuf` - Inline data transfer automatically disabled for GPU buffers - Client-side CPU checksum automatically skipped for GPU buffers - `registerGpuIOBuffer()` with fallback to `RDMABuf::createFromUserBuffer` ### Fuse Daemon - `IovTable` parses `.gdr.d{N}` symlink keys and `gdr://v1/device/{id}/size/{size}/ipc/{hex}` URIs - `GpuShmBuf`/`GpuShmBufForIO` for daemon-side imported GPU buffer management - `IoBufForIO` = `std::variant<ShmBufForIO, GpuShmBufForIO>` with `std::visit` dispatch in `IoRing::process()` - `lookupBufs` two-phase locking: `shmLock` then `gpuShmLock` (never nested) ### Build & Infra - `cmake/Cuda.cmake` for CUDA toolkit detection and `HF3FS_GDR_ENABLED` flag - CI workflow updated for optional GDR build matrix ### Docs - `docs/gdr-integration.md`: user-facing guide with prerequisites, API reference, 3 complete usage examples, runtime behavior, and known limitations --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 09:50:06 +00:00

bsakaguchi referenced this issue

2026-10-26 09:50:09 +00:00

[PR #420] fix(packaging): remove git dependency from package version #425