[GH-ISSUE #3] Direct transfer to gpu memory #3

New issue

Closed

opened 2026-10-26 09:43:13 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:43:13 +00:00

Owner

Originally created by @msharmavikram on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/3

Hello there,
Awesome project! Thanks for open sourcing!

It is unclear from the documentation if 3FS supports direct transfer from storage to gpu memory without cpu buffer. Could you share some insight here ?

Originally created by @msharmavikram on GitHub (Feb 28, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/3 Hello there, Awesome project! Thanks for open sourcing! It is unclear from the documentation if 3FS supports direct transfer from storage to gpu memory without cpu buffer. Could you share some insight here ?

bsakaguchi closed this issue

2026-10-26 09:43:13 +00:00

bsakaguchi commented

2026-10-26 09:43:13 +00:00

Author

Owner

@echaozh commented on GitHub (Feb 28, 2025):

The training input and the KVCache for inference are often read in the background when the previous batch is still being computed. The GPU memory is already loaded at the moment, so directly transferring into it is not very useful.

On the other hand, transferring directly from the GPU memory is also not very useful as there are multiple GPUs and the total d2h bandwidth is much larger than the storage NIC capability, and the transfer to the disk may delay the discard of the data on GPU .

@echaozh commented on GitHub (Feb 28, 2025): The training input and the KVCache for inference are often read in the background when the previous batch is still being computed. The GPU memory is already loaded at the moment, so directly transferring into it is not very useful. On the other hand, transferring directly from the GPU memory is also not very useful as there are multiple GPUs and the total d2h bandwidth is much larger than the storage NIC capability, and the transfer to the disk may delay the discard of the data on GPU .

bsakaguchi commented

2026-10-26 09:43:14 +00:00

Author

Owner

@zeronewb commented on GitHub (Feb 28, 2025):

Well, gds really useful, when you are dealing with huge ammount of media data, like mri scans or gene sequences. Typical deployment scenario: dgx like nodes with 12xNDR nics and single ib fabric, storage nodes also with multiple NDR nics.
Right now we have ovepriced gpfs/weka/vast or lustre with ton of quirks.
Hope that will change with time.
Anyway, thanks for the clarifying your position.
Cheers!

@zeronewb commented on GitHub (Feb 28, 2025): Well, gds really useful, when you are dealing with huge ammount of media data, like mri scans or gene sequences. Typical deployment scenario: dgx like nodes with 12xNDR nics and single ib fabric, storage nodes also with multiple NDR nics. Right now we have ovepriced gpfs/weka/vast or lustre with ton of quirks. Hope that will change with time. Anyway, thanks for the clarifying your position. Cheers!

bsakaguchi referenced this issue

2026-10-26 09:44:43 +00:00

[GH-ISSUE #142] meta_main start failed ,the meta_main service coredump #79