[GH-ISSUE #146] 元数据性能问题 #83

New issue

Closed

opened 2026-10-26 09:44:47 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:44:47 +00:00

Owner

Originally created by @StewieeChen on GitHub (Mar 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/146

我在自己的集群搭建了一套3fs集群，其中fdb部署在一块nvme盘上。但是mdtest测下来的性能只有几百，这种情况是否正常，有没有办法提高我的元数据性能。

这个性能符合预期么？

Originally created by @StewieeChen on GitHub (Mar 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/146 ![Image](https://github.com/user-attachments/assets/9199e6ec-cf11-4ac0-835c-7ab257aef7f1) 我在自己的集群搭建了一套3fs集群，其中fdb部署在一块nvme盘上。但是mdtest测下来的性能只有几百，这种情况是否正常，有没有办法提高我的元数据性能。这个性能符合预期么？

bsakaguchi closed this issue

2026-10-26 09:44:48 +00:00

bsakaguchi commented

2026-10-26 09:44:49 +00:00

Author

Owner

@yuchengkang commented on GitHub (Mar 11, 2025):

同为NVMe

@yuchengkang commented on GitHub (Mar 11, 2025): ![Image](https://github.com/user-attachments/assets/cc976028-1f99-4ce6-8563-3e3c9b7a198e) 同为NVMe

bsakaguchi commented

2026-10-26 09:44:50 +00:00

Author

Owner

@StewieeChen commented on GitHub (Mar 11, 2025):

同为NVMe

您好，请问一下，您这边具体的配置信息呢？

多少个元数据节点，以及对应的foundationdb的规模呢？以及对应NVMe的型号

fuse对应配置(类似于entry_timeout)是多少，然后给client预留了多少内存呢？以及给client使用的CPU具体型号呢？

@StewieeChen commented on GitHub (Mar 11, 2025): > ![Image](https://github.com/user-attachments/assets/cc976028-1f99-4ce6-8563-3e3c9b7a198e) > > 同为NVMe 您好，请问一下，您这边具体的配置信息呢？多少个元数据节点，以及对应的foundationdb的规模呢？以及对应NVMe的型号 fuse对应配置(类似于entry_timeout)是多少，然后给client预留了多少内存呢？以及给client使用的CPU具体型号呢？

bsakaguchi commented

2026-10-26 09:44:50 +00:00

Author

Owner

@wangyibin-gh commented on GitHub (Mar 13, 2025):

meta ops is not a saling point for 3fs. 3fs is designed with the goal of fast bulk data transfer and infrequent metadata ops. FoundationDB's SSI mechanism essentially serialize the meta ops which definitely isn't a wise way to get great meta performance. but is is a comprimise. as we all know, meta ops has always been an issue in distributed storage, it's very hard to get a solid solution quickly. so why bother investing so much energy in doing this for a sceario that won't necessarily requires a lot of meta ops?

yes, for ai training, there is a lot of small files. but the good thing is that they are mostly read-only. so deepseek invented their own dataset format ffrecord, which is quite straightforward design which combine huge amount of small files into one big file, which works well in 3fs. with the help of dataset/dataloader, the meta is retrieved in a few read IO, and subsequent read can be done once the shuffle order is determined. if you are looking for some filesystem with super fast metadata ops, 3fs isn't what you want to look into.

@wangyibin-gh commented on GitHub (Mar 13, 2025): meta ops is not a saling point for 3fs. 3fs is designed with the goal of fast bulk data transfer and infrequent metadata ops. FoundationDB's SSI mechanism essentially serialize the meta ops which definitely isn't a wise way to get great meta performance. but is is a comprimise. as we all know, meta ops has always been an issue in distributed storage, it's very hard to get a solid solution quickly. so why bother investing so much energy in doing this for a sceario that won't necessarily requires a lot of meta ops? yes, for ai training, there is a lot of small files. but the good thing is that they are mostly read-only. so deepseek invented their own dataset format ffrecord, which is quite straightforward design which combine huge amount of small files into one big file, which works well in 3fs. with the help of dataset/dataloader, the meta is retrieved in a few read IO, and subsequent read can be done once the shuffle order is determined. if you are looking for some filesystem with super fast metadata ops, 3fs isn't what you want to look into.

bsakaguchi commented

2026-10-26 09:44:50 +00:00

Author

Owner

@StewieeChen commented on GitHub (Mar 14, 2025):

meta ops is not a saling point for 3fs. 3fs is designed with the goal of fast bulk data transfer and infrequent metadata ops. FoundationDB's SSI mechanism essentially serialize the meta ops which definitely isn't a wise way to get great meta performance. but is is a comprimise. as we all know, meta ops has always been an issue in distributed storage, it's very hard to get a solid solution quickly. so why bother investing so much energy in doing this for a sceario that won't necessarily requires a lot of meta ops?

yes, for ai training, there is a lot of small files. but the good thing is that they are mostly read-only. so deepseek invented their own dataset format ffrecord, which is quite straightforward design which combine huge amount of small files into one big file, which works well in 3fs. with the help of dataset/dataloader, the meta is retrieved in a few read IO, and subsequent read can be done once the shuffle order is determined. if you are looking for some filesystem with super fast metadata ops, 3fs isn't what you want to look into.

thanks

@StewieeChen commented on GitHub (Mar 14, 2025): > meta ops is not a saling point for 3fs. 3fs is designed with the goal of fast bulk data transfer and infrequent metadata ops. FoundationDB's SSI mechanism essentially serialize the meta ops which definitely isn't a wise way to get great meta performance. but is is a comprimise. as we all know, meta ops has always been an issue in distributed storage, it's very hard to get a solid solution quickly. so why bother investing so much energy in doing this for a sceario that won't necessarily requires a lot of meta ops? > > yes, for ai training, there is a lot of small files. but the good thing is that they are mostly read-only. so deepseek invented their own dataset format ffrecord, which is quite straightforward design which combine huge amount of small files into one big file, which works well in 3fs. with the help of dataset/dataloader, the meta is retrieved in a few read IO, and subsequent read can be done once the shuffle order is determined. if you are looking for some filesystem with super fast metadata ops, 3fs isn't what you want to look into. thanks

bsakaguchi referenced this issue

2026-10-26 09:48:16 +00:00

[PR #83] fuse: don't limit max_threads to num of CPU cores #284