deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #286] How to build a cluster with multiple mgmtd nodes? #174

New issue
Closed
opened 2026-10-26 09:46:34 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 09:46:34 +00:00
Owner
Copy link

Originally created by @csyanxing on GitHub (May 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/286

In order to avoid the problem single point failure at mgmtd node, we want to build a cluster with 3 mgmtd nodes. I have tried to init-cluster and start mgmtd service on three different machines, and in the meta service config file we have fill the field "mgmtd_server_addresses" with 3 address of the mgmtd nodes. However, after meta service starts, the output of "list-nodes" command only contains one mgmtd node (PRIMARY_MGMTD) and 3 meta nodes. And when the mgmtd node fails over, the service could not be automatically changed to other mgmtd nodes.

So, how to build a cluster with multiple mgmtd nodes? Does there exist some config file or some special command for this?

Originally created by @csyanxing on GitHub (May 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/286 In order to avoid the problem single point failure at mgmtd node, we want to build a cluster with 3 mgmtd nodes. I have tried to init-cluster and start mgmtd service on three different machines, and in the meta service config file we have fill the field "mgmtd_server_addresses" with 3 address of the mgmtd nodes. However, after meta service starts, the output of "list-nodes" command only contains one mgmtd node (PRIMARY_MGMTD) and 3 meta nodes. And when the mgmtd node fails over, the service could not be automatically changed to other mgmtd nodes. So, how to build a cluster with multiple mgmtd nodes? Does there exist some config file or some special command for this?
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@huww98 commented on GitHub (May 21, 2025):

Do you have all your mgmtd connected to the same FDB cluster?

<!-- gh-comment-id:2897637099 --> @huww98 commented on GitHub (May 21, 2025): Do you have all your mgmtd connected to the same FDB cluster?
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@csyanxing commented on GitHub (May 22, 2025):

Do you have all your mgmtd connected to the same FDB cluster?

No, we use 3 different machines as mgmtd and they all use their local FoundationDB service. We will let the 3 mgmtd nodes connect to the same FDB and try again. Thanks for your remind!

<!-- gh-comment-id:2900438821 --> @csyanxing commented on GitHub (May 22, 2025): > Do you have all your mgmtd connected to the same FDB cluster? No, we use 3 different machines as mgmtd and they all use their local FoundationDB service. We will let the 3 mgmtd nodes connect to the same FDB and try again. Thanks for your remind!
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@csyanxing commented on GitHub (May 22, 2025):

Do you have all your mgmtd connected to the same FDB cluster?

And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant:

  • multiple FDB server which coordinate as an FDB cluster
  • multiple mgmtd nodes which connect to the same FDB cluster
  • multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address)

Is this statement correct?

<!-- gh-comment-id:2900467868 --> @csyanxing commented on GitHub (May 22, 2025): > Do you have all your mgmtd connected to the same FDB cluster? And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant: - multiple FDB server which coordinate as an FDB cluster - multiple mgmtd nodes which connect to the same FDB cluster - multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address) Is this statement correct?
bsakaguchi commented 2026-10-26 09:46:36 +00:00
Author
Owner
Copy link

@diabloneo commented on GitHub (May 22, 2025):

Do you have all your mgmtd connected to the same FDB cluster?

And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant:

  • multiple FDB server which coordinate as an FDB cluster
  • multiple mgmtd nodes which connect to the same FDB cluster
  • multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address)

Is this statement correct?

Yes，this is correct.

You can use https://github.com/open3fs/m3fs to deploy a cluster with a FDB cluster with multiple instances and multiple mgmtd nodes conveniently.

<!-- gh-comment-id:2901611649 --> @diabloneo commented on GitHub (May 22, 2025): > > Do you have all your mgmtd connected to the same FDB cluster? > > And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant: > > * multiple FDB server which coordinate as an FDB cluster > * multiple mgmtd nodes which connect to the same FDB cluster > * multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address) > > Is this statement correct? Yes，this is correct. You can use https://github.com/open3fs/m3fs to deploy a cluster with a FDB cluster with multiple instances and multiple mgmtd nodes conveniently.
bsakaguchi commented 2026-10-26 09:46:37 +00:00
Author
Owner
Copy link

@csyanxing commented on GitHub (May 22, 2025):

Do you have all your mgmtd connected to the same FDB cluster?

And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant:

  • multiple FDB server which coordinate as an FDB cluster
  • multiple mgmtd nodes which connect to the same FDB cluster
  • multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address)

Is this statement correct?

Yes，this is correct.

You can use https://github.com/open3fs/m3fs to deploy a cluster with a FDB cluster with multiple instances and multiple mgmtd nodes conveniently.

Thanks for your response! And m3fs is really a nice tool to simplify the deployment of 3fs, thanks for your sharing!

<!-- gh-comment-id:2901693375 --> @csyanxing commented on GitHub (May 22, 2025): > > > Do you have all your mgmtd connected to the same FDB cluster? > > > > > > And from our superficial understanding of 3fs, in order to minimise the risk of single point failure, maybe we need to deploy the following service as redundant: > > > > * multiple FDB server which coordinate as an FDB cluster > > * multiple mgmtd nodes which connect to the same FDB cluster > > * multiple meta nodes which connect to the same mgmtd server cluster (this can be implemented by fill the "mgmtd_server_addresses" config with multiple mgmtd address) > > > > Is this statement correct? > > Yes，this is correct. > > You can use https://github.com/open3fs/m3fs to deploy a cluster with a FDB cluster with multiple instances and multiple mgmtd nodes conveniently. Thanks for your response! And m3fs is really a nice tool to simplify the deployment of 3fs, thanks for your sharing!
bsakaguchi commented 2026-10-26 09:46:37 +00:00
Author
Owner
Copy link

@huww98 commented on GitHub (May 22, 2025):

If you are familiar with Kubernetes, maybe you will be interested in our Helm chart approach. It may need some tweaks to work on your environment, though.

<!-- gh-comment-id:2901782194 --> @huww98 commented on GitHub (May 22, 2025): If you are familiar with Kubernetes, maybe you will be interested in our [Helm chart approach](https://github.com/AliyunContainerService/alibabacloud-3FS/blob/main/deploy/container/README.md). It may need some tweaks to work on your environment, though.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#174
No description provided.