deepseek-ai/3FS

Fork 0

mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00

[GH-ISSUE #137] How to scale up a 3FS cluster #76

New issue

Closed

opened 2026-10-26 09:44:39 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-26 09:44:39 +00:00

Owner

Originally created by @huww98 on GitHub (Mar 10, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/137

Thanks for this awesome project. We have successfully deployed a small-scale experimental cluster on Alibaba Cloud.

Now I want to test how it scales up. But there seems to be no document about how to add additional storage node to the cluster.

Here is my guess based on the very rough understanding to 3FS:

Option 1: I see there is a Layout-ChainTableVersion field in the output of stat --display-layout. So, it seems we can create a completely disjoint chain table and upload it to the cluster as version 2, which covers both old and new nodes. But it seems we cannot migrate existing data to the new version transparently.

Option 2: Maybe we can gradually move the existing chain. By adding an additional target to a chain to allow it to sync data, then remove one of the old targets from the chain. But the number of chains is fixed. As we add more nodes, we will need more chains to distribute the data. Can we somehow split an existing chain into two? Like the placement group of CEPH.

How do you scale up the cluster in DeepSeek?

Thanks again!

Originally created by @huww98 on GitHub (Mar 10, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/137 Thanks for this awesome project. We have successfully deployed a small-scale experimental cluster on Alibaba Cloud. Now I want to test how it scales up. But there seems to be no document about how to add additional storage node to the cluster. Here is my guess based on the very rough understanding to 3FS: Option 1: I see there is a `Layout-ChainTableVersion` field in the output of `stat --display-layout`. So, it seems we can create a completely disjoint chain table and upload it to the cluster as version 2, which covers both old and new nodes. But it seems we cannot migrate existing data to the new version transparently. Option 2: Maybe we can gradually move the existing chain. By adding an additional target to a chain to allow it to sync data, then remove one of the old targets from the chain. But the number of chains is fixed. As we add more nodes, we will need more chains to distribute the data. Can we somehow split an existing chain into two? Like the placement group of CEPH. How do you scale up the cluster in DeepSeek? Thanks again!

bsakaguchi closed this issue

2026-10-26 09:44:39 +00:00

bsakaguchi commented

2026-10-26 09:44:41 +00:00

Author

Owner

@SF-Zhou commented on GitHub (Mar 13, 2025):

In fact, we are using both Option 1 and Option 2.

@SF-Zhou commented on GitHub (Mar 13, 2025): In fact, we are using both Option 1 and Option 2.

bsakaguchi commented

2026-10-26 09:44:42 +00:00

Author

Owner

@zk-kiger commented on GitHub (Mar 18, 2025):

Can you elaborate on how to scale up clusters when node reach the sdd capacity threshold?

@zk-kiger commented on GitHub (Mar 18, 2025): Can you elaborate on how to scale up clusters when node reach the sdd capacity threshold?

bsakaguchi commented

2026-10-26 09:44:42 +00:00

Author

Owner

@huww98 commented on GitHub (Mar 18, 2025):

From my guess:

When adding new node, we can add new then remove old target from chains, with update-chain command. If we don't have enough chains to keep nodes balanced, we need to create a new ChainTable with more chains (maybe 4x of the old one, from the experience of ceph pg). Then move all the data once to refresh the layout of existing files, maybe just cp and rm with fuse client.

Note that the migration to new ChainTable seems not transparent to user. e.g. If the user writes to the old file while we are copying it, the changes may be lost. I'm not satisfied with this, but I cannot come up with an easy solution. 3FS picks a subset of chain table for each file, which means we cannot just split a chain into two without change the stripe-size of each file. In contrast, ceph spread the data of every file in the whole pool.

Besides, it seems we cannot remove chains/targets even if we have migrated all the files in it. This is frustrating.

@huww98 commented on GitHub (Mar 18, 2025): From my guess: When adding new node, we can add new then remove old target from chains, with `update-chain` command. If we don't have enough chains to keep nodes balanced, we need to create a new ChainTable with more chains (maybe 4x of the old one, from the experience of ceph pg). Then move all the data once to refresh the layout of existing files, maybe just `cp` and `rm` with fuse client. Note that the migration to new ChainTable seems not transparent to user. e.g. If the user writes to the old file while we are copying it, the changes may be lost. I'm not satisfied with this, but I cannot come up with an easy solution. 3FS picks a subset of chain table for each file, which means we cannot just split a chain into two without change the stripe-size of each file. In contrast, ceph spread the data of every file in the whole pool. Besides, it seems we cannot remove chains/targets even if we have migrated all the files in it. This is frustrating.

bsakaguchi commented

2026-10-26 09:44:42 +00:00

Author

Owner

@shuiyihang commented on GitHub (May 7, 2025):

From my guess:

When adding new node, we can add new then remove old target from chains, with update-chain command. If we don't have enough chains to keep nodes balanced, we need to create a new ChainTable with more chains (maybe 4x of the old one, from the experience of ceph pg). Then move all the data once to refresh the layout of existing files, maybe just cp and rm with fuse client.

Note that the migration to new ChainTable seems not transparent to user. e.g. If the user writes to the old file while we are copying it, the changes may be lost. I'm not satisfied with this, but I cannot come up with an easy solution. 3FS picks a subset of chain table for each file, which means we cannot just split a chain into two without change the stripe-size of each file. In contrast, ceph spread the data of every file in the whole pool.

Besides, it seems we cannot remove chains/targets even if we have migrated all the files in it. This is frustrating.

我看他创建文件的时候可以指定layout（默认没有指定，继承自父目录，主要是指定三个参数：tableId，chunkSize，stripeSize），当添加新节点的时候，创建新的表。指定使用新的表，这样是否以

@shuiyihang commented on GitHub (May 7, 2025): > From my guess: > > When adding new node, we can add new then remove old target from chains, with `update-chain` command. If we don't have enough chains to keep nodes balanced, we need to create a new ChainTable with more chains (maybe 4x of the old one, from the experience of ceph pg). Then move all the data once to refresh the layout of existing files, maybe just `cp` and `rm` with fuse client. > > Note that the migration to new ChainTable seems not transparent to user. e.g. If the user writes to the old file while we are copying it, the changes may be lost. I'm not satisfied with this, but I cannot come up with an easy solution. 3FS picks a subset of chain table for each file, which means we cannot just split a chain into two without change the stripe-size of each file. In contrast, ceph spread the data of every file in the whole pool. > > Besides, it seems we cannot remove chains/targets even if we have migrated all the files in it. This is frustrating. 我看他创建文件的时候可以指定layout（默认没有指定，继承自父目录，主要是指定三个参数：tableId，chunkSize，stripeSize），当添加新节点的时候，创建新的表。指定使用新的表，这样是否以

bsakaguchi commented

2026-10-26 09:44:42 +00:00

Author

Owner

@xlcbingo1999 commented on GitHub (Jul 23, 2025):

Currently, no matter which scale-up method is used, scale-up can only be performed manually. Will 3FS consider developing an official scale-up automation script?

@xlcbingo1999 commented on GitHub (Jul 23, 2025): Currently, no matter which scale-up method is used, scale-up can only be performed manually. Will 3FS consider developing an official scale-up automation script?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#76

No description provided.

Rows
Columns