deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #195] 3fs fuse mount size wrong for two replications? #123

New issue
Closed
opened 2026-10-26 09:45:33 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:45:33 +00:00
Owner
Copy link

Originally created by @Icedroid on GitHub (Mar 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/195

# python3 /opt/3fs/deploy/data_placement/src/model/data_placement.py    -ql -relax -type CR --num_nodes 2 --replication_factor 2 --min_targets_per_disk 6
2025-03-19 14:16:21.398 | INFO     | __main__:run:125 - solving model with appsi_highs #0: DataPlacementModel-v=2,b=6,r=6,k=2,λ=6,lb=1,ub=0
2025-03-19 14:16:21.398 | INFO     | __main__:build_model:182 - self.num_nodes=2 self.num_targets_per_disk=6 self.group_size=2 self.num_groups=6 self.qlinearize=True self.relax_lb=1 self.relax_ub=0
2025-03-19 14:16:21.398 | INFO     | __main__:build_model:192 - self.sum_recovery_traffic_per_failure=6 self.max_recovery_traffic_on_peer=6
2025-03-19 14:16:21.398 | INFO     | __main__:build_model:196 - self.all_targets_used=True self.balanced_peer_traffic=True
2025-03-19 14:16:21.398 | INFO     | __main__:build_model:197 - self.num_targets_used=12 self.num_targets_total=12
2025-03-19 14:16:21.399 | INFO     | __main__:build_model:272 - lower bound imposed on peer traffic: self.relax_lb=1 self.qlinearize=True self.all_targets_used=True
Running HiGHS 1.8.0 (git hash: eda5cbe): Copyright (c) 2024 HiGHS under MIT licence terms
0 rows, 0 cols, 0 nonzeros  0s
Presolve: Optimal
  Cost   [0e+00, 0e+00]
  Bound  [1e+00, 1e+00]
  RHS    [1e+00, 6e+00]
Coefficient ranges:
  Matrix [1e+00, 1e+00]
Presolving model
0 rows, 0 cols, 0 nonzeros  0s
 
        Nodes      |    B&B Tree     |            Objective Bounds              |  Dynamic Constraints |       Work
     Proc. InQueue |  Leaves   Expl. | BestBound       BestSol              Gap |   Cuts   InLp Confl. | LpIters     Time
 
         0       0         0   0.00%   1               1                  0.00%        0      0      0         0     0.0s
 
Solving report
  Status            Optimal
  Primal bound      1
  Dual bound        1
  Gap               0% (tolerance: 0.01%)
  Solution status   feasible
                    1 (objective)
                    0 (bound viol.)
                    0 (int. viol.)
                    0 (row viol.)
  Timing            0.00 (total)
                    0.00 (presolve)
                    0.00 (postsolve)
  Nodes             0
  LP iterations     0 (total)
                    0 (strong br.)
                    0 (separation)
                    0 (heuristics)
2025-03-19 14:16:21.409 | SUCCESS  | __main__:solve:165 - optimal solution:
- Status: ok
  Termination condition: optimal
  Termination message: TerminationCondition.optimal
 
2025-03-19 14:16:21.409 | DEBUG    | __main__:check_solution:322 - 1,2: 6.0
2025-03-19 14:16:21.409 | DEBUG    | __main__:check_solution:322 - 2,1: 6.0
2025-03-19 14:16:21.410 | INFO     | __main__:check_solution:331 - min_peer_traffic=6.0 max_peer_traffic=6.0
2025-03-19 14:16:21.410 | INFO     | __main__:check_solution:332 - total_traffic=12.0 max_total_traffic=12
2025-03-19 14:16:21.535 | SUCCESS  | __main__:run:148 - saved solution to: output/DataPlacementModel-v_2-b_6-r_6-k_2-λ_6-lb_1-ub_0

Two storage node. every node has 12 disks, one disk is 7T.
I use hf3fs_use to mount , Why the size equal the all disks capacity. If two replicas, size should be 7*12=84T.
Is it my misunderstanding for replication_factor=2?

# df -h
Filesystem      Size  Used Avail Use% Mounted on
hf3fs.stage     168T  1.3T  167T   1% /3fs/stage
Originally created by @Icedroid on GitHub (Mar 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/195 ``` # python3 /opt/3fs/deploy/data_placement/src/model/data_placement.py -ql -relax -type CR --num_nodes 2 --replication_factor 2 --min_targets_per_disk 6 2025-03-19 14:16:21.398 | INFO | __main__:run:125 - solving model with appsi_highs #0: DataPlacementModel-v=2,b=6,r=6,k=2,λ=6,lb=1,ub=0 2025-03-19 14:16:21.398 | INFO | __main__:build_model:182 - self.num_nodes=2 self.num_targets_per_disk=6 self.group_size=2 self.num_groups=6 self.qlinearize=True self.relax_lb=1 self.relax_ub=0 2025-03-19 14:16:21.398 | INFO | __main__:build_model:192 - self.sum_recovery_traffic_per_failure=6 self.max_recovery_traffic_on_peer=6 2025-03-19 14:16:21.398 | INFO | __main__:build_model:196 - self.all_targets_used=True self.balanced_peer_traffic=True 2025-03-19 14:16:21.398 | INFO | __main__:build_model:197 - self.num_targets_used=12 self.num_targets_total=12 2025-03-19 14:16:21.399 | INFO | __main__:build_model:272 - lower bound imposed on peer traffic: self.relax_lb=1 self.qlinearize=True self.all_targets_used=True Running HiGHS 1.8.0 (git hash: eda5cbe): Copyright (c) 2024 HiGHS under MIT licence terms 0 rows, 0 cols, 0 nonzeros 0s Presolve: Optimal Cost [0e+00, 0e+00] Bound [1e+00, 1e+00] RHS [1e+00, 6e+00] Coefficient ranges: Matrix [1e+00, 1e+00] Presolving model 0 rows, 0 cols, 0 nonzeros 0s Nodes | B&B Tree | Objective Bounds | Dynamic Constraints | Work Proc. InQueue | Leaves Expl. | BestBound BestSol Gap | Cuts InLp Confl. | LpIters Time 0 0 0 0.00% 1 1 0.00% 0 0 0 0 0.0s Solving report Status Optimal Primal bound 1 Dual bound 1 Gap 0% (tolerance: 0.01%) Solution status feasible 1 (objective) 0 (bound viol.) 0 (int. viol.) 0 (row viol.) Timing 0.00 (total) 0.00 (presolve) 0.00 (postsolve) Nodes 0 LP iterations 0 (total) 0 (strong br.) 0 (separation) 0 (heuristics) 2025-03-19 14:16:21.409 | SUCCESS | __main__:solve:165 - optimal solution: - Status: ok Termination condition: optimal Termination message: TerminationCondition.optimal 2025-03-19 14:16:21.409 | DEBUG | __main__:check_solution:322 - 1,2: 6.0 2025-03-19 14:16:21.409 | DEBUG | __main__:check_solution:322 - 2,1: 6.0 2025-03-19 14:16:21.410 | INFO | __main__:check_solution:331 - min_peer_traffic=6.0 max_peer_traffic=6.0 2025-03-19 14:16:21.410 | INFO | __main__:check_solution:332 - total_traffic=12.0 max_total_traffic=12 2025-03-19 14:16:21.535 | SUCCESS | __main__:run:148 - saved solution to: output/DataPlacementModel-v_2-b_6-r_6-k_2-λ_6-lb_1-ub_0 ``` Two storage node. every node has 12 disks, one disk is 7T. I use hf3fs_use to mount , Why the size equal the all disks capacity. If two replicas, size should be 7*12=84T. Is it my misunderstanding for replication_factor=2? ``` # df -h Filesystem Size Used Avail Use% Mounted on hf3fs.stage 168T 1.3T 167T 1% /3fs/stage ```
bsakaguchi commented 2026-10-26 09:45:34 +00:00
Author
Owner
Copy link

@SF-Zhou commented on GitHub (Mar 20, 2025):

Because 3FS supports the coexistence of multiple replica counts, we cannot definitively determine the exact number of replicas. Therefore, when calculating total disk space usage, the system aggregates storage consumption without considering replica configurations. For instance, a 1MiB file with 2 replicas would be reported as occupying 2MiB of disk space in this statistical method.

<!-- gh-comment-id:2739655471 --> @SF-Zhou commented on GitHub (Mar 20, 2025): Because 3FS supports the coexistence of multiple replica counts, we cannot definitively determine the exact number of replicas. Therefore, when calculating total disk space usage, the system aggregates storage consumption without considering replica configurations. For instance, a 1MiB file with 2 replicas would be reported as occupying 2MiB of disk space in this statistical method.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#123
No description provided.