deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #223] Why checkNicType function treat "ib" prefix as Address:IPoIB ? #137

New issue
Closed
opened 2026-10-26 09:45:48 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-26 09:45:48 +00:00
Owner
Copy link

Originally created by @diabloneo on GitHub (Mar 30, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/223

In the checkNicType function (github.com/deepseek-ai/3FS@dfadd59868/src/common/net/Listener.cc (L33))

Nic name starts with ib has been mapped to IPoIB address type:

static bool checkNicType(std::string_view nic, Address::Type type) {
  switch (type) {
    case Address::TCP:
      return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe");
    case Address::IPoIB:
      return nic.starts_with("ib");
    case Address::RDMA:
      return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe");
    case Address::LOCAL:
      return nic.starts_with("lo");
    default:
      return false;
  }
}

This condition causes the nic named like "ib0" to be skipped in Listener::setup() for RDMA group:

  for (auto [name, addr] : nics.value()) {
    if (addr.up && (filters.empty() || filters.count(name) != 0) && checkNicType(name, networkType_)) {
      addressList_.push_back(Address{addr.ip.toLong(),
                                     config_.listen_port(),
                                     networkType_ == Address::LOCAL ? Address::TCP : networkType_});
    }
  }

Is there any reason for doing this?

Originally created by @diabloneo on GitHub (Mar 30, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/223 In the `checkNicType` function (https://github.com/deepseek-ai/3FS/blob/dfadd59868954fe016affc6bfac506864077026c/src/common/net/Listener.cc#L33) Nic name starts with **ib** has been mapped to `IPoIB` address type: ``` static bool checkNicType(std::string_view nic, Address::Type type) { switch (type) { case Address::TCP: return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe"); case Address::IPoIB: return nic.starts_with("ib"); case Address::RDMA: return nic.starts_with("en") || nic.starts_with("eth") || nic.starts_with("bond") || nic.starts_with("xgbe"); case Address::LOCAL: return nic.starts_with("lo"); default: return false; } } ``` This condition causes the nic named like "ib0" to be skipped in `Listener::setup()` for RDMA group: ``` for (auto [name, addr] : nics.value()) { if (addr.up && (filters.empty() || filters.count(name) != 0) && checkNicType(name, networkType_)) { addressList_.push_back(Address{addr.ip.toLong(), config_.listen_port(), networkType_ == Address::LOCAL ? Address::TCP : networkType_}); } } ``` Is there any reason for doing this?
bsakaguchi commented 2026-10-26 09:45:49 +00:00
Author
Owner
Copy link

@diabloneo commented on GitHub (Mar 30, 2025):

https://github.com/deepseek-ai/3FS/issues/156

I found a similar issue.

Does this mean that: if I want to use ib nic, I have to setup ipoib, then use IPoIB:// prefix in the configuration?

<!-- gh-comment-id:2764507485 --> @diabloneo commented on GitHub (Mar 30, 2025): https://github.com/deepseek-ai/3FS/issues/156 I found a similar issue. Does this mean that: if I want to use ib nic, I have to setup ipoib, then use `IPoIB://` prefix in the configuration?
bsakaguchi commented 2026-10-26 09:45:49 +00:00
Author
Owner
Copy link

@SF-Zhou commented on GitHub (Mar 31, 2025):

Establishing an RDMA connection actually requires both a TCP network card and an RDMA network card, and RDMA does not need to be in IPoIB mode.

<!-- gh-comment-id:2764826002 --> @SF-Zhou commented on GitHub (Mar 31, 2025): Establishing an RDMA connection actually requires both a TCP network card and an RDMA network card, and RDMA does not need to be in IPoIB mode.
bsakaguchi commented 2026-10-26 09:45:49 +00:00
Author
Owner
Copy link

@diabloneo commented on GitHub (Apr 1, 2025):

I solved the problem to a certain extent. I just quote the conclusion here:

After digging the 3FS code and made some experiments, I and @feeyman finally deployed a 3FS cluster on a IB enviroment.

Here are some conclusions:

The mgmtd servivce's network_type must be set to IPoIB to let the mgmtd service listening on the IB interface.
Other services' (except monitor) network_type still use the value RDMA.
The monitor service didn't work on the environment's IB interface. It's can be deployed on the other IP address on ethernet interface.
I published https://github.com/open3fs/m3fs/releases/tag/v0.1.7 to solve this problem. If you want to deploy 3FS on a IB environment, just set networkType: IB in the cluster.yml.

Visit https://github.com/open3fs/m3fs/issues/88 for more information.

<!-- gh-comment-id:2767804278 --> @diabloneo commented on GitHub (Apr 1, 2025): I solved the problem to a certain extent. I just quote the conclusion here: > After digging the 3FS code and made some experiments, I and @feeyman finally deployed a 3FS cluster on a IB enviroment. > > Here are some conclusions: > > The mgmtd servivce's network_type must be set to IPoIB to let the mgmtd service listening on the IB interface. > Other services' (except monitor) network_type still use the value RDMA. > The monitor service didn't work on the environment's IB interface. It's can be deployed on the other IP address on ethernet interface. > I published https://github.com/open3fs/m3fs/releases/tag/v0.1.7 to solve this problem. If you want to deploy 3FS on a IB environment, just set networkType: IB in the cluster.yml. Visit https://github.com/open3fs/m3fs/issues/88 for more information.
bsakaguchi commented 2026-10-26 09:45:49 +00:00
Author
Owner
Copy link

@Navacore commented on GitHub (Apr 20, 2025):

I don't know what they're accusing me of, please stop sending messages.

El dom, 30 de mar. de 2025 7:13 a. m., Neo Liu @.***>
escribió:

#156 https://github.com/deepseek-ai/3FS/issues/156

I found a similar issue.

Does this mean that: if I want to use ib nic, I have to setup ipoib, then
use IPoIB:// prefix in the configuration?

—
Reply to this email directly, view it on GitHub
https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQ43KJNP2EYKOPNQWKAJZYD2W7GVVAVCNFSM6AAAAAB2CH2W7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGUYDONBYGU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>
[image: diabloneo]diabloneo left a comment (deepseek-ai/3FS#223)
https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485

#156 https://github.com/deepseek-ai/3FS/issues/156

I found a similar issue.

Does this mean that: if I want to use ib nic, I have to setup ipoib, then
use IPoIB:// prefix in the configuration?

—
Reply to this email directly, view it on GitHub
https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQ43KJNP2EYKOPNQWKAJZYD2W7GVVAVCNFSM6AAAAAB2CH2W7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGUYDONBYGU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

<!-- gh-comment-id:2817383740 --> @Navacore commented on GitHub (Apr 20, 2025): I don't know what they're accusing me of, please stop sending messages. El dom, 30 de mar. de 2025 7:13 a. m., Neo Liu ***@***.***> escribió: > #156 <https://github.com/deepseek-ai/3FS/issues/156> > > I found a similar issue. > > Does this mean that: if I want to use ib nic, I have to setup ipoib, then > use IPoIB:// prefix in the configuration? > > — > Reply to this email directly, view it on GitHub > <https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQ43KJNP2EYKOPNQWKAJZYD2W7GVVAVCNFSM6AAAAAB2CH2W7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGUYDONBYGU> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> > [image: diabloneo]*diabloneo* left a comment (deepseek-ai/3FS#223) > <https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485> > > #156 <https://github.com/deepseek-ai/3FS/issues/156> > > I found a similar issue. > > Does this mean that: if I want to use ib nic, I have to setup ipoib, then > use IPoIB:// prefix in the configuration? > > — > Reply to this email directly, view it on GitHub > <https://github.com/deepseek-ai/3FS/issues/223#issuecomment-2764507485>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQ43KJNP2EYKOPNQWKAJZYD2W7GVVAVCNFSM6AAAAAB2CH2W7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRUGUYDONBYGU> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#137
No description provided.