[GH-ISSUE #418] [RFC] IPv6 / dual-stack support (TCP control plane + RDMA data plane) #241

New issue

Open

opened 2026-10-26 09:47:41 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:47:41 +00:00

Owner

Originally created by @ShenHuijun on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/418

Motivation

Some RDMA fabrics are IPv6-only: high-bandwidth NICs get ULA + SLAAC IPv6
addresses and no IPv4 at all. 3FS currently cannot run on such fabrics —
Address is IPv4-only by design, so neither the TCP control plane nor the
RDMA data plane can bind or dial IPv6 endpoints.

Root cause

src/common/utils/Address.h:

uint32_t ip + static_assert(sizeof(Address) == sizeof(uint64_t));
operator uint64_t() is used as the hash/equality/serialization primitive;
is_serde_copyable makes RPC serialization a raw 8-byte memcpy.

Fitting a 16-byte IPv6 address requires reworking this whole chain, not a
one-line patch.

What we have implemented and validated

We have a working dual-stack implementation (~19 files, +292/−87 against
current main), running for several weeks on a 2-node test deployment:

Address: 8B → 20B (16-byte address + family), IPv6 literal parsing
(TCP://[fd00::1]:8000), FNV hash + explicit operator==, factory helpers.
IfAddrs: collect AF_INET6 addresses (skipping link-local).
Listener: new ipv6_only option (default false, IPv4 deployments
unaffected); ConnectionPool / TcpSocket generalized via
sockaddr_storage with per-family branches.
RDMA: rdma_resolve_addr with sockaddr_in6, dual-stack peer info in QP
metadata exchange, RoCE v2 IPv6 GID selection.
Unit tests updated/added (TestAddress IPv6 cases, TestEcho,
TestIBSocket).

Validation: full cluster (mgmtd / meta / storage / FUSE) over IPv6 for both
TCP control plane and RDMA data plane; real LLM checkpoint loads through FUSE
at multi-GB/s per NIC; unit tests pass.

The design question we want maintainer input on

We kept is_serde_copyable and simply memcpy 20 bytes. This is the smallest
change, but it makes the wire format incompatible with existing v4 binaries:
old and new nodes cannot coexist, so no rolling upgrade. Options we see:

Accept as a breaking change (flag day / major version bump) — simplest.
Move Address to field-based serde for cross-version compatibility —
cleaner long-term, touches RPC schema.
Version-negotiated dual format — most compatible, most complexity.

Which direction would you prefer? We are happy to adapt the implementation.

Proposed contribution plan

If there is interest, we would split the work into reviewable PRs:

Address dual-stack core + unit tests;
TCP path (IfAddrs / Listener / ConnectionPool / TcpSocket);
RDMA path (IBConnect / IBSocket / GID selection).

Happy to share more benchmark details or rebase onto any preferred design.

Originally created by @ShenHuijun on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/418 ## Motivation Some RDMA fabrics are IPv6-only: high-bandwidth NICs get ULA + SLAAC IPv6 addresses and no IPv4 at all. 3FS currently cannot run on such fabrics — `Address` is IPv4-only by design, so neither the TCP control plane nor the RDMA data plane can bind or dial IPv6 endpoints. ## Root cause `src/common/utils/Address.h`: - `uint32_t ip` + `static_assert(sizeof(Address) == sizeof(uint64_t))`; - `operator uint64_t()` is used as the hash/equality/serialization primitive; - `is_serde_copyable` makes RPC serialization a raw 8-byte memcpy. Fitting a 16-byte IPv6 address requires reworking this whole chain, not a one-line patch. ## What we have implemented and validated We have a working dual-stack implementation (~19 files, +292/−87 against current main), running for several weeks on a 2-node test deployment: - `Address`: 8B → 20B (16-byte address + family), IPv6 literal parsing (`TCP://[fd00::1]:8000`), FNV hash + explicit `operator==`, factory helpers. - `IfAddrs`: collect `AF_INET6` addresses (skipping link-local). - `Listener`: new `ipv6_only` option (default false, IPv4 deployments unaffected); `ConnectionPool` / `TcpSocket` generalized via `sockaddr_storage` with per-family branches. - RDMA: `rdma_resolve_addr` with `sockaddr_in6`, dual-stack peer info in QP metadata exchange, RoCE v2 IPv6 GID selection. - Unit tests updated/added (`TestAddress` IPv6 cases, `TestEcho`, `TestIBSocket`). Validation: full cluster (mgmtd / meta / storage / FUSE) over IPv6 for both TCP control plane and RDMA data plane; real LLM checkpoint loads through FUSE at multi-GB/s per NIC; unit tests pass. ## The design question we want maintainer input on We kept `is_serde_copyable` and simply memcpy 20 bytes. This is the smallest change, but it makes the wire format incompatible with existing v4 binaries: old and new nodes cannot coexist, so no rolling upgrade. Options we see: 1. Accept as a breaking change (flag day / major version bump) — simplest. 2. Move `Address` to field-based serde for cross-version compatibility — cleaner long-term, touches RPC schema. 3. Version-negotiated dual format — most compatible, most complexity. Which direction would you prefer? We are happy to adapt the implementation. ## Proposed contribution plan If there is interest, we would split the work into reviewable PRs: 1. `Address` dual-stack core + unit tests; 2. TCP path (IfAddrs / Listener / ConnectionPool / TcpSocket); 3. RDMA path (IBConnect / IBSocket / GID selection). Happy to share more benchmark details or rebase onto any preferred design.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#241

No description provided.

Rows
Columns