[GH-ISSUE #431] RDMA handshake fails with IBDeviceNotFound(2016) / "peer doesn't have active port" despite the peer's RDMA port being ACTIVE with a valid GID #246

New issue

Open

opened 2026-10-26 09:47:43 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-26 09:47:43 +00:00

Owner

Copy link

Originally created by @sivabreddy on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/431

Summary

During a basic raw write to a freshly mounted 3FS filesystem (via the FUSE client), every write attempt fails with an RDMA connection error raised during the storage-client-to-storage-node handshake. The storage node's own log shows no error at all in the same time window, and the storage node's own /sys/class/infiniband/<dev>/ports/1/state shows ACTIVE with a valid, non-zero GID at the OS level. A controlled comparison test ruled out our own device_filter configuration as the cause.

Environment

• 3FS built from source per the project's own build instructions, on a Kubernetes cluster (main branch, built early-to-mid October 2026 — exact commit not captured in this report, happy to provide if it helps)
• RDMA provided via Mellanox mlx5 devices, exposed to pods as a Kubernetes shared RDMA device-plugin resource, attached via Multus CNI
• Test topology: 1 FoundationDB instance, 1 mgmtd, 1 meta, 2 storage nodes, 1 FUSE client (single-instance test deployment of each)
• device_filter in storage_main_launcher.toml restricted to one device (mlx5_0), the device confirmed to have a valid, routable GID on this cluster

Steps to reproduce

1. Stand up the full 3FS stack (FoundationDB, mgmtd, meta, 2x storage, FUSE client) with RDMA transport enabled
2. Mount the filesystem via hf3fs_fuse_main
3. Run a basic write through the mount, e.g. dd if=/dev/urandom of=/mnt/3fs/test/testfile bs=1M count=10

Expected

The write completes. (This exact test reportedly completed successfully twice earlier, on the same cluster, under a different device_filter device list.)

Actual

The write hangs, then fails. FUSE client error log:

IBConnect.cc:264 ERROR] IBSocket [connect RDMA://<storage-node-ip>:8000] peer doesn't have active port
IOWorker.cc:100 ERROR] error: RPC::IBDeviceNotFound(2016) IBSocket [connect RDMA://...] peer doesn't have active port
StorageMessenger.cc:31 ERROR] RPC communication error: RPC::SendFailed(2007), ... peer address: RDMA://<storage-node-ip>:8000
StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops ... after #3 retries, elapsed time: 60001ms, max retry time: 60000ms

The storage node named in the error (the connection target) shows zero errors or warnings in its own log for the same time window — only normal startup INFO lines.

What we traced so far

We traced IBDeviceNotFound(2016) to IBQueryRsp::selectDevice in src/common/net/ib/IBConnect.cc. From reading the surrounding code, this appears to fire when the peer's own advertised device list comes back empty during the connection handshake.

We then checked the storage node's own RDMA state directly at the OS level:

• /sys/class/infiniband/mlx5_0/ports/1/state → 4: ACTIVE
• /sys/class/infiniband/mlx5_0/ports/1/gids/* → a valid, non-zero GID is present

So at the OS/kernel level, the device the storage node should be advertising is active and has a valid GID — but the handshake-level code reports it has no usable device to offer.

Ruling out our own config as the cause

To check whether our own device_filter restriction (["mlx5_0"] only) was itself causing this, we reverted only the storage nodes to a broader, previously-used device_filter list (several mlx5_* devices) and retried the same write. The identical failure reproduced — same error code, same message — which rules out our device_filter setting as the explanation.

One more data point

With a broader device_filter list, this exact write test reportedly completed successfully twice in an earlier session on this same cluster. A later retest, with freshly created pods, could not reproduce that success at all — under either the broad or the restricted device_filter list. We were not able to isolate why the handshake-level device advertisement disagrees with the OS-level port state, or why behavior changed between the two sessions.

We're filing this in case it's a known failure class, or in case IBQueryRsp::selectDevice / whatever populates the peer's advertised device list is a useful starting point for your team to look at.

Happy to provide more detail (fuller logs, config excerpts) on request — we've redacted internal addressing (IPs, hostnames) from this report for our own confidentiality.

Originally created by @sivabreddy on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/431 ## Summary During a basic raw write to a freshly mounted 3FS filesystem (via the FUSE client), every write attempt fails with an RDMA connection error raised during the storage-client-to-storage-node handshake. The storage node's own log shows **no error at all** in the same time window, and the storage node's own `/sys/class/infiniband/<dev>/ports/1/state` shows `ACTIVE` with a valid, non-zero GID at the OS level. A controlled comparison test ruled out our own `device_filter` configuration as the cause. ## Environment - 3FS built from source per the project's own build instructions, on a Kubernetes cluster (main branch, built early-to-mid October 2026 — exact commit not captured in this report, happy to provide if it helps) - RDMA provided via Mellanox `mlx5` devices, exposed to pods as a Kubernetes shared RDMA device-plugin resource, attached via Multus CNI - Test topology: 1 FoundationDB instance, 1 `mgmtd`, 1 `meta`, 2 `storage` nodes, 1 FUSE client (single-instance test deployment of each) - `device_filter` in `storage_main_launcher.toml` restricted to one device (`mlx5_0`), the device confirmed to have a valid, routable GID on this cluster ## Steps to reproduce 1. Stand up the full 3FS stack (FoundationDB, `mgmtd`, `meta`, 2x `storage`, FUSE client) with RDMA transport enabled 2. Mount the filesystem via `hf3fs_fuse_main` 3. Run a basic write through the mount, e.g. `dd if=/dev/urandom of=/mnt/3fs/test/testfile bs=1M count=10` ## Expected The write completes. (This exact test reportedly completed successfully twice earlier, on the same cluster, under a different `device_filter` device list.) ## Actual The write hangs, then fails. FUSE client error log: ``` IBConnect.cc:264 ERROR] IBSocket [connect RDMA://<storage-node-ip>:8000] peer doesn't have active port IOWorker.cc:100 ERROR] error: RPC::IBDeviceNotFound(2016) IBSocket [connect RDMA://...] peer doesn't have active port StorageMessenger.cc:31 ERROR] RPC communication error: RPC::SendFailed(2007), ... peer address: RDMA://<storage-node-ip>:8000 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops ... after #3 retries, elapsed time: 60001ms, max retry time: 60000ms ``` The storage node named in the error (the connection target) shows **zero errors or warnings** in its own log for the same time window — only normal startup INFO lines. ## What we traced so far We traced `IBDeviceNotFound(2016)` to `IBQueryRsp::selectDevice` in `src/common/net/ib/IBConnect.cc`. From reading the surrounding code, this appears to fire when the **peer's own advertised device list comes back empty** during the connection handshake. We then checked the storage node's own RDMA state directly at the OS level: - `/sys/class/infiniband/mlx5_0/ports/1/state` → `4: ACTIVE` - `/sys/class/infiniband/mlx5_0/ports/1/gids/*` → a valid, non-zero GID is present So at the OS/kernel level, the device the storage node should be advertising is active and has a valid GID — but the handshake-level code reports it has no usable device to offer. ## Ruling out our own config as the cause To check whether our own `device_filter` restriction (`["mlx5_0"]` only) was itself causing this, we reverted only the `storage` nodes to a broader, previously-used `device_filter` list (several `mlx5_*` devices) and retried the same write. **The identical failure reproduced** — same error code, same message — which rules out our `device_filter` setting as the explanation. ## One more data point With a broader `device_filter` list, this exact write test reportedly completed successfully twice in an earlier session on this same cluster. A later retest, with freshly created pods, could not reproduce that success at all — under either the broad or the restricted `device_filter` list. We were not able to isolate why the handshake-level device advertisement disagrees with the OS-level port state, or why behavior changed between the two sessions. We're filing this in case it's a known failure class, or in case `IBQueryRsp::selectDevice` / whatever populates the peer's advertised device list is a useful starting point for your team to look at. Happy to provide more detail (fuller logs, config excerpts) on request — we've redacted internal addressing (IPs, hostnames) from this report for our own confidentiality.

bsakaguchi commented 2026-10-26 09:47:45 +00:00

Author

Owner

Copy link

@sivabreddy commented on GitHub (Oct 19, 2026):

Update: root cause found, with a small patch

The cause is in queryRoCEv2GID() (src/common/net/ib/IBDevice.cc). It occurs when more than one RDMA pod runs on the same node and the RDMA subsystem is in netns shared mode. With a small change (below), the full stack works on our cluster: writes, reads, cross-node chain replication and the FUSE mount. The build was commit 22fca04 (the current main) + the patch.

Corrections to my original report

• "The storage node shows zero errors": I read only the container stdout. 3FS writes its log to /var/log/3fs/*.log, which I did not read in the first session. From the code, the storage process logs the device drop at startup (failed to query GID type from sysfs, then Failed to open IBDevice ..., skip it). I did not keep the logs from the unpatched binary.
• The "exact commit not captured" in my report: main has not changed since 22fca04, so the build was that commit.
• It still holds that device_filter is not the cause.

What happens

• Each pod gets the shared mlx5_0 device (RoCE, port ACTIVE) through Multus, with its own network interface and its own RoCE v2 GID (the pod's IPv4 address). rdma system shows netns shared.
• In a pod, ibv_query_gid() also returns the GIDs of other pods' interfaces on the same node. Reading /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/<index> for those entries fails with EINVAL.
• queryRoCEv2GID() checks the indexes in order. On the first non-zero, non-link-local GID, it reads the sysfs type. If that read fails, it returns kIBOpenPortFailed, so IBDevice::open fails and the device is skipped (skip_unusable_device = true).
• So only the RDMA pod that owns the lowest GID index on a node can use the device. Every other pod has no usable RDMA device.
• Our configs had allow_no_usable_devices = true, so storage_main still started. IBConnectService::query then returned an empty device list, and the client got IBDeviceNotFound(2016) "peer doesn't have active port".
• This probably explains why the test passed in one session and failed in a later one: the pod start order decides which pod owns the lowest GID index. I did not verify this for the earlier session.

Summary of a probe in a second RDMA pod on the same node (a small C program with the same logic as queryRoCEv2GID()): indexes 0 and 1 are link-local and are skipped. Index 2 is the first pod's IPv4 GID, and reading its sysfs type fails, so the device is dropped. In the first pod, the same program reaches its own RoCE v2 GID at index 3 and succeeds.

Patch

--- a/src/common/net/ib/IBDevice.cc
+++ b/src/common/net/ib/IBDevice.cc
@@ -115,8 +115,17 @@ Result<std::pair<ibv_gid, uint8_t>> queryRoCEv2GID(ibv_context *ctx, uint8_t por
     auto gidType = std::string();
     auto ok = folly::readFile(path.c_str(), gidType);
     if (!ok) {
-      XLOGF(CRITICAL, "{}:{} failed to query GID type from sysfs {}", devname, portNum, path);
-      return makeError(RPCCode::kIBOpenPortFailed, "query GID type failed");
+      // In a container with RDMA netns "shared" mode, ibv_query_gid() also returns GIDs that belong
+      // to netdevs in other network namespaces, but their sysfs gid_attrs are hidden (EINVAL).
+      // Such a GID is not usable from this namespace, so skip it instead of failing the device.
+      XLOGF(WARN,
+            "{}:{} skip GID {}, ip {}: cannot read GID type from sysfs {} (not in this netns?)",
+            devname,
+            portNum,
+            index,
+            ip.str(),
+            path);
+      continue;
     }

On bare metal, or with netns exclusive, every GID type is readable, so the behavior does not change. On a kernel without gid_attrs, the device still fails ("RoCE v2 GID not found" instead of "query GID type failed").

One side effect: queryRoCEv2GID() also runs from IBDevice::openPort() for each new connection, so the WARN lines repeat for each connection. If you prefer, the message can be DBG, or rate-limited.

With the patch, an RDMA pod that shares a node with two other RDMA pods logs:

IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 2, ip ::ffff:<pod-1 IP>: cannot read GID type from sysfs ... (not in this netns?)
IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 3, ip ::ffff:<pod-1 IP>: ...
IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 6, ip ::ffff:<pod-2 IP>: ...
IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 7, ip ::ffff:<pod-2 IP>: ...
IBDevice.cc:458 INFO] IBDevice mlx5_0 add active port 1, linklayer ETHERNET, ... RoCE v2 GID ...<own IP>

We also had up to 5 RDMA pods on one node; that pod skipped 8 GIDs of 4 other pods.

Verified with the patch (commit 22fca04 + patch, 2 nodes)

• mgmtd, meta and 2 × storage are HEARTBEAT_CONNECTED. The chain is SERVING, with the head target on node 1 and the tail target on node 2, both UPTODATE.
• admin_cli write-file / read-file: 5 writes (2 × 10 MB, 3 × 2 MB). The 2 × 10 MB files read back with matching MD5. Both targets show 20 MB used.
• FUSE mount: 2 × 50 MB written and read back with matching MD5 (the reads may have come from the page cache).
• Two vLLM instances on different nodes used the FUSE mount as a shared KV-cache tier (OffloadingConnector, fs tier). Instance B found and loaded all 67 blocks that instance A wrote (read bytes = written bytes = 562,036,736, 2 trials).
• RDMA is used for the data path:
  • A 4 GiB direct-I/O read through FUSE raised the client node's rx_write_requests by 2,097,155 (about 4 GiB / 2,048-byte RoCE packets), while the pod's TCP interface moved 1.8 MB.
  • A 4 GiB write moved 4 MB over TCP and raised rx_read_requests by 15,076.
• Shared KV cache for Llama-3.1-70B (TP4, one engine per node), written on one node and read on the other:
  • SGLang HiCache with the hf3fs backend (USRBIO client built from the same patched source): loads of 7K–53K-token prefixes took 0.30–2.07 s, compared with 0.52–6.11 s to recompute (1.7–2.6× faster). 8 concurrent 14K-token requests: 3.1 s wall time, compared with 8.6 s (24,000 of 112,128 tokens were computed because of the host-memory tier size).
  • vLLM fs tier on the FUSE mount: all blocks loaded, but serially, at about 0.6 GB/s.
• FUSE throughput with dd (direct I/O, 4 MiB, 1 GiB per stream) on a 2-replica chain:
  • write 168 / 706 / 1,413 MB/s and read 581 / 2,366 / 4,544 MB/s for 1 / 4 / 8 streams.
  • For comparison: the local disk does 2,953 MB/s write and 4,658 MB/s read (1 stream), and node-to-node ib_write_bw does 379 Gb/s.
• Tail storage service deleted during reads: the chain went to SERVING(1/2), and reads through FUSE continued with a stall of about 20 s. The chain was SERVING with both targets UPTODATE 95 s after the service was back.

Observation after the storage failure (not a 3FS bug)

One SGLang instance that read through USRBIO kept failing its reads after the chain was healthy again, until the process restarted. The cause is in SGLang's USRBIO client: after ioring.wait(min_results=n, timeout=…) returns partial results, SGLang does not reap the remaining I/Os, so the ring stays full and every later prepare() fails with EAGAIN. A client-side fix solved it. We will report it to SGLang.

Two smaller notes

• With allow_no_usable_devices = true, the device loss is silent at the RPC level. A client message such as "peer has no usable RDMA device" would point to the peer's startup log.
• The admin_cli storage client must not use force_use_tcp = true for writes: the storage server logs update no RDMA socket and the write fails.

You can apply the patch directly if you agree with the approach; I can also open a PR. Another option, which I did not test, is to check gid_attrs/ndevs/<index> before using a GID. I can provide more logs.

<!-- gh-comment-id:6258953727 --> @sivabreddy commented on GitHub (Oct 19, 2026): ## Update: root cause found, with a small patch The cause is in `queryRoCEv2GID()` (`src/common/net/ib/IBDevice.cc`). It occurs when **more than one RDMA pod runs on the same node** and the RDMA subsystem is in netns `shared` mode. With a small change (below), the full stack works on our cluster: writes, reads, cross-node chain replication and the FUSE mount. The build was commit `22fca04` (the current `main`) + the patch. ### Corrections to my original report - "The storage node shows zero errors": I read only the container stdout. 3FS writes its log to `/var/log/3fs/*.log`, which I did not read in the first session. From the code, the storage process logs the device drop at startup (`failed to query GID type from sysfs`, then `Failed to open IBDevice ..., skip it`). I did not keep the logs from the unpatched binary. - The "exact commit not captured" in my report: `main` has not changed since `22fca04`, so the build was that commit. - It still holds that `device_filter` is not the cause. ### What happens - Each pod gets the shared `mlx5_0` device (RoCE, port `ACTIVE`) through Multus, with its own network interface and its own RoCE v2 GID (the pod's IPv4 address). `rdma system` shows `netns shared`. - In a pod, `ibv_query_gid()` also returns the GIDs of **other pods'** interfaces on the same node. Reading `/sys/class/infiniband/mlx5_0/ports/1/gid_attrs/types/<index>` for those entries fails with `EINVAL`. - `queryRoCEv2GID()` checks the indexes in order. On the first non-zero, non-link-local GID, it reads the sysfs type. If that read fails, it returns `kIBOpenPortFailed`, so `IBDevice::open` fails and the device is skipped (`skip_unusable_device = true`). - So only the RDMA pod that owns the lowest GID index on a node can use the device. Every other pod has no usable RDMA device. - Our configs had `allow_no_usable_devices = true`, so `storage_main` still started. `IBConnectService::query` then returned an empty device list, and the client got `IBDeviceNotFound(2016)` "peer doesn't have active port". - This probably explains why the test passed in one session and failed in a later one: the pod start order decides which pod owns the lowest GID index. I did not verify this for the earlier session. Summary of a probe in a second RDMA pod on the same node (a small C program with the same logic as `queryRoCEv2GID()`): indexes 0 and 1 are link-local and are skipped. Index 2 is the first pod's IPv4 GID, and reading its sysfs type fails, so the device is dropped. In the first pod, the same program reaches its own RoCE v2 GID at index 3 and succeeds. ### Patch ```diff --- a/src/common/net/ib/IBDevice.cc +++ b/src/common/net/ib/IBDevice.cc @@ -115,8 +115,17 @@ Result<std::pair<ibv_gid, uint8_t>> queryRoCEv2GID(ibv_context *ctx, uint8_t por auto gidType = std::string(); auto ok = folly::readFile(path.c_str(), gidType); if (!ok) { - XLOGF(CRITICAL, "{}:{} failed to query GID type from sysfs {}", devname, portNum, path); - return makeError(RPCCode::kIBOpenPortFailed, "query GID type failed"); + // In a container with RDMA netns "shared" mode, ibv_query_gid() also returns GIDs that belong + // to netdevs in other network namespaces, but their sysfs gid_attrs are hidden (EINVAL). + // Such a GID is not usable from this namespace, so skip it instead of failing the device. + XLOGF(WARN, + "{}:{} skip GID {}, ip {}: cannot read GID type from sysfs {} (not in this netns?)", + devname, + portNum, + index, + ip.str(), + path); + continue; } ``` On bare metal, or with netns `exclusive`, every GID type is readable, so the behavior does not change. On a kernel without `gid_attrs`, the device still fails ("RoCE v2 GID not found" instead of "query GID type failed"). One side effect: `queryRoCEv2GID()` also runs from `IBDevice::openPort()` for each new connection, so the WARN lines repeat for each connection. If you prefer, the message can be DBG, or rate-limited. With the patch, an RDMA pod that shares a node with two other RDMA pods logs: ``` IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 2, ip ::ffff:<pod-1 IP>: cannot read GID type from sysfs ... (not in this netns?) IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 3, ip ::ffff:<pod-1 IP>: ... IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 6, ip ::ffff:<pod-2 IP>: ... IBDevice.cc:122 WARNING] mlx5_0:1 skip GID 7, ip ::ffff:<pod-2 IP>: ... IBDevice.cc:458 INFO] IBDevice mlx5_0 add active port 1, linklayer ETHERNET, ... RoCE v2 GID ...<own IP> ``` We also had up to 5 RDMA pods on one node; that pod skipped 8 GIDs of 4 other pods. ### Verified with the patch (commit `22fca04` + patch, 2 nodes) - `mgmtd`, `meta` and 2 × `storage` are `HEARTBEAT_CONNECTED`. The chain is `SERVING`, with the head target on node 1 and the tail target on node 2, both `UPTODATE`. - `admin_cli write-file` / `read-file`: 5 writes (2 × 10 MB, 3 × 2 MB). The 2 × 10 MB files read back with matching MD5. Both targets show 20 MB used. - FUSE mount: 2 × 50 MB written and read back with matching MD5 (the reads may have come from the page cache). - Two vLLM instances on different nodes used the FUSE mount as a shared KV-cache tier (`OffloadingConnector`, `fs` tier). Instance B found and loaded all 67 blocks that instance A wrote (read bytes = written bytes = 562,036,736, 2 trials). - RDMA is used for the data path: - A 4 GiB direct-I/O read through FUSE raised the client node's `rx_write_requests` by 2,097,155 (about 4 GiB / 2,048-byte RoCE packets), while the pod's TCP interface moved 1.8 MB. - A 4 GiB write moved 4 MB over TCP and raised `rx_read_requests` by 15,076. - Shared KV cache for Llama-3.1-70B (TP4, one engine per node), written on one node and read on the other: - SGLang HiCache with the `hf3fs` backend (USRBIO client built from the same patched source): loads of 7K–53K-token prefixes took 0.30–2.07 s, compared with 0.52–6.11 s to recompute (1.7–2.6× faster). 8 concurrent 14K-token requests: 3.1 s wall time, compared with 8.6 s (24,000 of 112,128 tokens were computed because of the host-memory tier size). - vLLM `fs` tier on the FUSE mount: all blocks loaded, but serially, at about 0.6 GB/s. - FUSE throughput with `dd` (direct I/O, 4 MiB, 1 GiB per stream) on a 2-replica chain: - write 168 / 706 / 1,413 MB/s and read 581 / 2,366 / 4,544 MB/s for 1 / 4 / 8 streams. - For comparison: the local disk does 2,953 MB/s write and 4,658 MB/s read (1 stream), and node-to-node `ib_write_bw` does 379 Gb/s. - Tail storage service deleted during reads: the chain went to `SERVING(1/2)`, and reads through FUSE continued with a stall of about 20 s. The chain was `SERVING` with both targets `UPTODATE` 95 s after the service was back. ### Observation after the storage failure (not a 3FS bug) One SGLang instance that read through USRBIO kept failing its reads after the chain was healthy again, until the process restarted. The cause is in SGLang's USRBIO client: after `ioring.wait(min_results=n, timeout=…)` returns partial results, SGLang does not reap the remaining I/Os, so the ring stays full and every later `prepare()` fails with EAGAIN. A client-side fix solved it. We will report it to SGLang. ### Two smaller notes - With `allow_no_usable_devices = true`, the device loss is silent at the RPC level. A client message such as "peer has no usable RDMA device" would point to the peer's startup log. - The `admin_cli` storage client must not use `force_use_tcp = true` for writes: the storage server logs `update no RDMA socket` and the write fails. You can apply the patch directly if you agree with the approach; I can also open a PR. Another option, which I did not test, is to check `gid_attrs/ndevs/<index>` before using a GID. I can provide more logs.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

SF-Zhou-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#246

No description provided.