[PR #333] [MERGED] Refine(IB): ib connect using 1byte probe msg instead of empty payload. #376

New issue

Closed

opened 2026-10-26 09:49:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:49:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/3FS/pull/333
Author: @SimonCqk
Created: 8/29/2025
Status: ✅ Merged
Merged: 1/22/2026
Merged by: @yiyuanliu

Base: main ← Head: main

📝 Commits (3)

abee2ef refine(IB): ib connect using 1byte probe msg instead of empty payload.
2b55e1f Apply suggestion from @Copilot
17134e5 Fix(IB): use immediate message for backward-compatible connect probe.

📊 Changes

3 files changed (+32 additions, -8 deletions)

View changed files

📝 src/common/net/ib/IBConnect.cc (+5 -7)
📝 src/common/net/ib/IBSocket.cc (+26 -1)
📝 src/common/net/ib/IBSocket.h (+1 -0)

📄 Description

Fix #320

We observed that RDMA connections consistently fail between different VMs. However, RDMA communication works when performed on the same host (loopback through the same physical NIC). After adding extensive debug logging, we traced the problem to the connection setup phase. It appears that the transport layer in these specific virtualized environments(e.g. GCP) silently drops empty (0-byte) RDMA packets, in the current 3FS implementation, the IBConnect class sends a 0-byte event to notify the peer as part of the connection handshake. This 0-byte packet is the one being dropped, and as a test, we modified the implementation to send a 1-byte event instead of a 0-byte one.
With this change, the RDMA connection is established successfully across different VMs, and the system works as expected in both native IB bare-metal environment and cloud-vendor virtualized RDMA environment.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/3FS/pull/333 **Author:** [@SimonCqk](https://github.com/SimonCqk) **Created:** 8/29/2025 **Status:** ✅ Merged **Merged:** 1/22/2026 **Merged by:** [@yiyuanliu](https://github.com/yiyuanliu) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (3) - [`abee2ef`](https://github.com/deepseek-ai/3FS/commit/abee2ef17154a1ad02f682e60ae3d3e28520dcf5) refine(IB): ib connect using 1byte probe msg instead of empty payload. - [`2b55e1f`](https://github.com/deepseek-ai/3FS/commit/2b55e1f0ea9b0d0db5c174c060b210e898c48c2a) Apply suggestion from @Copilot - [`17134e5`](https://github.com/deepseek-ai/3FS/commit/17134e597dea88214782c6b17950cc4c508a7bf2) Fix(IB): use immediate message for backward-compatible connect probe. ### 📊 Changes **3 files changed** (+32 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `src/common/net/ib/IBConnect.cc` (+5 -7) 📝 `src/common/net/ib/IBSocket.cc` (+26 -1) 📝 `src/common/net/ib/IBSocket.h` (+1 -0) </details> ### 📄 Description Fix #320 We observed that RDMA connections consistently fail between different VMs. However, RDMA communication works when performed on the same host (loopback through the same physical NIC). After adding extensive debug logging, we traced the problem to the connection setup phase. It appears that the transport layer in these specific virtualized environments(e.g. GCP) silently drops empty (0-byte) RDMA packets, in the current 3FS implementation, the IBConnect class sends a 0-byte event to notify the peer as part of the connection handshake. This 0-byte packet is the one being dropped, and as a test, we modified the implementation to send a 1-byte event instead of a 0-byte one. With this change, the RDMA connection is established successfully across different VMs, and the system works as expected in both native IB bare-metal environment and cloud-vendor virtualized RDMA environment. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 09:49:21 +00:00