[PR #430] IBSocket: post small sends inline #431

New issue

Open

opened 2026-10-26 09:50:11 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:50:11 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/3FS/pull/430
Author: @alxrxs
Created: 9/28/2026
Status: 🔄 Open

Base: main ← Head: ib-inline-small-sends

📝 Commits (4)

125bcf1 IBSocket: post small sends inline
cf9f237 IBSocket: refine the inline size limit and the QP creation fallback
f8c36ec IBSocket: try known inline limits, then step down
2fdfef5 IBSocket: step down only on EINVAL and reuse each device's inline size

📊 Changes

4 files changed (+64 additions, -1 deletions)

View changed files

📝 src/common/net/ib/IBConnect.cc (+47 -1)
📝 src/common/net/ib/IBDevice.h (+6 -0)
📝 src/common/net/ib/IBSocket.cc (+5 -0)
📝 src/common/net/ib/IBSocket.h (+6 -0)

📄 Description

IBSocket::postSend() posts every send buffer by reference and the QP is created with max_inline_data = 0, so for a small RPC request or response the NIC first has to read the message from host memory. In a separate ping-pong microbenchmark that sends the same small RDMA write with and without IBV_SEND_INLINE, inlining cut one-way latency by about 0.43 us on Xeon 8480C hosts with ConnectX-7 and 0.89 us on EPYC 7742 hosts with ConnectX-6.

This requests 236 bytes of inline data in qpCreate(), less if the device refuses (see below), and sets IBV_SEND_INLINE in postSend() when the message fits the granted size. 236 bytes is the largest Send whose inlined WQE fits the 256-byte BlueFlame buffer of current mlx5 NICs. The 236-byte limit applies only on Mellanox/NVIDIA NICs (vendor ID 0x02c9); other devices inline up to the size they grant. The verbs API cannot report the inline limit, so devices whose driver has a fixed one try it first, by vendor ID (Intel 216, 101 or 48 depending on the irdma generation, 101 on the E810 and 48 on the X722; Alibaba eRDMA 96), and a device that refuses a size is asked for 16 bytes less each time until it accepts, down to none. Only EINVAL counts as a refusal; any other failure is reported as before. Each device remembers the size its last QP was created with, so later connections start there instead of stepping down again. With the default max_sge of 16 the send WQE stays the same size. IBSocket already posts its empty liveness-check and close writes with IBV_SEND_INLINE. Nothing on the wire changes, so old and new peers still work together.

I haven't measured 3FS itself. I compiled IBConnect.cc, IBSocket.cc, IBDevice.cc and RDMABuf.cc (clang 17, -Wall -Wextra, no warnings) but could not build the whole tree or run it.

Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/3FS/pull/430 **Author:** [@alxrxs](https://github.com/alxrxs) **Created:** 9/28/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `ib-inline-small-sends` --- ### 📝 Commits (4) - [`125bcf1`](https://github.com/deepseek-ai/3FS/commit/125bcf16545271b405ce2a4c7905883aaf9e5d90) IBSocket: post small sends inline - [`cf9f237`](https://github.com/deepseek-ai/3FS/commit/cf9f23740e929f50ee12983c49647c8b6a57cf76) IBSocket: refine the inline size limit and the QP creation fallback - [`f8c36ec`](https://github.com/deepseek-ai/3FS/commit/f8c36ec483e43cc9f0623d6f86a974d57dd943c5) IBSocket: try known inline limits, then step down - [`2fdfef5`](https://github.com/deepseek-ai/3FS/commit/2fdfef555c2330e8ce629a172e92386e15bde0e4) IBSocket: step down only on EINVAL and reuse each device's inline size ### 📊 Changes **4 files changed** (+64 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `src/common/net/ib/IBConnect.cc` (+47 -1) 📝 `src/common/net/ib/IBDevice.h` (+6 -0) 📝 `src/common/net/ib/IBSocket.cc` (+5 -0) 📝 `src/common/net/ib/IBSocket.h` (+6 -0) </details> ### 📄 Description IBSocket::postSend() posts every send buffer by reference and the QP is created with max_inline_data = 0, so for a small RPC request or response the NIC first has to read the message from host memory. In a separate ping-pong microbenchmark that sends the same small RDMA write with and without IBV_SEND_INLINE, inlining cut one-way latency by about 0.43 us on Xeon 8480C hosts with ConnectX-7 and 0.89 us on EPYC 7742 hosts with ConnectX-6. This requests 236 bytes of inline data in qpCreate(), less if the device refuses (see below), and sets IBV_SEND_INLINE in postSend() when the message fits the granted size. 236 bytes is the largest Send whose inlined WQE fits the 256-byte BlueFlame buffer of current mlx5 NICs. The 236-byte limit applies only on Mellanox/NVIDIA NICs (vendor ID 0x02c9); other devices inline up to the size they grant. The verbs API cannot report the inline limit, so devices whose driver has a fixed one try it first, by vendor ID (Intel 216, 101 or 48 depending on the irdma generation, 101 on the E810 and 48 on the X722; Alibaba eRDMA 96), and a device that refuses a size is asked for 16 bytes less each time until it accepts, down to none. Only EINVAL counts as a refusal; any other failure is reported as before. Each device remembers the size its last QP was created with, so later connections start there instead of stepping down again. With the default max_sge of 16 the send WQE stays the same size. IBSocket already posts its empty liveness-check and close writes with IBV_SEND_INLINE. Nothing on the wire changes, so old and new peers still work together. I haven't measured 3FS itself. I compiled IBConnect.cc, IBSocket.cc, IBDevice.cc and RDMABuf.cc (clang 17, -Wall -Wextra, no warnings) but could not build the whole tree or run it. Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 09:50:11 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/3FS#431

No description provided.

Rows
Columns