deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #437] [MERGED] [Feature] Eager RDMA for lower latency, fenceless token delivery #569

New issue

Closed

opened 2026-10-27 21:04:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:08 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/437
Author: @Gaojiaqi
Created: 9/29/2025
Status: ✅ Merged
Merged: 9/30/2025
Merged by: @Chaser-wind

Base: eager-rdma ← Head: eager-rdma

📝 Commits (4)

ea6f2e0 Add RDMA eager library and kernel integration
6504341 Integrate with python and C++ API
0aaa674 Update readme
de76790 Add performance result

📊 Changes

15 files changed (+1468 additions, -306 deletions)

View changed files

📝 README.md (+83 -0)
📝 csrc/config.hpp (+9 -3)
📝 csrc/deep_ep.cpp (+44 -17)
📝 csrc/deep_ep.hpp (+10 -4)
📝 csrc/kernels/api.cuh (+6 -3)
📝 csrc/kernels/configs.cuh (+2 -2)
➕ csrc/kernels/eager.cuh (+583 -0)
➕ csrc/kernels/eager.h (+52 -0)
📝 csrc/kernels/ibgda_device.cuh (+9 -0)
📝 csrc/kernels/internode_ll.cu (+477 -211)
📝 deep_ep/buffer.py (+25 -13)
➕ figures/EP16eager.png (+0 -0)
➕ figures/EP32eager.png (+0 -0)
📝 tests/test_low_latency.py (+106 -48)
📝 tests/utils.py (+62 -5)

📄 Description

TL;DR: This feature branch replaces the standard “write-then-atomic ACK” path with an Eager RDMA layout that bundles data and per-token signals into a single IBV_WR_RDMA_WRITE. It removes the extra RTT latency introduced by atomics on multi-path NICs and enables fine-grained overlap at the receiver. In our tests (Hopper + BlueField-3, multi-path enabled), Eager RDMA reduces latency by up to 20%.

Background

DeepEP’s current implementation follows the InfiniBand Architecture Specification and uses a standard acknowledgement sequence on the same QP:

Sender issues a burst of IBV_WR_RDMA_WRITE (data “tokens”).
Sender follows with an IBV_WR_RDMA_ATOMIC_FETCH_AND_ADD as the confirmation.
Receiver waits for the atomic to detect completion.

This guarantees that all tokens posted before the atomic are visible when the atomic is observed.

On modern multi-path capable adapters (e.g., NVIDIA ConnectX-7, BlueField-3) and multi-path fabrics (InfiniBand Adaptive Routing, multi-path Ethernet), the network may reorder the atomic and write packets. To preserve ordering semantics, the stack will refrain from issuing the atomic until all in-flight writes on that QP have completed (i.e., the fence)—effectively injecting one RTT of latency for each atomic that follows writes.

Eager RDMA removes atomics and fences from the critical path by interleaving data and a small “signal” field inside every write WQE. The sender emits a single IBV_WR_RDMA_WRITE that carries both payload and signals; the receiver polls those signals directly in device memory to determine readiness. This further enables per-token acknowledgement, receiver can process tokens as soon as they’re ready (fine-grained overlap between intra-node D2D copies and inter-node sending/receiving).

Requirements

GPU: NVIDIA Hopper (SM90) or newer.
MTU: 4096 bytes.

Ordering Guarantees

On Hopper or newer GPUs, the DMA writes arriving from the RNIC to GPU memory and the memory update order visible to GPU kernels are ordered when all of the following hold:

Architecture: GPU is Hopper (SM90) or newer.
CUDA pointer attribute: CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is set via cuPointerSetAttribute (NVSHMEM sets this by default).
IB verbs MR: the MR is registered without IBV_ACCESS_RELAXED_ORDERING (By disabling NVSHMEM_IB_ENABLE_RELAXED_ORDERING).

Under these conditions, when a full-MTU RDMA write completes, the GPU’s memory view is ordered—i.e., once the last 16-byte signal is updated, the preceding 4080 bytes of data in that MTU tile are also visible.

Layout & Data Path

Sender layout

Each 4096-byte MTU tile is laid out as:

Data	Signal	Data	Signal	…	Data	Signal
4080B	16B	4080B	16B	…	(≤4080B)	16B

Data: up to 4080 bytes.
Signal: 16 bytes (per-token signal field).
The final tile may have a short data segment (≤4080B) followed by its 16B signal.

The sender maps its normal memory view to the Eager RDMA view and issues IBV_WR_RDMA_WRITE WQEs with this layout. Original atomics are replaced by RDMA writes.

Receiver progress

The receiver polls the embedded 16B signal of each token.
When a token’s signals all match the expected value, that token’s data is guaranteed ready.
The receiver can immediately consume/copy that token and continue polling subsequent tokens—achieving per-token overlap and reducing overall collective completion time.

Programming Model & APIs

Eager RDMA provides two ways to integrate with existing kernels:

Wrapped load/store APIs (global memory):
If your code uses explicit loads/stores, call the Eager RDMA wrappers to access the interleaved view safely.
In-place TMA view transforms (shared memory):
If your code uses TMA (Tensor Memory Accelerator), use the provided in-place transform to construct/deconstruct the Eager RDMA layout when staging through shared memory.

The goal is to keep your kernel logic largely unchanged while ensuring that reads/writes respect the interleaved data+signal format and ordered visibility rules above.

Performance

Setup: Hopper GPU + BlueField-3 RNIC.
Result: Eager RDMA achieves up to 20% latency reduction vs. the original DeepEP path.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/437 **Author:** [@Gaojiaqi](https://github.com/Gaojiaqi) **Created:** 9/29/2025 **Status:** ✅ Merged **Merged:** 9/30/2025 **Merged by:** [@Chaser-wind](https://github.com/Chaser-wind) **Base:** `eager-rdma` ← **Head:** `eager-rdma` --- ### 📝 Commits (4) - [`ea6f2e0`](https://github.com/deepseek-ai/DeepEP/commit/ea6f2e093968df913da590377e01bd8f53e1f2cf) Add RDMA eager library and kernel integration - [`6504341`](https://github.com/deepseek-ai/DeepEP/commit/6504341feaae0e679a8941f50b81287e47550c4b) Integrate with python and C++ API - [`0aaa674`](https://github.com/deepseek-ai/DeepEP/commit/0aaa6744c890c5af051f622252975c4393795000) Update readme - [`de76790`](https://github.com/deepseek-ai/DeepEP/commit/de76790606a6c273bd1072e8efd5afdbd9c9e24d) Add performance result ### 📊 Changes **15 files changed** (+1468 additions, -306 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+83 -0) 📝 `csrc/config.hpp` (+9 -3) 📝 `csrc/deep_ep.cpp` (+44 -17) 📝 `csrc/deep_ep.hpp` (+10 -4) 📝 `csrc/kernels/api.cuh` (+6 -3) 📝 `csrc/kernels/configs.cuh` (+2 -2) ➕ `csrc/kernels/eager.cuh` (+583 -0) ➕ `csrc/kernels/eager.h` (+52 -0) 📝 `csrc/kernels/ibgda_device.cuh` (+9 -0) 📝 `csrc/kernels/internode_ll.cu` (+477 -211) 📝 `deep_ep/buffer.py` (+25 -13) ➕ `figures/EP16eager.png` (+0 -0) ➕ `figures/EP32eager.png` (+0 -0) 📝 `tests/test_low_latency.py` (+106 -48) 📝 `tests/utils.py` (+62 -5) </details> ### 📄 Description **TL;DR:** This feature branch replaces the standard “write-then-atomic ACK” path with an **Eager RDMA** layout that bundles data and per-token signals into a single `IBV_WR_RDMA_WRITE`. It removes the extra RTT latency introduced by atomics on multi-path NICs and enables fine-grained overlap at the receiver. In our tests (Hopper + BlueField-3, multi-path enabled), Eager RDMA reduces latency by **up to 20%**. ## Background DeepEP’s current implementation follows the InfiniBand Architecture Specification and uses a standard acknowledgement sequence on the **same QP**: * Sender issues a burst of `IBV_WR_RDMA_WRITE` (data “tokens”). * Sender follows with an `IBV_WR_RDMA_ATOMIC_FETCH_AND_ADD` as the confirmation. * Receiver waits for the atomic to detect completion. This guarantees that all tokens posted before the atomic are visible when the atomic is observed. On modern multi-path capable adapters (*e.g.*, NVIDIA ConnectX-7, BlueField-3) and multi-path fabrics (InfiniBand Adaptive Routing, multi-path Ethernet), the network may reorder the atomic and write packets. To preserve ordering semantics, the stack will refrain from issuing the atomic until all in-flight writes on that QP have completed (*i.e.*, the fence)—effectively injecting **one RTT** of latency for each atomic that follows writes. **Eager RDMA** removes atomics and fences from the critical path by **interleaving data and a small “signal” field** inside every write WQE. The sender emits a single `IBV_WR_RDMA_WRITE` that carries both payload and signals; the receiver polls those signals directly in device memory to determine readiness. This further enables **per-token acknowledgement**, receiver can process tokens as soon as they’re ready (fine-grained overlap between intra-node D2D copies and inter-node sending/receiving). ## Requirements 1. **GPU:** NVIDIA **Hopper (SM90)** or newer. 2. **MTU:** **4096 bytes**. ## Ordering Guarantees On Hopper or newer GPUs, the DMA writes arriving from the RNIC to GPU memory and the memory update order visible to GPU kernels are **ordered** when all of the following hold: 1. **Architecture:** GPU is Hopper (SM90) or newer. 2. **CUDA pointer attribute:** `CU_POINTER_ATTRIBUTE_SYNC_MEMOPS` is set via `cuPointerSetAttribute` (NVSHMEM sets this by default). 3. **IB verbs MR:** the MR is registered **without** `IBV_ACCESS_RELAXED_ORDERING` (By disabling `NVSHMEM_IB_ENABLE_RELAXED_ORDERING`). Under these conditions, when a full-MTU RDMA write completes, the GPU’s memory view is ordered—i.e., once the **last 16-byte signal** is updated, the preceding **4080 bytes** of data in that MTU tile are also visible. ## Layout & Data Path ### Sender layout Each 4096-byte MTU tile is laid out as: | Data | Signal | Data | Signal | … | Data | Signal | |:----:|:------:|:----:|:------:|:--:|:----:|:------:| | 4080B | 16B | 4080B | 16B | … | (≤4080B) | 16B | - **Data:** up to **4080 bytes**. - **Signal:** **16 bytes** (per-token signal field). - The final tile may have a short data segment (≤4080B) followed by its 16B signal. The sender maps its **normal memory view** to the **Eager RDMA view** and issues `IBV_WR_RDMA_WRITE` WQEs with this layout. Original atomics are **replaced** by RDMA writes. ### Receiver progress 1. The receiver polls the embedded **16B signal** of each token. 2. When a token’s signals all match the expected value, that token’s data is guaranteed ready. 3. The receiver can immediately consume/copy that token and continue polling subsequent tokens—achieving **per-token overlap** and reducing overall collective completion time. ## Programming Model & APIs Eager RDMA provides two ways to integrate with existing kernels: - **Wrapped load/store APIs (global memory):** If your code uses explicit loads/stores, call the Eager RDMA wrappers to access the interleaved view safely. - **In-place TMA view transforms (shared memory):** If your code uses **TMA** (Tensor Memory Accelerator), use the provided in-place transform to **construct/deconstruct** the Eager RDMA layout when staging through shared memory. > The goal is to keep your kernel logic largely unchanged while ensuring that reads/writes respect the interleaved data+signal format and ordered visibility rules above. ## Performance - **Setup:** Hopper GPU + **BlueField-3** RNIC. - **Result:** Eager RDMA achieves **up to 20% latency reduction** vs. the original DeepEP path. <div style="display: flex; justify-content: space-between;"> <img width="49%" alt="EP16eager" src="https://github.com/user-attachments/assets/3fd6b5ef-f5d1-49c5-bc14-6721f6a17214" /> <img width="49%" alt="EP32eager" src="https://github.com/user-attachments/assets/a6b9a9b7-8e26-4a01-a357-20a31dafd114" /> </div> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:04:08 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#569

No description provided.

Rows
Columns