[GH-ISSUE #793] [Question & bug]Two findings on the V2.5 scale-out tail-signal path: window ordering (question) and scale-out tail counter stride may collapses the NIC's atomic rate (bug) #424

New issue

Open

opened 2026-10-26 10:00:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:27 +00:00

Owner

Originally created by @Difers on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/793

First of all, thank you for DeepEP. The V2.5 hybrid dispatch design is impressive work.
We have been benchmarking V2.5 (93eb6eb) on a 2-node × 8-GPU cluster, with one dedicated 400 Gb/s ConnectX-8 per GPU (802.3ad LAG, RoCE v2) and the NCCL 2.32.3 GDAKI backend, and we ran into two issues on the scale-out tail-signal path.

Part 1 (question): is NCCL_WIN_DEFAULT safe when the NIC keeps PCI_ATOMIC_MODE=0?

README.md (PCI atomic mode) recommends mlxconfig set PCI_ATOMIC_MODE=4 for RDMA atomic
performance.
csrc/kernels/comm/context.cpp:199 registers the main symmetric window unconditionally with
NCCL_WIN_STRICT_ORDERING, i.e. PCIe relaxed ordering is disabled on that MR.

Our understanding of why 2 is needed, based on https://github.com/deepseek-ai/DeepEP/issues/661: the scale-out tail signal is an
RDMA atomic on a VA inside the same window as the data. With PCI_ATOMIC_MODE=4 the NIC issues a
native PCIe AtomicOp (NPR-with-data), and PCIe ordering rule C2b permits an NPR-with-data to pass
prior posted writes. Under relaxed ordering the tail increment could therefore become visible
before the token payload has landed, which breaks the data-before-signal guarantee that
red_add_rel relies on. With PCI_ATOMIC_MODE=0 the NIC performs a PCI read-modify-write
instead, and a Read may not pass prior posted writes even under relaxed ordering, so the same
hazard does not arise.

Does that mean strong ordering is required because of the recommended atomic mode, and that a
deployment which leaves PCI_ATOMIC_MODE at 0 can use relaxed ordering safely?
We ask because our cluster is probably in that regime: the NICs are presumably at the default PCI_ATOMIC_MODE=0, yet the window is strictly ordered, and dispatch is limited to 12 GB/s at 32 SMs.

Part 2 (possible bug):scale-out tail counter stride (8 KiB) collapses the NIC's atomic rate.

EPSignals::scaleout_channel_signaled_tailis dimensioned with the compile-time maximum rank count, so consecutive channels' tails are 8192 B apart. That stride reaches the wire unchanged — red_add_rel computes ptr - lsa_base_ptr, ncclGin_VASignalAdd carries it as a window offset, and GDAKI places it directly into the raddr of an ATOMIC_FA WQE. We observed that this appears to reduce the responder NIC's sustained atomic rate.

To isolate the effect of this stride, we built a small standalone reproducer that does not depend on DeepEP, NCCL, or a GPU: 32 RC QPs, each issuing 8 B IBV_WR_ATOMIC_FETCH_AND_ADD on its own counter in host memory. The only variable is the per-QP counter stride. Atomics are posted in chains of 8 with only the last one signalled, since signalling every WR makes the requester's CPU the bottleneck and masks the effect.

The results show a cliff between 1024 B and 4096 B，All rates in Mops (10^6 atomic ops/s). :

stride	ConnectX-8 400G, 2 hosts	ConnectX-7 200G, different network	same CX7, loopback, no switch
64 B	24.33	18.32	19.65
256 B	17.51	17.15	15.24
1024 B	4.66	4.47	4.53
4096 B	1.30	1.21	1.22
8192 B	1.30	1.21	1.21
65536 B	1.30	1.21	1.23

atomic_stride_repro.c

Originally created by @Difers on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/793 First of all, thank you for DeepEP. The V2.5 hybrid dispatch design is impressive work. We have been benchmarking V2.5 (93eb6eb) on a 2-node × 8-GPU cluster, with one dedicated 400 Gb/s ConnectX-8 per GPU (802.3ad LAG, RoCE v2) and the NCCL 2.32.3 GDAKI backend, and we ran into two issues on the scale-out tail-signal path. **Part 1 (question): is NCCL_WIN_DEFAULT safe when the NIC keeps PCI_ATOMIC_MODE=0?** 1. `README.md` (PCI atomic mode) recommends `mlxconfig set PCI_ATOMIC_MODE=4` for RDMA atomic performance. 2. [`csrc/kernels/comm/context.cpp:199`](https://github.com/deepseek-ai/DeepEP/blob/f6aa38fe06cd4a0489e7730365d4e0114970a2f3/csrc/kernels/comm/context.cpp#L199) registers the main symmetric window unconditionally with `NCCL_WIN_STRICT_ORDERING`, i.e. PCIe relaxed ordering is disabled on that MR. Our understanding of why 2 is needed, based on https://github.com/deepseek-ai/DeepEP/issues/661: the scale-out tail signal is an RDMA atomic on a VA inside the same window as the data. With `PCI_ATOMIC_MODE=4` the NIC issues a native PCIe AtomicOp (NPR-with-data), and PCIe ordering rule C2b permits an NPR-with-data to pass prior posted writes. Under relaxed ordering the tail increment could therefore become visible before the token payload has landed, which breaks the data-before-signal guarantee that `red_add_rel` relies on. With `PCI_ATOMIC_MODE=0` the NIC performs a PCI read-modify-write instead, and a Read may not pass prior posted writes even under relaxed ordering, so the same hazard does not arise. Does that mean strong ordering is required *because of* the recommended atomic mode, and that a deployment which leaves `PCI_ATOMIC_MODE` at 0 can use relaxed ordering safely? We ask because our cluster is probably in that regime: the NICs are presumably at the default PCI_ATOMIC_MODE=0, yet the window is strictly ordered, and dispatch is limited to 12 GB/s at 32 SMs. **Part 2 (possible bug):scale-out tail counter stride (8 KiB) collapses the NIC's atomic rate.** [`EPSignals::scaleout_channel_signaled_tail`](https://github.com/deepseek-ai/DeepEP/blob/f6aa38fe06cd4a0489e7730365d4e0114970a2f3/deep_ep/include/deep_ep/layout/ep/workspace.cuh#L19)is dimensioned with the compile-time maximum rank count, so consecutive channels' tails are 8192 B apart. That stride reaches the wire unchanged — red_add_rel computes ptr - lsa_base_ptr, ncclGin_VASignalAdd carries it as a window offset, and GDAKI places it directly into the raddr of an ATOMIC_FA WQE. We observed that this appears to reduce the responder NIC's sustained atomic rate. To isolate the effect of this stride, we built a small standalone reproducer that does not depend on DeepEP, NCCL, or a GPU: 32 RC QPs, each issuing 8 B IBV_WR_ATOMIC_FETCH_AND_ADD on its own counter in host memory. The only variable is the per-QP counter stride. Atomics are posted in chains of 8 with only the last one signalled, since signalling every WR makes the requester's CPU the bottleneck and masks the effect. The results show a cliff between 1024 B and 4096 B，All rates in Mops (10^6 atomic ops/s). : stride | ConnectX-8 400G, 2 hosts | ConnectX-7 200G, different network | same CX7, loopback, no switch -- | -- | -- | -- 64 B | 24.33 | 18.32 | 19.65 256 B | 17.51 | 17.15 | 15.24 1024 B | 4.66 | 4.47 | 4.53 4096 B | 1.30 | 1.21 | 1.22 8192 B | 1.30 | 1.21 | 1.21 65536 B | 1.30 | 1.21 | 1.23 [atomic_stride_repro.c](https://github.com/user-attachments/files/33864290/atomic_stride_repro.c)

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#424

No description provided.

Rows
Columns