deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #794] Pack channel tail signals by runtime rank count (fix ~2x v2.5 hybrid dispatch regression) #789

New issue

Open

opened 2026-10-27 21:06:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:06:22 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/794
Author: @sssssakuraaaaa
Created: 10/23/2026
Status: 🔄 Open

Base: main ← Head: fix/dense-tail-signal-layout

📝 Commits (1)

f04a242 Pack channel tail signals by runtime rank count (fix v2.5 hybrid dispatch regression)

📊 Changes

1 file changed (+10 additions, -2 deletions)

View changed files

📝 deep_ep/include/deep_ep/layout/ep/workspace.cuh (+10 -2)

📄 Description

Summary

The v2.5 refactor (#763, commit def8651) replaced the hand-laid, runtime-strided
tail-signal buffers with static 2D arrays in EPSignals
(deep_ep/include/deep_ep/layout/ep/workspace.cuh):

int64_t scaleout_channel_signaled_tail[kNumMaxChannels][kNumMaxRanks]; // kNumMaxRanks = 1024
int     channel_scaleup_tail          [kNumMaxChannels][kNumMaxRanks];

and the accessors index them directly as [channel_idx][rank_idx]. The row stride is
therefore the compile-time inner dimension kNumMaxRanks = 1024, so consecutive
channels' tail signals are kNumMaxRanks elements apart — 8 KiB for the int64_t
scaleout tail.

These tails are a producer/consumer handshake: the scaleout warp does an RDMA remote
atomic-add (gin.red_add_rel) on the peer's scaleout_channel_signaled_tail[channel][rank],
and the forward warp ld_acquire-polls the same counter before consuming each chunk.

With a small runtime rank count (e.g. EP16 → num_scaleout_ranks = 2), the per-channel
counters end up spread 8 KiB apart. That stride reaches the wire unchanged — red_add_rel
computes ptr - lsa_base_ptr, ncclGin carries it as a window offset, and the GDAKI backend
places it directly into the raddr of the ATOMIC_FA WQE — and the responder NIC's
sustained atomic rate collapses as the per-counter stride grows. The forward warp's
tail-wait (fwdwait) balloons ~2.5x and hybrid inter-node dispatch time roughly doubles.
This is a NIC atomic-rate / remote-address problem, not a bandwidth problem (payload bandwidth
is unchanged), and raising num_sms does not help.

This matches the independent, hardware-level finding in #793, whose standalone reproducer (32
RC QPs issuing 8 B IBV_WR_ATOMIC_FETCH_AND_ADD, no GPU / NCCL / DeepEP involved) shows the
responder atomic rate dropping off a cliff between 1 KiB and 4 KiB of per-counter stride (e.g.
~17 Mops at 256 B → ~1.3 Mops at 4–8 KiB), and reproduces even in loopback with no switch.

Fix

Pack both tail arrays densely by the runtime rank count, restoring the pre-#763 (v2.1.0)
layout. This reuses the exact indexing idiom already present in this file
(get_scaleout_rank_count_ptr: scaleout_rank_idx * num_scaleup_ranks + scaleup_rank_idx):

return &signals.scaleout_channel_signaled_tail[0][0] + (channel_idx * num_scaleout_ranks + scaleout_rank_idx);
return &signals.channel_scaleup_tail[0][0]          + (channel_idx * num_scaleup_ranks  + scaleup_rank_idx);

The max flat index kNumMaxChannels * num_scaleout_ranks stays within the
kNumMaxChannels * kNumMaxRanks backing storage, since num_scaleout_ranks and
num_scaleup_ranks are both <= kNumMaxRanks. All current callers already go through these
two accessors, so no other call sites change.

Benchmark

B300 + NCCL 2.32.3 + CUDA 13.0, EP16 (2 nodes x 8 GPU) hybrid, num_sms=32,
EP_OVERRIDE_RDMA_SL=162, token sweep 1024→16384:

dispatch time: 408/718/1080/2068/3872 us → 187/299/464/777/1426 us (~2.2x–2.7x faster),
back on par with v2.1.0 (dispatch SO recovers to ~82/102/132/157/171 GB/s).
combine time: ~1.07x–1.53x faster.

Repro

HW: NVIDIA B300, 2 nodes x 8 GPU (EP16), hybrid mode (--allow-hybrid-mode 1)
SW: NCCL 2.32.3 (GIN backend), CUDA 13.0
Test: tests/ep/test_ep.py, experts=256, hidden=7168, topk=8, num_sms=32,
EP_OVERRIDE_RDMA_SL=162, token sweep

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/794 **Author:** [@sssssakuraaaaa](https://github.com/sssssakuraaaaa) **Created:** 10/23/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/dense-tail-signal-layout` --- ### 📝 Commits (1) - [`f04a242`](https://github.com/deepseek-ai/DeepEP/commit/f04a24252c92d6ba25fc7a1a34a97a8fd93ae2ba) Pack channel tail signals by runtime rank count (fix v2.5 hybrid dispatch regression) ### 📊 Changes **1 file changed** (+10 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/include/deep_ep/layout/ep/workspace.cuh` (+10 -2) </details> ### 📄 Description ## Summary The v2.5 refactor (#763, commit `def8651`) replaced the hand-laid, runtime-strided tail-signal buffers with static 2D arrays in `EPSignals` (`deep_ep/include/deep_ep/layout/ep/workspace.cuh`): ```cpp int64_t scaleout_channel_signaled_tail[kNumMaxChannels][kNumMaxRanks]; // kNumMaxRanks = 1024 int channel_scaleup_tail [kNumMaxChannels][kNumMaxRanks]; ``` and the accessors index them directly as `[channel_idx][rank_idx]`. The row stride is therefore the **compile-time** inner dimension `kNumMaxRanks = 1024`, so consecutive channels' tail signals are `kNumMaxRanks` elements apart — **8 KiB** for the `int64_t` scaleout tail. These tails are a producer/consumer handshake: the scaleout warp does an RDMA remote atomic-add (`gin.red_add_rel`) on the peer's `scaleout_channel_signaled_tail[channel][rank]`, and the forward warp `ld_acquire`-polls the same counter before consuming each chunk. With a small **runtime** rank count (e.g. EP16 → `num_scaleout_ranks = 2`), the per-channel counters end up spread 8 KiB apart. That stride reaches the wire unchanged — `red_add_rel` computes `ptr - lsa_base_ptr`, `ncclGin` carries it as a window offset, and the GDAKI backend places it directly into the `raddr` of the `ATOMIC_FA` WQE — and **the responder NIC's sustained atomic rate collapses as the per-counter stride grows**. The forward warp's tail-wait (`fwdwait`) balloons ~2.5x and **hybrid inter-node dispatch time roughly doubles**. This is a NIC atomic-rate / remote-address problem, not a bandwidth problem (payload bandwidth is unchanged), and raising `num_sms` does not help. This matches the independent, hardware-level finding in #793, whose standalone reproducer (32 RC QPs issuing 8 B `IBV_WR_ATOMIC_FETCH_AND_ADD`, no GPU / NCCL / DeepEP involved) shows the responder atomic rate dropping off a cliff between 1 KiB and 4 KiB of per-counter stride (e.g. ~17 Mops at 256 B → ~1.3 Mops at 4–8 KiB), and reproduces even in loopback with no switch. ## Fix Pack both tail arrays **densely by the runtime rank count**, restoring the pre-#763 (v2.1.0) layout. This reuses the exact indexing idiom already present in this file (`get_scaleout_rank_count_ptr`: `scaleout_rank_idx * num_scaleup_ranks + scaleup_rank_idx`): ```cpp return &signals.scaleout_channel_signaled_tail[0][0] + (channel_idx * num_scaleout_ranks + scaleout_rank_idx); return &signals.channel_scaleup_tail[0][0] + (channel_idx * num_scaleup_ranks + scaleup_rank_idx); ``` The max flat index `kNumMaxChannels * num_scaleout_ranks` stays within the `kNumMaxChannels * kNumMaxRanks` backing storage, since `num_scaleout_ranks` and `num_scaleup_ranks` are both `<= kNumMaxRanks`. All current callers already go through these two accessors, so no other call sites change. ## Benchmark B300 + NCCL 2.32.3 + CUDA 13.0, EP16 (2 nodes x 8 GPU) hybrid, `num_sms=32`, `EP_OVERRIDE_RDMA_SL=162`, token sweep 1024→16384: - **dispatch time**: 408/718/1080/2068/3872 us → **187/299/464/777/1426 us** (~2.2x–2.7x faster), back on par with v2.1.0 (dispatch SO recovers to ~82/102/132/157/171 GB/s). - **combine time**: ~1.07x–1.53x faster. ## Repro - HW: NVIDIA B300, 2 nodes x 8 GPU (EP16), hybrid mode (`--allow-hybrid-mode 1`) - SW: NCCL 2.32.3 (GIN backend), CUDA 13.0 - Test: `tests/ep/test_ep.py`, experts=256, hidden=7168, topk=8, num_sms=32, `EP_OVERRIDE_RDMA_SL=162`, token sweep --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:06:22 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#789

No description provided.

Rows
Columns