[PR #153] [MERGED] Shuffling the starting index of target rank for different ranks and channels #444

New issue

Closed

opened 2026-10-26 10:00:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:49 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/153
Author: @wangfakang
Created: 5/12/2025
Status: ✅ Merged
Merged: 5/12/2025
Merged by: @LyricZhao

Base: main ← Head: opt-shuffled_dst

📝 Commits (1)

63c29d0 To mitigate incast congestion, shuffle the starting index of target rank for different ranks and channels

📊 Changes

1 file changed (+2 additions, -1 deletions)

View changed files

📝 csrc/kernels/internode.cu (+2 -1)

📄 Description

To mitigate incast congestion, shuffle the starting index of target rank for different ranks and channels. cc @LyricZhao

Example

When the number of ranks and channels is unequal (e.g., 4 ranks and 5 channels), distributing the initial target rank positions through rank-channel shuffling can balance the send/receive network traffic per rank, thereby mitigating incast congestion.
The following demonstrates the target rank positions computed per iteration for each rank and channel under two shuffling modes.

channel shuffle

Round-0

RDMA-RANK/CHANNEL-ID	Channel1	Channel2	Channel3
Rank0	1	2	3
Rank1	1	2	3
Rank2	1	2	3
Rank3	1	2	3

Round-1

RDMA-RANK/CHANNEL-ID	Channel0	Channel1	Channel2	Channel4
Rank0	1	2	3	1
Rank1	1	2	3	1
Rank2	1	2	3	1
Rank3	1	2	3	1

channel-rank shuffle

Round-0

RDMA-RANK/CHANNEL-ID	Channel0	Channel1	Channel2	Channel3	Channel4
Rank0	0	1	2	3	0
Rank1	1	2	3	0	1
Rank2	2	3	0	1	2
Rank3	3	0	1	2	3

Round-1

RDMA-RANK/CHANNEL-ID	Channel0	Channel1	Channel2	Channel3	Channel4
Rank0	1	2	3	0	1
Rank1	2	3	0	1	2
Rank2	3	0	1	2	3
Rank3	0	1	2	3	0

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/153 **Author:** [@wangfakang](https://github.com/wangfakang) **Created:** 5/12/2025 **Status:** ✅ Merged **Merged:** 5/12/2025 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `opt-shuffled_dst` --- ### 📝 Commits (1) - [`63c29d0`](https://github.com/deepseek-ai/DeepEP/commit/63c29d06a0821579963e13824390ab0922ec4910) To mitigate incast congestion, shuffle the starting index of target rank for different ranks and channels ### 📊 Changes **1 file changed** (+2 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/internode.cu` (+2 -1) </details> ### 📄 Description To mitigate incast congestion, shuffle the starting index of target rank for different ranks and channels. cc @LyricZhao # Example When the number of ranks and channels is unequal (e.g., 4 ranks and 5 channels), distributing the initial target rank positions through rank-channel shuffling can balance the send/receive network traffic per rank, thereby mitigating incast congestion. The following demonstrates the target rank positions computed per iteration for each rank and channel under two shuffling modes. ## channel shuffle Round-0 | RDMA-RANK/CHANNEL-ID | Channel0 | Channel1 |Channel2 |Channel3 |Channel4 | | :----: | :----: | :----: | :----: | :----: | :----: | | Rank0 |0 |1 |2 |3 |0 | | Rank1 |0 |1 |2 |3 |0 | | Rank2 |0 |1 |2 |3 |0 | | Rank3 |0 |1 |2 |3 |0 | Round-1 | RDMA-RANK/CHANNEL-ID | Channel0 | Channel1 |Channel2 |Channel3 |Channel4 | | :----: | :----: | :----: | :----: | :----: | :----: | | Rank0 |1 |2 |3 |0 |1 | | Rank1 |1 |2 |3 |0 |1 | | Rank2 |1 |2 |3 |0 |1 | | Rank3 |1 |2 |3 |0 |1 | ## channel-rank shuffle Round-0 | RDMA-RANK/CHANNEL-ID | Channel0 | Channel1 |Channel2 |Channel3 |Channel4 | | :----: | :----: | :----: | :----: | :----: | :----: | | Rank0 |0 |1 |2 |3 |0 | | Rank1 |1 |2 |3 |0 |1 | | Rank2 |2 |3 |0 |1 |2 | | Rank3 |3 |0 |1 |2 |3 | Round-1 | RDMA-RANK/CHANNEL-ID | Channel0 | Channel1 |Channel2 |Channel3 |Channel4 | | :----: | :----: | :----: | :----: | :----: | :----: | | Rank0 |1 |2 |3 |0 |1 | | Rank1 |2 |3 |0 |1 |2 | | Rank2 |3 |0 |1 |2 |3 | | Rank3 |0 |1 |2 |3 |0 | --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 10:00:49 +00:00