deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #63] About the setting of kNumWarpsPerGroup and kNumWarpGroups #59

New issue

Closed

opened 2026-10-26 09:53:32 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 09:53:32 +00:00

Owner

Originally created by @GitAlice123 on GitHub (Mar 10, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/63

Why exactly you set kNumWarpsPerGroup=10, kNumWarpGroups=3?

Originally created by @GitAlice123 on GitHub (Mar 10, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/63 ![Image](https://github.com/user-attachments/assets/bbc8ddf7-bf83-42a2-a4c3-3d9f453bc61b) Why exactly you set kNumWarpsPerGroup=10, kNumWarpGroups=3?

bsakaguchi closed this issue

2026-10-26 09:53:32 +00:00

bsakaguchi commented

2026-10-26 09:53:33 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 11, 2025):

V3/R1 has 288 experts (256 + extra duplicate load-balance experts) and each warp-group is responsible for 1 expert, using 4 groups → 288 / 4 = 72 SMs, using 3 groups -> 96 SMs, using 2 groups -> 144 SMs (> 132 SMs, bad, cooperative launch GG), for better device occupancy (96 SMs for communication only is not bad, or 132 - 96 SMs can do shared experts' computation for extreme utilization), we choose 3 groups;
To make more warps work (more in-SM occupancy), we choose 10 warps per group (10 * 3 = 30 warps, maximum is 32 warps per block);
For dispatch FP8 cast, using V3 hidden 7168, BF16 (2 bytes), 32 threads per warp, int4 read (16 bytes), we will use 7168 * 2 / (32 * 16) = 28 warp-rounds to cast, if we are using less than 28 warps then it will cost 2 rounds, so 10 * 3 - 1 = 29 (- 1 due to some other jobs for the extra warp, please read the code), 29 > 28, so we only require all warps to cast one round.

Overall, it is a delicate tradeoff for V3/R1 towards extreme performance, you can change them for your own model settings.

@LyricZhao commented on GitHub (Mar 11, 2025): 1. V3/R1 has 288 experts (256 + extra duplicate load-balance experts) and each warp-group is responsible for 1 expert, using 4 groups → 288 / 4 = 72 SMs, using 3 groups -> 96 SMs, using 2 groups -> 144 SMs (> 132 SMs, bad, cooperative launch GG), for better device occupancy (96 SMs for communication only is not bad, or 132 - 96 SMs can do shared experts' computation for extreme utilization), we choose 3 groups; 2. To make more warps work (more in-SM occupancy), we choose 10 warps per group (10 * 3 = 30 warps, maximum is 32 warps per block); 3. For dispatch FP8 cast, using V3 hidden 7168, BF16 (2 bytes), 32 threads per warp, `int4` read (16 bytes), we will use `7168 * 2 / (32 * 16) = 28` warp-rounds to cast, if we are using less than 28 warps then it will cost 2 rounds, so `10 * 3 - 1 = 29` (`- 1` due to some other jobs for the extra warp, please read the code), 29 > 28, so we only require all warps to cast one round. Overall, it is a delicate tradeoff for V3/R1 towards extreme performance, you can change them for your own model settings.

bsakaguchi commented

2026-10-26 09:53:33 +00:00

Author

Owner

@alpha-baby commented on GitHub (Jun 13, 2025):

V3/R1 has 288 experts (256 + extra duplicate load-balance experts) and each warp-group is responsible for 1 expert, using 4 groups → 288 / 4 = 72 SMs, using 3 groups -> 96 SMs, using 2 groups -> 144 SMs (> 132 SMs, bad, cooperative launch GG), for better device occupancy (96 SMs for communication only is not bad, or 132 - 96 SMs can do shared experts' computation for extreme utilization), we choose 3 groups;

To make more warps work (more in-SM occupancy), we choose 10 warps per group (10 * 3 = 30 warps, maximum is 32 warps per block);

For dispatch FP8 cast, using V3 hidden 7168, BF16 (2 bytes), 32 threads per warp, int4 read (16 bytes), we will use 7168 * 2 / (32 * 16) = 28 warp-rounds to cast, if we are using less than 28 warps then it will cost 2 rounds, so 10 * 3 - 1 = 29 (- 1 due to some other jobs for the extra warp, please read the code), 29 > 28, so we only require all warps to cast one round.

Overall, it is a delicate tradeoff for V3/R1 towards extreme performance, you can change them for your own model settings.

Can we automatically find the best configuration of kNumWarpsPerGroup and kNumWarpGroups according to this rule? If so, I can submit a PR to support this capability, so that we can compile only one DeeoEp program to adapt to different types of GPUs.

@alpha-baby commented on GitHub (Jun 13, 2025): > 1. V3/R1 has 288 experts (256 + extra duplicate load-balance experts) and each warp-group is responsible for 1 expert, using 4 groups → 288 / 4 = 72 SMs, using 3 groups -> 96 SMs, using 2 groups -> 144 SMs (> 132 SMs, bad, cooperative launch GG), for better device occupancy (96 SMs for communication only is not bad, or 132 - 96 SMs can do shared experts' computation for extreme utilization), we choose 3 groups; > 2. To make more warps work (more in-SM occupancy), we choose 10 warps per group (10 * 3 = 30 warps, maximum is 32 warps per block); > 3. For dispatch FP8 cast, using V3 hidden 7168, BF16 (2 bytes), 32 threads per warp, `int4` read (16 bytes), we will use `7168 * 2 / (32 * 16) = 28` warp-rounds to cast, if we are using less than 28 warps then it will cost 2 rounds, so `10 * 3 - 1 = 29` (`- 1` due to some other jobs for the extra warp, please read the code), 29 > 28, so we only require all warps to cast one round. > > Overall, it is a delicate tradeoff for V3/R1 towards extreme performance, you can change them for your own model settings. Can we automatically find the best configuration of kNumWarpsPerGroup and kNumWarpGroups according to this rule? If so, I can submit a PR to support this capability, so that we can compile only one DeeoEp program to adapt to different types of GPUs.

bsakaguchi commented

2026-10-26 09:53:33 +00:00

Author

Owner

@LyricZhao commented on GitHub (Jun 16, 2025):

Can we automatically find the best configuration of kNumWarpsPerGroup and kNumWarpGroups according to this rule? If so, I can submit a PR to support this capability, so that we can compile only one DeeoEp program to adapt to different types of GPUs.

Fixed in #213, close it now.

@LyricZhao commented on GitHub (Jun 16, 2025): > Can we automatically find the best configuration of kNumWarpsPerGroup and kNumWarpGroups according to this rule? If so, I can submit a PR to support this capability, so that we can compile only one DeeoEp program to adapt to different types of GPUs. Fixed in #213, close it now.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#59

No description provided.

Rows
Columns