[GH-ISSUE #526] Question about launch configs of normal kernels #338

New issue

Open

opened 2026-10-26 09:58:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:58:48 +00:00

Owner

Originally created by @Strivin0311 on GitHub (Dec 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/526

Hi~first of all, thank you for the amazing work !

I have some questions about the necessity of the launch configs for normal kernels.

I notice that for both intranode and internode normal kernels, you uses the same macro SETUP_LAUNCH_CONFIG to set the cudaLaunchAttribute with two attributes that affect computation and communication overlapping:

cudaLaunchAttributeCooperative (cooperative = 1), which enforces all the blocks to be launched simultaneously, resulting in the SM margin value left by the computation (persistent) kernels to be at least as large as the number of SMs used by comm kernels (num_sms), with the constraint "one block per SM" mentioned in the previous issue.
cudaLaunchAttributeClusterDimension (clusterDim.x = 2), which requires each pair of two SMs to be positioned in the same GPC, resulting in even larger SM margin value than num_sms, since leaving exact num_sms SMs still can not guarantee any GPC has left at least two SMs.

However, I've walked through the souce code of normal kernels, but found it a block-level kernel without the usage of any cluster features (e.g. cg::this_grid().sync or distributed shared memory), except for the TMA multicast used in tma_load_1d (and if I overlooked something, please point me to it).

It is generally admitted that SM margin seems too expensive to be set with a large value, especially quite a waste for the timeline range when the overlapped comm kernel is finished but the computation kernel has not.

And from my experience, NCCL only requires a small value of SM margin (4~8) to be fully overlapped with the computation kernel (while NCCL has incorporated CGA to leverage cluster capabilities since version 2.16, setting NCCL_CGA_CLUSTER_SIZE=1 is acceptable and still delivers excellent performance.).

Therefore, I wonder the necessity to attach these two kernels with two aggressive attributes above:

Are the two attributes kept solely for TMA efficiency? If so, what degradation should we expect when they are removed ?
Are there any other reasons to keep these two attributes ? If so, is there any chance to design a good-enough normal kernel that avoids these hard constraints like NCCL ?

Looking forward to your insights !

Originally created by @Strivin0311 on GitHub (Dec 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/526 Hi~first of all, thank you for the amazing work ! I have some questions about the necessity of the launch configs for normal kernels. I notice that for both `intranode` and `internode` normal kernels, you uses the same macro `SETUP_LAUNCH_CONFIG` to set the `cudaLaunchAttribute` with two attributes that **affect computation and communication overlapping**: 1. `cudaLaunchAttributeCooperative` (`cooperative = 1`), which enforces all the blocks to be launched simultaneously, resulting in **the SM margin value left by the computation (*persistent*) kernels to be at least as large as the number of SMs used by comm kernels (`num_sms`)**, with the constraint "one block per SM" mentioned in the previous [issue](https://github.com/deepseek-ai/DeepEP/issues/44). 2. `cudaLaunchAttributeClusterDimension` (`clusterDim.x = 2`), which requires each pair of two SMs to be positioned in the same GPC, resulting in **even larger SM margin value than `num_sms`**, since leaving exact `num_sms` SMs still can not guarantee any GPC has left at least two SMs. However, I've walked through the souce code of normal kernels, but found it a block-level kernel without the usage of any cluster features (e.g. `cg::this_grid().sync` or distributed shared memory), except for the TMA multicast used in `tma_load_1d` (*and if I overlooked something, please point me to it*). It is generally admitted that SM margin seems too expensive to be set with a large value, especially quite a waste for the timeline range when the overlapped comm kernel is finished but the computation kernel has not. And from my experience, NCCL only requires a small value of SM margin (`4~8`) to be fully overlapped with the computation kernel (*while NCCL has incorporated CGA to leverage cluster capabilities since version 2.16, setting `NCCL_CGA_CLUSTER_SIZE=1` is acceptable and still delivers excellent performance.*). Therefore, I wonder the necessity to attach these two kernels with two aggressive attributes above: 1. Are the two attributes kept solely for TMA efficiency? If so, what degradation should we expect when they are removed ? 2. Are there any other reasons to keep these two attributes ? If so, is there any chance to design a good-enough normal kernel that avoids these hard constraints like NCCL ? Looking forward to your insights !

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#338

No description provided.

Rows
Columns