mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #107] Why does the execution time of a certain kernel increase by tens of times during the first few calls to the communication kernel? #98
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#98
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Autumn1998 on GitHub (Apr 2, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/107
For example, the inter-node combine consumed ~300ms in the first step while the combine kernel should be only ~5ms.
The cached_notify also used ~15ms while it should be ~60us.
The kernels worked well after the first layer, any idea for this >?
@LyricZhao commented on GitHub (Apr 3, 2025):
The progress of the GPUs should be synced, there might be some launch gap between ranks. And after the warmup stages (e.g. first layer), the barrier will do the sync.