[GH-ISSUE #131] Ways to do overlapping of computation and commucation. #117

New issue

Closed

opened 2026-10-26 09:54:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:54:34 +00:00

Owner

Originally created by @kuozhang on GitHub (Apr 22, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/131

Hello, I'm trying to use DeepEP and intergate it into Megatron (with this commit github.com/NVIDIA/Megatron-LM@0d389f5e99), and accomplish overlapping of computation and commucation, I've noticed DeepEP is using previous_event and async_finish args in dispatch and combine methods to support the overlapping , whereas the FlexTokenDispatcher in that commit doesn't adopt any of them. For me, there are two ways to do that:

modify FlexTokenDispatcher and pass args previous_event, async_finish etc. to make use of DeepEP's overlapping interface。
keep FlexTokenDispatcher as it is, and manage the stream and event outside by myself before calling dispatch and combine under the context of torch.cuda.stream(), which means DeepEP's pevious_event, async_finish won't be used.

Can you guys give me some suggestions about the two ways in term of performance?

Originally created by @kuozhang on GitHub (Apr 22, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/131 Hello, I'm trying to use DeepEP and intergate it into Megatron (with this commit https://github.com/NVIDIA/Megatron-LM/commit/0d389f5e9935957b146c12b1dcdddc35cf9deb85), and accomplish overlapping of computation and commucation, I've noticed DeepEP is using previous_event and async_finish args in dispatch and combine methods to support the overlapping , whereas the FlexTokenDispatcher in that commit doesn't adopt any of them. For me, there are two ways to do that: * modify FlexTokenDispatcher and pass args previous_event, async_finish etc. to make use of DeepEP's overlapping interface。 * keep FlexTokenDispatcher as it is, and manage the stream and event outside by myself before calling dispatch and combine under the context of torch.cuda.stream(), which means DeepEP's pevious_event, async_finish won't be used. Can you guys give me some suggestions about the two ways in term of performance?

bsakaguchi closed this issue

2026-10-26 09:54:34 +00:00

bsakaguchi commented

2026-10-26 09:54:35 +00:00

Author

Owner

@LyricZhao commented on GitHub (Apr 25, 2025):

I highly recommend you to use the first way, as the current DeepEP design may fail to cover some edge cases for the second (e.g. some record_stream/reuse GPU mem problems).

In terms of performance, I guess no diff between these if they can perfectly overlap.

@LyricZhao commented on GitHub (Apr 25, 2025): I highly recommend you to use the first way, as the current DeepEP design may fail to cover some edge cases for the second (e.g. some `record_stream`/reuse GPU mem problems). In terms of performance, I guess no diff between these if they can perfectly overlap.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#117

No description provided.

Rows
Columns