deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #651] ElasticBuffer.barrier doesn't ensure global sync in hybrid mode #383

New issue
Closed
opened 2026-10-26 09:59:49 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:59:49 +00:00
Owner
Copy link

Originally created by @fishautumn on GitHub (May 30, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/651

Problem

We found test_ep.y first round dispatch is usually slower than later.
It uses ElasticBuffer.barrier by default in bench_kineto, and can be walk around by switching to dist.all_reduce.

Reason

in gpu_barrier implementation:
github.com/deepseek-ai/DeepEP@d4f41e4e93/deep_ep/include/deep_ep/common/comm.cuh (L213)

        // Do scaleup and scaleout barrier in parallel
        EP_DEVICE_ASSERT(kNumSMs >= 2 and "At least 2 SMs for a hybrid barrier");
        if (sm_idx == 0) {
            // First SM do the scaleup barrier
           ...
        } else {
            // The remaining SMs do the scaleout barrier
            ...
        }

It uses 2 SM for rail and LSA domain separately.
If a rank enters barrier quite late, members of its rail and LSA domain will be blocked;
while other ranks won't be blocked, and their dispatch kernel will have long execution time.

+------ node0 -------+  +------- node1 ------+
| +--------------+   |  |   +--------------+ |
| | rank0 (----) | <-|--|-- | rank2 (wait) | | ...... rail0
| +--------------+   |  |   +--------------+ |
|        ^           |  |          ^         |
|        |           |  |          |         |
|        |           |  |          |         |
| +--------------+   |  |   +--------------+ |
| | rank1 (wait) | <-|--|-- | rank3 (exit) | | ...... rail1
| +--------------+   |  |   +--------------+ |
+--------------------+  +--------------------+
  • rank0 hasn't enter barrier
  • rank1 and rank2 enter and wait for rank0's signal
  • rank3 enters barrier, receives signals from rank1 and rank2, then exits

Possible Solution

Run rail and LSA domain sequentially

Originally created by @fishautumn on GitHub (May 30, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/651 ## Problem We found `test_ep.y` first round dispatch is usually slower than later. It uses `ElasticBuffer.barrier` by default in `bench_kineto`, and can be walk around by switching to `dist.all_reduce`. ## Reason in `gpu_barrier` implementation: https://github.com/deepseek-ai/DeepEP/blob/d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb/deep_ep/include/deep_ep/common/comm.cuh#L213 ```cpp // Do scaleup and scaleout barrier in parallel EP_DEVICE_ASSERT(kNumSMs >= 2 and "At least 2 SMs for a hybrid barrier"); if (sm_idx == 0) { // First SM do the scaleup barrier ... } else { // The remaining SMs do the scaleout barrier ... } ``` It uses 2 SM for rail and LSA domain separately. If a rank enters barrier quite late, members of its rail and LSA domain will be blocked; while other ranks won't be blocked, and their dispatch kernel will have long execution time. ``` +------ node0 -------+ +------- node1 ------+ | +--------------+ | | +--------------+ | | | rank0 (----) | <-|--|-- | rank2 (wait) | | ...... rail0 | +--------------+ | | +--------------+ | | ^ | | ^ | | | | | | | | | | | | | | +--------------+ | | +--------------+ | | | rank1 (wait) | <-|--|-- | rank3 (exit) | | ...... rail1 | +--------------+ | | +--------------+ | +--------------------+ +--------------------+ ``` * rank0 hasn't enter barrier * rank1 and rank2 enter and wait for rank0's signal * rank3 enters barrier, receives signals from rank1 and rank2, then exits ## Possible Solution Run rail and LSA domain sequentially
bsakaguchi commented 2026-10-26 09:59:50 +00:00
Author
Owner
Copy link

@gangxie112 commented on GitHub (Jun 1, 2026):

@sphish

<!-- gh-comment-id:4589190253 --> @gangxie112 commented on GitHub (Jun 1, 2026): @sphish
bsakaguchi commented 2026-10-26 09:59:50 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 7, 2026):

Fixed in https://github.com/deepseek-ai/DeepEP/pull/674. Thank you!

<!-- gh-comment-id:4901472683 --> @sphish commented on GitHub (Jul 7, 2026): Fixed in https://github.com/deepseek-ai/DeepEP/pull/674. Thank you!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#383
No description provided.