mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #261] normal kernels show significant performance gap between running test_internode.py and inference use #193
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#193
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @MistyAir on GitHub (Jun 26, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/261
I am trying to run DeepEP on 4xH100 node with RoCE network. The tests works fine, and I checked kernel duration. For example, each internode::combine takes about 3-4ms.
But when I use DeepEP in inference engine, the latency raised a lot (about 10-12 ms, could raise up to 50ms or more).
This is test result of test_internode.py
But in inference engine, I got this:
Each kernel took extremely long time
If I modify topk_idx before all dispatching (It seems does not matter balanced or not. different among tokens but the same among layers) I got even worse result:
I using DeepEP on commit
b8d90fb753I found that if run a kernel several times with the same handle (like we do in the tests), the kernel performance will turns much better.
@LyricZhao commented on GitHub (Jun 27, 2025):
Without handle, there will be a CPU synchronization, I am not sure whether you have some CPU issues (e.g. CPU core so busy, etc.).
You should make sure
topk_idxwell balanced, the traffic of RDMA and NVLink is somehow balanced.Or maybe try to add a
dist.barrierbefore dispatch/combine.@MistyAir commented on GitHub (Jun 27, 2025):
Thank you for replying. Maybe I will try barrier first.
BTW, I didn't observe the same issue while running EP16. Both EP16/EP32 have EPLB enabled.
@MistyAir commented on GitHub (Jun 30, 2025):
I added barrier in compute stream and it shows communication process itself didn't take that much time. Communication kernels last so long because some of ranks haven't finish their computation and other ranks were waiting them inside the combine kernel.