mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #327] low latency test error with Separate profiling #229
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#229
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @shifangx on GitHub (Jul 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/327
Assert len(durations) % num_kernels_per_period == 0 in bench_kineto() failed for GB200, even for 1 node with 2 processes.
File "/DeepEP/tests/utils.py", line 212, in bench_kineto
assert len(durations) % num_kernels_per_period == 0
@sphish commented on GitHub (Jul 29, 2025):
We haven't encountered this issue. This assertion indicates that the number of events collected by torch profiler doesn't match the number of test iterations. Perhaps you can print all the profiled events to see where the problem occurs.
@MaoZiming commented on GitHub (Sep 15, 2025):
@shifangx I have seen similar issues. Did you manage to resolve it?
@shifangx commented on GitHub (Sep 15, 2025):
This issue has been solved by update to the newest commit of DeepEP.
@MaoZiming commented on GitHub (Sep 15, 2025):
Thank you. @shifangx Do you know which commit fixes this (or do you mean the latest commit)?
@MaoZiming commented on GitHub (Sep 15, 2025):
github.com/deepseek-ai/DeepEP@c5facf5c6f (diff-9e08e8c776)The fix seems to be the
torch.cuda.synchronize()inbench_kineto.Thank you