deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #240] UE8M0(PR206) features cause severe a regression issue and cause low-latency stuck #184

New issue
Closed
opened 2026-10-26 09:56:03 +00:00 by bsakaguchi · 17 comments
bsakaguchi commented 2026-10-26 09:56:03 +00:00
Owner
Copy link

Originally created by @jeffye-dev on GitHub (Jun 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/240

In recent SGLANG PD disaggregation integration tests, we found it 100% stuck in DeepEP dispatch-combine call.
And the low_latency.py unit test stack looks as below when it's stuck:

__torch_function__ (torch/utils/_device.py:104)
calc_diff (utils.py:34)
test_main (test_low_latency.py:96)
test_loop (test_low_latency.py:170)
_wrap (torch/multiprocessing/spawn.py:90)
run (multiprocessing/process.py:108)
_bootstrap (multiprocessing/process.py:314)
_main (multiprocessing/spawn.py:129)
spawn_main (multiprocessing/spawn.py:116)
<module> (<string>:1)

With deeply regression tests for historical commits, I addressed this PR : https://github.com/deepseek-ai/DeepEP/pull/206

@shifangx would you please take a look at this or resolve it?

Originally created by @jeffye-dev on GitHub (Jun 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/240 In recent SGLANG PD disaggregation integration tests, we found it 100% stuck in DeepEP dispatch-combine call. And the low_latency.py unit test stack looks as below when it's stuck: > __torch_function__ (torch/utils/_device.py:104) calc_diff (utils.py:34) test_main (test_low_latency.py:96) test_loop (test_low_latency.py:170) _wrap (torch/multiprocessing/spawn.py:90) run (multiprocessing/process.py:108) _bootstrap (multiprocessing/process.py:314) _main (multiprocessing/spawn.py:129) spawn_main (multiprocessing/spawn.py:116) <module> (<string>:1) With deeply regression tests for historical commits, I addressed this PR : https://github.com/deepseek-ai/DeepEP/pull/206 @shifangx would you please take a look at this or resolve it?
bsakaguchi commented 2026-10-26 09:56:03 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jun 21, 2025):

I will take a look at this.

<!-- gh-comment-id:2993229711 --> @shifangx commented on GitHub (Jun 21, 2025): I will take a look at this.
bsakaguchi commented 2026-10-26 09:56:04 +00:00
Author
Owner
Copy link

@jeffye-dev commented on GitHub (Jun 25, 2025):

any progress ?

<!-- gh-comment-id:3002360991 --> @jeffye-dev commented on GitHub (Jun 25, 2025): any progress ?
bsakaguchi commented 2026-10-26 09:56:04 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jun 26, 2025):

I have not been able to reproduce the issue yet.
This test case requires the use of IBGDA and needs the cluster administrator to help with the configuration. I have just contacted the cluster to urge the administrator to assist with the setup.

<!-- gh-comment-id:3007616331 --> @shifangx commented on GitHub (Jun 26, 2025): I have not been able to reproduce the issue yet. This test case requires the use of IBGDA and needs the cluster administrator to help with the configuration. I have just contacted the cluster to urge the administrator to assist with the setup.
bsakaguchi commented 2026-10-26 09:56:05 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Jun 27, 2025):

I am also experiencing the same hang issue. In my case, it occurs with a setup of 4 machines (H20*8), where 2 machines function normally. The program is also stuck in calc_diff

Image

<!-- gh-comment-id:3012190601 --> @polarstormx commented on GitHub (Jun 27, 2025): I am also experiencing the same hang issue. In my case, it occurs with a setup of 4 machines (H20*8), where 2 machines function normally. The program is also stuck in calc_diff ![Image](https://github.com/user-attachments/assets/7bfd41f1-0ff5-4887-90b8-aa75ba007b4d)
bsakaguchi commented 2026-10-26 09:56:06 +00:00
Author
Owner
Copy link

@jeffye-dev commented on GitHub (Jul 2, 2025):

I use 6*H100 with IB network cards to have low-latency tests, it's 100% reproducible. This stops me using the latest version of DeepEP. I have to use the older DeepEP.

Would you revert this commit if it cannot be resolved in short term? @shifangx @zhyncs

<!-- gh-comment-id:3026330488 --> @jeffye-dev commented on GitHub (Jul 2, 2025): I use 6*H100 with IB network cards to have low-latency tests, it's 100% reproducible. This stops me using the latest version of DeepEP. I have to use the older DeepEP. Would you revert this commit if it cannot be resolved in short term? @shifangx @zhyncs
bsakaguchi commented 2026-10-26 09:56:07 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jul 3, 2025):

I have tested on GB200 with 4 nodes, and each node has 4GPUs, test passed.
I can not reproduce this issue on GB200.
@jeffye-dev @polarstormx , Can you help to do some test on your system?
During testing, we will go through all possible value combinations of return_recv_hook, dispatch_use_fp8, round_scale, and use_ue8m0.
（1）Could you help record which combinations cause the tests to fail?
（2）Can you try to skip the test for round_scale==True?
in test_main(), skip test for round_scale, just as the following. and print return_recv_hook, dispatch_use_fp8, round_scale, use_ue8m0
Image

<!-- gh-comment-id:3029980469 --> @shifangx commented on GitHub (Jul 3, 2025): I have tested on GB200 with 4 nodes, and each node has 4GPUs, test passed. I can not reproduce this issue on GB200. @jeffye-dev @polarstormx , Can you help to do some test on your system? During testing, we will go through all possible value combinations of return_recv_hook, dispatch_use_fp8, round_scale, and use_ue8m0. （1）Could you help record which combinations cause the tests to fail? （2）Can you try to skip the test for round_scale==True? in test_main(), skip test for round_scale, just as the following. and print return_recv_hook, dispatch_use_fp8, round_scale, use_ue8m0 ![Image](https://github.com/user-attachments/assets/fe94f9ea-b37d-497e-b04f-fbd99d8b4e9e)
bsakaguchi commented 2026-10-26 09:56:07 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Jul 3, 2025):

@shifangx
I have also tested on 4*4 GB200, and it runs successfully.

Here's my tests on 4*8 H20.

When set do_check= False or skip round_scale==True, the test can run successfully.

It hangs before: return_recv_hook: False, dispatch_use_fp8: True, round_scale: True, use_ue8m0: False

Image

Image

<!-- gh-comment-id:3030264511 --> @polarstormx commented on GitHub (Jul 3, 2025): @shifangx I have also tested on 4*4 GB200, and it runs successfully. Here's my tests on 4*8 H20. When set do_check= False or skip round_scale==True, the test can run successfully. It hangs before: return_recv_hook: False, dispatch_use_fp8: True, round_scale: True, use_ue8m0: False ![Image](https://github.com/user-attachments/assets/7b1f57d0-950c-46ed-a9d0-7adacb456f36) ![Image](https://github.com/user-attachments/assets/edd0c17f-1466-45b5-91e5-5d5e061523ea)
bsakaguchi commented 2026-10-26 09:56:08 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jul 3, 2025):

@polarstormx Thank you very much for your feedback.

<!-- gh-comment-id:3030291579 --> @shifangx commented on GitHub (Jul 3, 2025): @polarstormx Thank you very much for your feedback.
bsakaguchi commented 2026-10-26 09:56:08 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jul 3, 2025):

I use 6*H100 with IB network cards to have low-latency tests, it's 100% reproducible. This stops me using the latest version of DeepEP. I have to use the older DeepEP.

Would you revert this commit if it cannot be resolved in short term? @shifangx @zhyncs

Hi, @jeffye-dev, This issue may only occur when round_scale is set to True.
Maybe you can still use the latest version on H100, by making sure round_scale==False.
If you still encounter issues after setting round_scale to False, please leave a comment here.

<!-- gh-comment-id:3030307368 --> @shifangx commented on GitHub (Jul 3, 2025): > I use 6*H100 with IB network cards to have low-latency tests, it's 100% reproducible. This stops me using the latest version of DeepEP. I have to use the older DeepEP. > > Would you revert this commit if it cannot be resolved in short term? [@shifangx](https://github.com/shifangx) [@zhyncs](https://github.com/zhyncs) Hi, @jeffye-dev, This issue may only occur when round_scale is set to True. Maybe you can still use the latest version on H100, by making sure round_scale==False. If you still encounter issues after setting round_scale to False, please leave a comment here.
bsakaguchi commented 2026-10-26 09:56:08 +00:00
Author
Owner
Copy link

@0x777a6c commented on GitHub (Jul 3, 2025):

It seems to be the same problem in #209 . And the reporter mentioned that it may be caused by wrong envoironment variables.

<!-- gh-comment-id:3030448528 --> @0x777a6c commented on GitHub (Jul 3, 2025): It seems to be the same problem in #209 . And the reporter mentioned that it may be caused by wrong envoironment variables.
bsakaguchi commented 2026-10-26 09:56:08 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jul 5, 2025):

I have also tested on 4*4 GB200, and it runs successfully.

@polarstormx , what about 8*4 GB200?
I have tested on GB200. It runs successfully on 2*4 GB200 ,4*4 GB200, 6*4 GB200, but hang on 8*4 GB200 and 12*4 GB200.

<!-- gh-comment-id:3037671378 --> @shifangx commented on GitHub (Jul 5, 2025): > I have also tested on 4*4 GB200, and it runs successfully. @polarstormx , what about `8*4 GB200`? I have tested on GB200. It runs successfully on `2*4 GB200` ,`4*4 GB200`, `6*4 GB200`, but hang on `8*4 GB200` and `12*4 GB200`.
bsakaguchi commented 2026-10-26 09:56:09 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Jul 7, 2025):

what about 8*4 GB200

@shifangx Unfortunately, I only have a 4x4 GB200 , so I can't test the larger configuration

<!-- gh-comment-id:3043264732 --> @polarstormx commented on GitHub (Jul 7, 2025): > what about `8*4 GB200` @shifangx Unfortunately, I only have a 4x4 GB200 , so I can't test the larger configuration
bsakaguchi commented 2026-10-26 09:56:09 +00:00
Author
Owner
Copy link

@kzlxd commented on GitHub (Jul 8, 2025):

what the root cause？

<!-- gh-comment-id:3047793019 --> @kzlxd commented on GitHub (Jul 8, 2025): what the root cause？
bsakaguchi commented 2026-10-26 09:56:09 +00:00
Author
Owner
Copy link

@shifangx commented on GitHub (Jul 8, 2025):

root cause:
One rank fails at an assert and then exits.
All the other ranks are waiting for that rank at some point, which causes the hang.

I ran experiments using 12*4 GB200s and found that after commenting out this assert, the program no longer hangs.

github.com/deepseek-ai/DeepEP@c50f3d6fcd/tests/test_low_latency.py (L87-L100)

Image
<!-- gh-comment-id:3048363418 --> @shifangx commented on GitHub (Jul 8, 2025): **root cause: One rank fails at an assert and then exits. All the other ranks are waiting for that rank at some point, which causes the hang.** I ran experiments using 12*4 GB200s and found that after commenting out this assert, the program no longer hangs. https://github.com/deepseek-ai/DeepEP/blob/c50f3d6fcd800154dc41288fbefe194f33eb59cb/tests/test_low_latency.py#L87-L100 <img width="1329" height="385" alt="Image" src="https://github.com/user-attachments/assets/d68ae032-5413-4d00-bffb-1f0444b36054" />
bsakaguchi commented 2026-10-26 09:56:10 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Jul 8, 2025):

@shifangx Thanks! The assert didn't print out correctly after it was triggered, which made the issue quite confusing.
What is the theoretical lower bound for precision when using ue8m0 for scaling? We should probably relax the check here.

<!-- gh-comment-id:3048455011 --> @polarstormx commented on GitHub (Jul 8, 2025): @shifangx Thanks! The assert didn't print out correctly after it was triggered, which made the issue quite confusing. What is the theoretical lower bound for precision when using ue8m0 for scaling? We should probably relax the check here.
bsakaguchi commented 2026-10-26 09:56:10 +00:00
Author
Owner
Copy link

@jeffye-dev commented on GitHub (Jul 14, 2025):

Thanks，when setting round_scale=False the issue is gone. close this issue then

<!-- gh-comment-id:3069242917 --> @jeffye-dev commented on GitHub (Jul 14, 2025): Thanks，when setting round_scale=False the issue is gone. close this issue then
bsakaguchi commented 2026-10-26 09:56:10 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 15, 2025):

@shifangx Thanks! The assert didn't print out correctly after it was triggered, which made the issue quite confusing. What is the theoretical lower bound for precision when using ue8m0 for scaling? We should probably relax the check here.

@polarstormx https://github.com/deepseek-ai/DeepEP/pull/292 The destructor of DeepEP may cause the Python exception handling process to hang. Using explicitly_destroy allows the assert statements to be printed out.

<!-- gh-comment-id:3071707215 --> @sphish commented on GitHub (Jul 15, 2025): > [@shifangx](https://github.com/shifangx) Thanks! The assert didn't print out correctly after it was triggered, which made the issue quite confusing. What is the theoretical lower bound for precision when using ue8m0 for scaling? We should probably relax the check here. @polarstormx https://github.com/deepseek-ai/DeepEP/pull/292 The destructor of DeepEP may cause the Python exception handling process to hang. Using `explicitly_destroy` allows the assert statements to be printed out.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#184
No description provided.