deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #20] why enabling adaptive routing for normal internode kernels may lead to deadlocks or data corruption issues？ #20

New issue

Closed

opened 2026-10-26 09:52:37 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-26 09:52:37 +00:00

Owner

Originally created by @VincentHITlemon on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/20

Enabling adaptive routing for normal internode kernels may lead to deadlocks or data corruption issues.

I think adaptive routing should be a simple network function, why enabling adaptive routing for normal internode kernels may lead to deadlocks or data corruption issues？

Originally created by @VincentHITlemon on GitHub (Feb 26, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/20 **Enabling adaptive routing for normal internode kernels may lead to deadlocks or data corruption issues.** I think adaptive routing should be a simple network function, why enabling adaptive routing for normal internode kernels may lead to deadlocks or data corruption issues？

bsakaguchi closed this issue

2026-10-26 09:52:37 +00:00

bsakaguchi commented

2026-10-26 09:52:38 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 26, 2025):

The remote receivers will judge the data arrival by a flag. For senders, they send tokens firstly, the flag secondly. With AR enabled, the flag could arrive before the data arrival. So the receivers may read the wrong data.

Later we will add AR support, cc @sphish

@LyricZhao commented on GitHub (Feb 26, 2025): The remote receivers will judge the data arrival by a flag. For senders, they send tokens firstly, the flag secondly. With AR enabled, the flag could arrive before the data arrival. So the receivers may read the wrong data. Later we will add AR support, cc @sphish

bsakaguchi commented

2026-10-26 09:52:38 +00:00

Author

Owner

@GangLuICT commented on GitHub (Feb 27, 2025):

The remote receivers will judge the data arrival by a flag. For senders, they send tokens firstly, the flag secondly. With AR enabled, the flag could arrive before the data arrival. So the receivers may read the wrong data.

Later we will add AR support, cc @sphish

It seems few modification is needed to guarantee the consistency.
If so, why didn't you support AR from the begining?
Is it because that may introduce lower bandwidth?

@GangLuICT commented on GitHub (Feb 27, 2025): > The remote receivers will judge the data arrival by a flag. For senders, they send tokens firstly, the flag secondly. With AR enabled, the flag could arrive before the data arrival. So the receivers may read the wrong data. > > Later we will add AR support, cc [@sphish](https://github.com/sphish) It seems few modification is needed to guarantee the consistency. If so, why didn't you support AR from the begining? Is it because that may introduce lower bandwidth?

bsakaguchi commented

2026-10-26 09:52:38 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 27, 2025):

If so, why didn't you support AR from the beginning?

Because that's not very easy.

@LyricZhao commented on GitHub (Feb 27, 2025): > If so, why didn't you support AR from the beginning? Because that's not very easy.

bsakaguchi commented

2026-10-26 09:52:38 +00:00

Author

Owner

@sphish commented on GitHub (Feb 27, 2025):

If so, why didn't you support AR from the beginning?

Because that's not very easy.

Specifically, supporting adaptive routing while maintaining high performance requires a completion-based notification mechanism. However, NVSHMEM does not natively provide such APIs, making the development more challenging.

@sphish commented on GitHub (Feb 27, 2025): > > If so, why didn't you support AR from the beginning? > > Because that's not very easy. Specifically, supporting adaptive routing while maintaining high performance requires a completion-based notification mechanism. However, NVSHMEM does not natively provide such APIs, making the development more challenging.

bsakaguchi commented

2026-10-26 09:52:39 +00:00

Author

Owner

@wenduobaba commented on GitHub (Mar 6, 2025):

@sphish Just noticed this PR got merged https://github.com/deepseek-ai/DeepEP/pull/45 - IIUC now both normal and LL internode dispatch/combine support AR. Is that correct?

@wenduobaba commented on GitHub (Mar 6, 2025): @sphish Just noticed this PR got merged https://github.com/deepseek-ai/DeepEP/pull/45 - IIUC now both normal and LL internode dispatch/combine support AR. Is that correct?

bsakaguchi commented

2026-10-26 09:52:40 +00:00

Author

Owner

@sphish commented on GitHub (Mar 6, 2025):

@sphish Just noticed this PR got merged #45 - IIUC now both normal and LL internode dispatch/combine support AR. Is that correct?刚刚注意到这个 PR 合并了 #45 - 按我的理解，现在 both normal 和 LL internode dispatch/combine 都支持 AR 了。这是正确的吗？

@wenduobaba Yes

@sphish commented on GitHub (Mar 6, 2025): > [@sphish](https://github.com/sphish) Just noticed this PR got merged [#45](https://github.com/deepseek-ai/DeepEP/pull/45) - IIUC now both normal and LL internode dispatch/combine support AR. Is that correct?刚刚注意到这个 PR 合并了 [#45](https://github.com/deepseek-ai/DeepEP/pull/45) - 按我的理解，现在 both normal 和 LL internode dispatch/combine 都支持 AR 了。这是正确的吗？ @wenduobaba Yes

bsakaguchi referenced this issue

2026-10-26 09:53:39 +00:00

[GH-ISSUE #74] test_internode.py hangs at internode.cu:dispatch(...) #68

bsakaguchi referenced this issue

2026-10-26 09:56:29 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200

bsakaguchi referenced this issue

2026-10-26 09:56:40 +00:00

[GH-ISSUE #290] Running the test case on a 2-node H20 3E machine failed with internode.cu:1029 'unspecified launch failure' #208