[GH-ISSUE #378] multi node low latency mode hanging #256

New issue

Closed

opened 2026-10-26 09:57:30 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:57:30 +00:00

Owner

Originally created by @xutizhou on GitHub (Aug 27, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/378

Hi expert, I am running ll at 4 H20 node and it hangs at allocating buffer.

Here is my environment setting.

server:

export MASTER_ADDR=10.6.131.30

export WORLD_SIZE=4

export MASTER_PORT=40303

export RANK=0

client：

export MASTER_ADDR=10.6.131.30

export WORLD_SIZE=4

export MASTER_PORT=40303

export RANK=1



export MASTER_ADDR=10.6.131.30

export WORLD_SIZE=4

export MASTER_PORT=40303

export RANK=2



export MASTER_ADDR=10.6.131.30

export WORLD_SIZE=4

export MASTER_PORT=40303

export RANK=3

python /workspace/tests/test_internode.py --test-ll-compatibility

it works at node=2 but hangs when node=4. What might be causing this? How should I adjust my testing approach?

Originally created by @xutizhou on GitHub (Aug 27, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/378 Hi expert, I am running ll at 4 H20 node and it hangs at allocating buffer. <img width="2097" height="293" alt="Image" src="https://github.com/user-attachments/assets/41abc5f9-0d2c-48f3-8755-698c172bc7b7" /> Here is my environment setting. server: ```shell export MASTER_ADDR=10.6.131.30 export WORLD_SIZE=4 export MASTER_PORT=40303 export RANK=0 ``` client： ```shell export MASTER_ADDR=10.6.131.30 export WORLD_SIZE=4 export MASTER_PORT=40303 export RANK=1 export MASTER_ADDR=10.6.131.30 export WORLD_SIZE=4 export MASTER_PORT=40303 export RANK=2 export MASTER_ADDR=10.6.131.30 export WORLD_SIZE=4 export MASTER_PORT=40303 export RANK=3 ``` ```python python /workspace/tests/test_internode.py --test-ll-compatibility ``` it works at node=2 but hangs when node=4. What might be causing this? How should I adjust my testing approach?

bsakaguchi closed this issue

2026-10-26 09:57:30 +00:00

bsakaguchi commented

2026-10-26 09:57:32 +00:00

Author

Owner

@xutizhou commented on GitHub (Sep 1, 2025):

This is due the fail of accuracy test, after commenting that, this issue is solved.

@xutizhou commented on GitHub (Sep 1, 2025): This is due the fail of accuracy test, after commenting that, this issue is solved.

bsakaguchi referenced this issue

2026-10-27 21:03:04 +00:00

[PR #256] [MERGED] Add get_comm_stream. #487