[GH-ISSUE #368] test_low_latency.py hangs with 64 GPUs while working fine with 32 GPUs #253

New issue

Open

opened 2026-10-26 09:57:28 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:57:28 +00:00

Owner

Originally created by @tjiang1898 on GitHub (Aug 20, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/368

With latest code (commit: f0d34aa), test_low_latency.py hangs on 8 node (64 H100 GPUs) with --num-experts 256. However, it worked fine when I ran it on 4 nodes (32 H100 GPUs). When it hung, I saw following warning log on some nodes:

Allocating buffer size: 1908.410496 MB ...
WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.

Originally created by @tjiang1898 on GitHub (Aug 20, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/368 With latest code (commit: f0d34aa), test_low_latency.py hangs on 8 node (64 H100 GPUs) with `--num-experts 256`. However, it worked fine when I ran it on 4 nodes (32 H100 GPUs). When it hung, I saw following warning log on some nodes: ``` Allocating buffer size: 1908.410496 MB ... WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources. ```

bsakaguchi commented

2026-10-26 09:57:29 +00:00

Author

Owner

@sphish commented on GitHub (Aug 21, 2025):

It looks like one of the processes encountered an error and exited. Do you have a more complete log?

@sphish commented on GitHub (Aug 21, 2025): It looks like one of the processes encountered an error and exited. Do you have a more complete log?

bsakaguchi referenced this issue

2026-10-27 21:03:05 +00:00

[PR #253] [MERGED] Remove memory fence in NVLink barrier. #489