[GH-ISSUE #40] Why do we need to disable NVSHMEM_IB_ENABLE_RELAXED_ORDERING? #37

New issue

Closed

opened 2026-10-26 09:53:05 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:53:05 +00:00

Owner

Originally created by @Tuvie on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/40

Hi,

There is a new commit recently which disable NVSHMEM_IB_ENABLE_RELAXED_ORDERING in buffer.py:
github.com/deepseek-ai/DeepEP@3885404ffb

Generally, we think relaxed ordering can help to improve the performance.

Do we have some reasons to disable it? What does it mean that "avoid out-of-order messages"?

Thanks

Originally created by @Tuvie on GitHub (Mar 4, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/40 Hi, There is a new commit recently which disable NVSHMEM_IB_ENABLE_RELAXED_ORDERING in buffer.py: https://github.com/deepseek-ai/DeepEP/commit/3885404ffb31a3d9cbd238aa1a366408881bb069 Generally, we think relaxed ordering can help to improve the performance. Do we have some reasons to disable it? What does it mean that "avoid out-of-order messages"? Thanks

bsakaguchi closed this issue

2026-10-26 09:53:05 +00:00

bsakaguchi commented

2026-10-26 09:53:06 +00:00

Author

Owner

@wx-csy commented on GitHub (Mar 4, 2025):

This is actually a hack to the Infiniband protocol. According to the Infiniband specification, two consecutive WRITE operations are allowed to be reordered, i.e., the responder might see the second write before the first one. From the implementation perspective, this might happen in at least the following two places:

when enabling AR, the network packets might be delivered out-of-order to the destination (though I believe this behavior is not standard-compliant);
the PCIe subsystem might reorder the DMA transactions to memory.

To prevent reordering, we disable AR to prevent the first case and disable RELAXED_ORDERING to prevent the second case.

@wx-csy commented on GitHub (Mar 4, 2025): This is actually a hack to the Infiniband protocol. According to the Infiniband specification, two consecutive `WRITE` operations are allowed to be reordered, i.e., the responder might see the second write before the first one. From the implementation perspective, this might happen in at least the following two places: 1. when enabling AR, the network packets might be delivered out-of-order to the destination (though I believe this behavior is not standard-compliant); 2. the PCIe subsystem might reorder the DMA transactions to memory. To prevent reordering, we disable AR to prevent the first case and disable `RELAXED_ORDERING` to prevent the second case.

bsakaguchi commented

2026-10-26 09:53:07 +00:00

Author

Owner

@Tuvie commented on GitHub (Mar 4, 2025):

Thanks for you reply.

Is this only necessary for normal internode mode? Or it is also necessary for both normal and low latency mode?

@Tuvie commented on GitHub (Mar 4, 2025): Thanks for you reply. Is this only necessary for normal internode mode? Or it is also necessary for both normal and low latency mode?

bsakaguchi commented

2026-10-26 09:53:07 +00:00

Author

Owner

@wx-csy commented on GitHub (Mar 4, 2025):

It is only necessary for normal mode as LL mode uses a standard-compliant way to prevent reordering.

@wx-csy commented on GitHub (Mar 4, 2025): It is only necessary for normal mode as LL mode uses a standard-compliant way to prevent reordering.

bsakaguchi commented

2026-10-26 09:53:07 +00:00

Author

Owner

@Tuvie commented on GitHub (Mar 4, 2025):

Thanks.

@Tuvie commented on GitHub (Mar 4, 2025): Thanks.

bsakaguchi referenced this issue

2026-10-26 09:56:30 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200