deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #661] [Issue] Would VA signal with RO enabled cause unexpected data correctness problems in DeepEP V2? #388

New issue

Closed

opened 2026-10-26 09:59:54 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented

2026-10-26 09:59:54 +00:00

Owner

Originally created by @baymaxhuang on GitHub (Jun 9, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/661

The DeepEP v2 uses GIN VA signal with RO enabled to notify the peers to read tokens from the receive buffer. And the DeepEP document recommends configuring PCI_ATOMIC_MODE=4 to improve atomic performance, which would use the PCI Atomic, see pci-atomic-mode. According to the discussion in this issue https://github.com/NVIDIA/nccl/issues/2210, the RO PCI Atomic operation could bypass prior write operations. Maybe there would be a data correctness problem in DeepEP v2 if we configure PCI_ATOMIC_MODE=4 ?

Originally created by @baymaxhuang on GitHub (Jun 9, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/661 The DeepEP v2 uses GIN VA signal with RO enabled to notify the peers to read tokens from the receive buffer. And the DeepEP document recommends configuring `PCI_ATOMIC_MODE=4` to improve atomic performance, which would use the PCI Atomic, see [pci-atomic-mode](https://github.com/deepseek-ai/DeepEP#pci-atomic-mode). According to the discussion in this issue https://github.com/NVIDIA/nccl/issues/2210, the RO PCI Atomic operation could bypass prior write operations. Maybe there would be a data correctness problem in DeepEP v2 if we configure `PCI_ATOMIC_MODE=4` ?

bsakaguchi closed this issue

2026-10-26 09:59:54 +00:00

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@baymaxhuang commented on GitHub (Jun 9, 2026):

@sphish @LyricZhao could you please help confirm this?

@baymaxhuang commented on GitHub (Jun 9, 2026): @sphish @LyricZhao could you please help confirm this?

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@gangxie112 commented on GitHub (Jun 9, 2026):

Per the PCIe specification, a Posted Write request with RO=1 can be bypassed by subsequent Non-Posted requests. If the associated MR also enables RO, the FAA operation will not wait for the write-combining buffer to be flushed.
RO FAA acts as a PCIe ordering point. It forces all prior posted writes (including those from RO) to be flushed to memory before completion. This is why NCCL_WIN_STRICT_ORDERING maps to NCCL_NET_MR_FLAG_FORCE_SO, and consequently disables the IBV_ACCESS_RELAXED_ORDERING flag during MR registration.

@gangxie112 commented on GitHub (Jun 9, 2026): Per the PCIe specification, a Posted Write request with RO=1 can be bypassed by subsequent Non-Posted requests. If the associated MR also enables RO, the FAA operation will not wait for the write-combining buffer to be flushed. RO FAA acts as a PCIe ordering point. It forces all prior posted writes (including those from RO) to be flushed to memory before completion. This is why NCCL_WIN_STRICT_ORDERING maps to NCCL_NET_MR_FLAG_FORCE_SO, and consequently disables the IBV_ACCESS_RELAXED_ORDERING flag during MR registration.

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@sphish commented on GitHub (Jun 10, 2026):

Thanks for the feedback. I'll look into this.

@sphish commented on GitHub (Jun 10, 2026): Thanks for the feedback. I'll look into this.

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@sphish commented on GitHub (Jun 10, 2026):

This is indeed an issue. Have you encountered any data errors resulting from it?

@sphish commented on GitHub (Jun 10, 2026): This is indeed an issue. Have you encountered any data errors resulting from it?

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@baymaxhuang commented on GitHub (Jun 10, 2026):

@sphish, thanks for confirming it. Indeed, we have not encountered any data check errors when testing with test_ep.py. And we are planning to do more pressure tests on it because the data error problem is a very critical issue in model training.

@baymaxhuang commented on GitHub (Jun 10, 2026): @sphish, thanks for confirming it. Indeed, we have not encountered any data check errors when testing with test_ep.py. And we are planning to do more pressure tests on it because the data error problem is a very critical issue in model training.

bsakaguchi commented

2026-10-26 09:59:58 +00:00

Author

Owner

@kimimaroo commented on GitHub (Jun 29, 2026):

Hi @baymaxhuang, you mentioned earlier that you were planning to run more pressure tests on this. Any results to share? Especially curious about PCI_ATOMIC_MODE=4 with long-running data checks — given NVIDIA/nccl#2210 now confirms via PCIe spec B2b/C2b that mode 4 is the actually-unsafe case.

@kimimaroo commented on GitHub (Jun 29, 2026): Hi @baymaxhuang, you mentioned earlier that you were planning to run more pressure tests on this. Any results to share? Especially curious about PCI_ATOMIC_MODE=4 with long-running data checks — given NVIDIA/nccl#2210 now confirms via PCIe spec B2b/C2b that mode 4 is the actually-unsafe case.

bsakaguchi commented

2026-10-26 09:59:58 +00:00

Author

Owner

@baymaxhuang commented on GitHub (Jun 29, 2026):

Hi, @kimimaroo, I have not reproduced this issue with the config of PCI_ATOMIC_MODE=4. Maybe it is a very rare case, and we will be careful with it.

@baymaxhuang commented on GitHub (Jun 29, 2026): Hi, @kimimaroo, I have not reproduced this issue with the config of PCI_ATOMIC_MODE=4. Maybe it is a very rare case, and we will be careful with it.

bsakaguchi commented

2026-10-26 09:59:58 +00:00

Author

Owner

@kimimaroo commented on GitHub (Jun 30, 2026):

A thought on why PCI_ATOMIC_MODE=4 might not reproduce in practice — IB Spec o9-20 says:

An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:

Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data.

Arrival and completion of a subsequent SEND message.

Update of a memory element by a subsequent ATOMIC operation.

The third clause makes a subsequent ATOMIC a valid ordering point for prior WRITE contents on the same QP. So even if PCIe C2b permits the Atomic TLP to bypass posted writes, the HCA must enforce ordering above PCIe (clear RO on the atomic TLP, drain outstanding writes before issuing, or some internal fence) — otherwise it violates o9-20. That would explain why mode 4 doesn't corrupt data in practice.

@kimimaroo commented on GitHub (Jun 30, 2026): A thought on why PCI_ATOMIC_MODE=4 might not reproduce in practice — IB Spec o9-20 says: > An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: > - Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data. > - Arrival and completion of a subsequent SEND message. > - **Update of a memory element by a subsequent ATOMIC operation.** The third clause makes a subsequent ATOMIC a valid ordering point for prior WRITE contents on the same QP. So even if PCIe C2b permits the Atomic TLP to bypass posted writes, the HCA must enforce ordering above PCIe (clear RO on the atomic TLP, drain outstanding writes before issuing, or some internal fence) — otherwise it violates o9-20. That would explain why mode 4 doesn't corrupt data in practice.

bsakaguchi commented

2026-10-26 09:59:58 +00:00

Author

Owner

@sphish commented on GitHub (Jul 7, 2026):

Fixed in https://github.com/deepseek-ai/DeepEP/pull/674. Thank you!

@sphish commented on GitHub (Jul 7, 2026): Fixed in https://github.com/deepseek-ai/DeepEP/pull/674. Thank you!

bsakaguchi referenced this issue

2026-10-27 21:03:54 +00:00

[PR #388] Another computation communication overlap #549