[PR #181] [MERGED] Fix dispatch timeout problem: concurrent submissions of qp should be forbidden. #452

New issue

Closed

opened 2026-10-26 10:00:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/181
Author: @wzc-wuzhicheng
Created: 6/3/2025
Status: ✅ Merged
Merged: 6/13/2025
Merged by: @LyricZhao

Base: main ← Head: fix-dispatch-timeout

📝 Commits (1)

6bd6585 Use one qp per sm for internode normal kernels

📊 Changes

2 files changed (+3 additions, -3 deletions)

View changed files

📝 csrc/kernels/internode.cu (+2 -2)
📝 tests/test_internode.py (+1 -1)

📄 Description

kRDMASenderCoordinator and kForwarderCoordinator use the same qp to commit requests with channel_id, which will break down the atomicity of doorbell operation:

assuming that tx_wq.prod_idx is now A
nvshmemi_ibgda_put_nbi_warp in kRDMASenderCoordinator reserves slots, the expected tx_wq.prod_idx is B.
nvshmemi_ibgda_amo_nonfetch_add in kForwarderCoordinator reserves slots, writes wqe and kicks doorbell, the expected tx_wq.prod_idx is C, and NIC will send wqes from A to C, but nvshmemi_ibgda_put_nbi_warp may not have completed writing wqe yet, and the corresponding write will lost.
nvshmemi_ibgda_put_nbi_warp in kRDMASenderCoordinator writes wqe and fails to kick doorbell because B < C.

kRDMASenderCoordinator and kForwarderCoordinator will use different qp to commit requests with sm_id now.

NOTE: The timeout problem is not easy to reproduce in gpu:nic=1:1 environment , but I found that there are someways to reproduce it more easily:

set up gpu:nic=2:1 with NVSHMEM_ENABLE_NIC_PE_MAPPING and NVSHMEM_HCA_LIST
use smaller nvl_chunk_size and rdma_chunk_size in tests/test_internode.py, which will call nvshmem API calls more frequently.

related issues:
#137
#158

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/181 **Author:** [@wzc-wuzhicheng](https://github.com/wzc-wuzhicheng) **Created:** 6/3/2025 **Status:** ✅ Merged **Merged:** 6/13/2025 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `fix-dispatch-timeout` --- ### 📝 Commits (1) - [`6bd6585`](https://github.com/deepseek-ai/DeepEP/commit/6bd6585aee5c7914c46766cb12ce91fcdafe50af) Use one qp per sm for internode normal kernels ### 📊 Changes **2 files changed** (+3 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/internode.cu` (+2 -2) 📝 `tests/test_internode.py` (+1 -1) </details> ### 📄 Description kRDMASenderCoordinator and kForwarderCoordinator use the same qp to commit requests with channel_id, which will break down the atomicity of doorbell operation: 1. assuming that tx_wq.prod_idx is now A 2. nvshmemi_ibgda_put_nbi_warp in kRDMASenderCoordinator reserves slots, the expected tx_wq.prod_idx is B. 3. nvshmemi_ibgda_amo_nonfetch_add in kForwarderCoordinator reserves slots, writes wqe and kicks doorbell, the expected tx_wq.prod_idx is C, and NIC will send wqes from A to C, but nvshmemi_ibgda_put_nbi_warp may not have completed writing wqe yet, and the corresponding write will lost. 4. nvshmemi_ibgda_put_nbi_warp in kRDMASenderCoordinator writes wqe and fails to kick doorbell because B < C. kRDMASenderCoordinator and kForwarderCoordinator will use different qp to commit requests with sm_id now. NOTE: The timeout problem is not easy to reproduce in gpu:nic=1:1 environment , but I found that there are someways to reproduce it more easily: - set up gpu:nic=2:1 with NVSHMEM_ENABLE_NIC_PE_MAPPING and NVSHMEM_HCA_LIST - use smaller nvl_chunk_size and rdma_chunk_size in tests/test_internode.py, which will call nvshmem API calls more frequently. related issues: #137 #158 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 10:00:54 +00:00