[PR #371] [MERGED] fix potential hang in pollcq #537

New issue

Closed

opened 2026-10-27 21:03:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:47 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/371
Author: @Thunderbrook
Created: 8/21/2025
Status: ✅ Merged
Merged: 8/25/2025
Merged by: @sphish

Base: main ← Head: main

📝 Commits (1)

6c43246 fix potential hang in pollcq

📊 Changes

1 file changed (+1 additions, -1 deletions)

View changed files

📝 csrc/kernels/ibgda_device.cuh (+1 -1)

📄 Description

Hi, DeepEP will now execute pollcq before post wqe. The first time pollcq is performed, cqe is invalid (DeepEP does not check it now), but the existing logic will determine wqe_counter, when nvshmem is initialized, it will be initialized to 0xff. Therefore, when the queue depth is small, it may hang here.

DeepEP:

do {
        wqe_counter = HtoBE16(ld_na_relaxed(&cqe64->wqe_counter));
    } while ((static_cast<uint16_t>(static_cast<uint16_t>(idx) - wqe_counter - static_cast<uint16_t>(2)) < ncqes));

nvshmem:

status = cudaMemset(cq_mobject->base.gpu_ptr, 0xff, cq_mobject->base.size);

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/371 **Author:** [@Thunderbrook](https://github.com/Thunderbrook) **Created:** 8/21/2025 **Status:** ✅ Merged **Merged:** 8/25/2025 **Merged by:** [@sphish](https://github.com/sphish) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`6c43246`](https://github.com/deepseek-ai/DeepEP/commit/6c4324609dc9ad936d5006ea4eff7301a8777ae8) fix potential hang in pollcq ### 📊 Changes **1 file changed** (+1 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/ibgda_device.cuh` (+1 -1) </details> ### 📄 Description Hi, DeepEP will now execute pollcq before post wqe. The first time pollcq is performed, cqe is invalid (DeepEP does not check it now), but the existing logic will determine wqe_counter, when nvshmem is initialized, it will be initialized to 0xff. Therefore, when the queue depth is small, it may hang here. DeepEP: ```cpp do { wqe_counter = HtoBE16(ld_na_relaxed(&cqe64->wqe_counter)); } while ((static_cast<uint16_t>(static_cast<uint16_t>(idx) - wqe_counter - static_cast<uint16_t>(2)) < ncqes)); ``` nvshmem: ```cpp status = cudaMemset(cq_mobject->base.gpu_ptr, 0xff, cq_mobject->base.size); ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:47 +00:00