[PR #310] [MERGED] Fix for data error and kernel hung because of inflight rdma channel head update #515

New issue

Closed

opened 2026-10-27 21:03:31 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:31 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/310
Author: @GuangguanWang
Created: 7/17/2025
Status: ✅ Merged
Merged: 7/18/2025
Merged by: @sphish

Base: main ← Head: main

📝 Commits (1)

b65b22e Fix for data error and kernel hung because of inflight rdma channel head update

📊 Changes

1 file changed (+20 additions, -0 deletions)

View changed files

📝 csrc/kernels/internode.cu (+20 -0)

📄 Description

Fix for data error and kernel hung because of inflight rdma channel head update

When dispatch/combine, neither sender nor receiver waits for the finish of the rdma channel head update, which may result in the remaining inflight head update wqes even after the kernel finished. Once the infight wqes arrive after the rdma channel head buffer cleaning for the next round of dispatch/combine, the rdma channel head buffer will be re- written to a none-zero value. The rdma sender can reuse the data buffer before the rdma receivers consume the date buffer because of the wrong rdma channel head, cauing date error and kernel hung.
For performance considering, to overlap the inflight wqes' RTT, fix this issue by waiting for all previous inflight wqes to complete before cleaning rdma buffers in the next round of dispatch/combine.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/310 **Author:** [@GuangguanWang](https://github.com/GuangguanWang) **Created:** 7/17/2025 **Status:** ✅ Merged **Merged:** 7/18/2025 **Merged by:** [@sphish](https://github.com/sphish) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`b65b22e`](https://github.com/deepseek-ai/DeepEP/commit/b65b22ed3fbdd4f29baffdca2cc493d3f6926097) Fix for data error and kernel hung because of inflight rdma channel head update ### 📊 Changes **1 file changed** (+20 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/internode.cu` (+20 -0) </details> ### 📄 Description Fix for data error and kernel hung because of inflight rdma channel head update When dispatch/combine, neither sender nor receiver waits for the finish of the rdma channel head update, which may result in the remaining inflight head update wqes even after the kernel finished. Once the infight wqes arrive after the rdma channel head buffer cleaning for the next round of dispatch/combine, the rdma channel head buffer will be re- written to a none-zero value. The rdma sender can reuse the data buffer before the rdma receivers consume the date buffer because of the wrong rdma channel head, cauing date error and kernel hung. For performance considering, to overlap the inflight wqes' RTT, fix this issue by waiting for all previous inflight wqes to complete before cleaning rdma buffers in the next round of dispatch/combine. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:31 +00:00