[PR #317] [MERGED] Avoid hang due to small rdma_chunk_size #516

New issue

Closed

opened 2026-10-27 21:03:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/317
Author: @ZhiyiHu1999
Created: 7/21/2025
Status: ✅ Merged
Merged: 7/21/2025
Merged by: @LyricZhao

Base: main ← Head: fix/hang_due_to_small_rdma_chunk_size

📝 Commits (1)

cf18a3d fix hang due to small rdma_chunk_size

📊 Changes

1 file changed (+1 additions, -0 deletions)

View changed files

📝 csrc/kernels/internode.cu (+1 -0)

📄 Description

For example, if we have two nodes, num_warps_per_forwarder will be 16 / 2 = 8.

github.com/deepseek-ai/DeepEP@e6012370b0/csrc/kernels/internode.cu (L1550)

If num_max_rdma_chunked_send_tokens is 4 < 8, then token_end_idx - token_start_idx = 4, the 'forwarder_nvl_head' for subwarp 4, 5, 6, 7 will always be 0 and min_head in forwarder kCoordinator will always be 0. So, the NVLSender warp won't work after the NVL buffer is full for the first time because the head value it reads is always 0.

github.com/deepseek-ai/DeepEP@e6012370b0/csrc/kernels/internode.cu (L1570-L1605)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/317 **Author:** [@ZhiyiHu1999](https://github.com/ZhiyiHu1999) **Created:** 7/21/2025 **Status:** ✅ Merged **Merged:** 7/21/2025 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `fix/hang_due_to_small_rdma_chunk_size` --- ### 📝 Commits (1) - [`cf18a3d`](https://github.com/deepseek-ai/DeepEP/commit/cf18a3d04e6b5c7dc37508d2c10096f5e7b4f096) fix hang due to small rdma_chunk_size ### 📊 Changes **1 file changed** (+1 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/internode.cu` (+1 -0) </details> ### 📄 Description For example, if we have two nodes, `num_warps_per_forwarder` will be 16 / 2 = 8. https://github.com/deepseek-ai/DeepEP/blob/e6012370b01a86e821ace99193f5d912b3f66d06/csrc/kernels/internode.cu#L1550 If `num_max_rdma_chunked_send_tokens` is 4 < 8, then `token_end_idx` - `token_start_idx` = 4, the 'forwarder_nvl_head' for subwarp 4, 5, 6, 7 will always be 0 and `min_head` in forwarder kCoordinator will always be 0. So, the `NVLSender `warp won't work after the NVL buffer is full for the first time because the head value it reads is always 0. https://github.com/deepseek-ai/DeepEP/blob/e6012370b01a86e821ace99193f5d912b3f66d06/csrc/kernels/internode.cu#L1570-L1605 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:35 +00:00