[PR #695] Fix LL dispatch RC QP assert for all-P2P topologies #712

New issue

Open

opened 2026-10-27 21:05:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/695
Author: @xiayan0118
Created: 7/22/2026
Status: 🔄 Open

Base: main ← Head: fix/ll-dispatch-rc-qp-p2p-gate

📝 Commits (1)

87d8305 Fix LL dispatch RC QP assert for all-P2P topologies

📊 Changes

1 file changed (+15 additions, -2 deletions)

View changed files

📝 csrc/kernels/legacy/internode_ll.cu (+15 -2)

📄 Description

Summary

Low-latency dispatch currently asserts num_rc_per_pe >= num_local_experts unconditionally on SM 0. With NVSHMEM 3.6, optional IBGDA may leave num_rc_per_pe == 0 even when every remote rank is directly reachable through P2P. That poisons single-host / all-local jobs (e.g. B200) even though every send/count path already falls back to the P2P pointer when nvshmemi_get_p2p_ptr(...) != 0.

Gate the RC QP requirement on per-peer P2P reachability: require RC QPs only if at least one non-self peer has no P2P pointer.

Related reports: #134, #488 (same num_rc_per_pe family; this PR specifically covers the LL dispatch all-P2P case).

Change

In csrc/kernels/legacy/internode_ll.cu dispatch:

Scan non-self peers with nvshmemi_get_p2p_ptr(rdma_recv_count, rank, dst_rank)
Assert num_rc_per_pe >= num_local_experts only when needs_ibgda is true

Test plan

Single-node / all-local LL dispatch with IBGDA unavailable (num_rc_per_pe == 0) succeeds when all peers are P2P-reachable
Multi-node / non-P2P topology still fails the assert when RC QPs are insufficient
Existing LL dispatch correctness on NVLink all-local ranks

Made with Cursor

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/695 **Author:** [@xiayan0118](https://github.com/xiayan0118) **Created:** 7/22/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/ll-dispatch-rc-qp-p2p-gate` --- ### 📝 Commits (1) - [`87d8305`](https://github.com/deepseek-ai/DeepEP/commit/87d8305cecba451479c516944fdf213c7f08bfcc) Fix LL dispatch RC QP assert for all-P2P topologies ### 📊 Changes **1 file changed** (+15 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/internode_ll.cu` (+15 -2) </details> ### 📄 Description ## Summary Low-latency dispatch currently asserts `num_rc_per_pe >= num_local_experts` unconditionally on SM 0. With NVSHMEM 3.6, optional IBGDA may leave `num_rc_per_pe == 0` even when every remote rank is directly reachable through P2P. That poisons single-host / all-local jobs (e.g. B200) even though every send/count path already falls back to the P2P pointer when `nvshmemi_get_p2p_ptr(...) != 0`. Gate the RC QP requirement on per-peer P2P reachability: require RC QPs only if at least one non-self peer has no P2P pointer. Related reports: #134, #488 (same `num_rc_per_pe` family; this PR specifically covers the LL dispatch all-P2P case). ## Change In `csrc/kernels/legacy/internode_ll.cu` `dispatch`: - Scan non-self peers with `nvshmemi_get_p2p_ptr(rdma_recv_count, rank, dst_rank)` - Assert `num_rc_per_pe >= num_local_experts` only when `needs_ibgda` is true ## Test plan - [ ] Single-node / all-local LL dispatch with IBGDA unavailable (`num_rc_per_pe == 0`) succeeds when all peers are P2P-reachable - [ ] Multi-node / non-P2P topology still fails the assert when RC QPs are insufficient - [ ] Existing LL dispatch correctness on NVLink all-local ranks Made with [Cursor](https://cursor.com) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:30 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:38 +00:00

[PR #712] Reject NCCL runtimes older than the build #725