[PR #687] [CLOSED] NCCL devComm: allocate according to devCommRuntimeVersionSize #706

New issue

Closed

opened 2026-10-27 21:05:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/687
Author: @kgioioso
Created: 7/13/2026
Status: ❌ Closed

Base: main ← Head: nccl-version-compat

📝 Commits (1)

fdaa64b NCCL Device API compat: use runtime version for ncclDevCommCreate

📊 Changes

7 files changed (+42 additions, -33 deletions)

View changed files

📝 csrc/kernels/backend/api.cuh (+1 -1)
📝 csrc/kernels/backend/nccl.cu (+18 -9)
📝 csrc/kernels/elastic/barrier.hpp (+3 -3)
📝 csrc/kernels/elastic/combine.hpp (+4 -4)
📝 csrc/kernels/elastic/dispatch.hpp (+4 -4)
📝 csrc/kernels/elastic/engram.hpp (+6 -6)
📝 csrc/kernels/elastic/pp_send_recv.hpp (+6 -6)

📄 Description

Summary

Since the DeepEP device code is JIT compiled, it uses the runtime version of ncclDevComm, not the compile-time version. But the devComm is allocated on the host according to the host-compile-version. If the size of DevComm changes across versions, the allocated devComm may not be big enough.

Changes

This PR makes changes to ensure the NCCL device API is functional even if the runtime version is not equal to the compile-time version:

malloc the devComm according to devCommRuntimeSize
set reqs->useRuntimeVersion. This indicates to NCCL that the device code is compiled using the runtime version
Pass ncclDevComm_t* instead of ncclDevComm_t in most of the code. This makes it more explicit that the size is variable.

Testing

Run test_ep.py:

Compile with 2.30.7, run with 2.30.7 -> success
Compile with 2.31 (dev branch on github), run with 2.31 -> success
Compile with 2.30.7, run with 2.31 -> failure. This condition will not work with 2.30.7 because devCommRuntimeSize is not yet exposed. But, in the future, compiling with x and running with x+1 will work.
Compile with 2.31, run with 2.30.7 -> failure (forward compat is not supported)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/687 **Author:** [@kgioioso](https://github.com/kgioioso) **Created:** 7/13/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `nccl-version-compat` --- ### 📝 Commits (1) - [`fdaa64b`](https://github.com/deepseek-ai/DeepEP/commit/fdaa64b5448e2c174ea01da2c41bab1131b46d13) NCCL Device API compat: use runtime version for ncclDevCommCreate ### 📊 Changes **7 files changed** (+42 additions, -33 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/backend/api.cuh` (+1 -1) 📝 `csrc/kernels/backend/nccl.cu` (+18 -9) 📝 `csrc/kernels/elastic/barrier.hpp` (+3 -3) 📝 `csrc/kernels/elastic/combine.hpp` (+4 -4) 📝 `csrc/kernels/elastic/dispatch.hpp` (+4 -4) 📝 `csrc/kernels/elastic/engram.hpp` (+6 -6) 📝 `csrc/kernels/elastic/pp_send_recv.hpp` (+6 -6) </details> ### 📄 Description # Summary Since the DeepEP device code is JIT compiled, it uses the runtime version of ncclDevComm, not the compile-time version. But the devComm is allocated on the host according to the host-compile-version. If the size of DevComm changes across versions, the allocated devComm may not be big enough. # Changes This PR makes changes to ensure the NCCL device API is functional even if the runtime version is not equal to the compile-time version: 1. malloc the devComm according to devCommRuntimeSize 2. set reqs->useRuntimeVersion. This indicates to NCCL that the device code is compiled using the runtime version 3. Pass `ncclDevComm_t*` instead of `ncclDevComm_t` in most of the code. This makes it more explicit that the size is variable. # Testing Run `test_ep.py`: * Compile with 2.30.7, run with 2.30.7 -> success * Compile with 2.31 (`dev` branch on github), run with 2.31 -> success * Compile with 2.30.7, run with 2.31 -> failure. This condition will not work with 2.30.7 because `devCommRuntimeSize` is not yet exposed. But, in the future, compiling with x and running with x+1 will work. * Compile with 2.31, run with 2.30.7 -> failure (forward compat is not supported) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:27 +00:00