deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[PR #591] [MERGED] Hybrid ep nixl2 #650

New issue
Closed
opened 2026-10-27 21:04:52 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-27 21:04:52 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/591
Author: @QiaoK
Created: 3/12/2026
Status: ✅ Merged
Merged: 4/16/2026
Merged by: @jershi425

Base: hybrid-ep ← Head: hybrid-ep-nixl2

📝 Commits (10+)

  • 8bc735b add nixl connector stubs
  • a441b04 JIT: Fix potential race condition from multiple processes modifying the same file path
  • d053b69 NIXL: Copy connector and warp code from hybrid-ep
  • cd6d4df NIXL: Implement NIXLCoordinator
  • 87af414 NIXL: Wire NIXLCoordinator into DeepEP
  • bf71d60 NIXL: Add USE_NIXL build path in setup.py
  • c10ad9f NIXL: Add JIT compiler support for USE_NIXL build
  • dbac297 NIXL: Fix Device API v2 compat, FP8 stride mismatch, and JIT build
  • 33e0158 refactor: decouple internode backends and unify coordinator
  • 6ddeb4c Consolidate NIXL warp, fix param_t ABI

📊 Changes

18 files changed (+2286 additions, -433 deletions)

View changed files

➕ .use_nixl (+0 -0)
📝 README.md (+10 -0)
📝 csrc/hybrid_ep/backend/hybrid_ep_backend.cuh (+252 -142)
📝 csrc/hybrid_ep/buffer/internode.cuh (+8 -229)
📝 csrc/hybrid_ep/buffer/internode_doca.cu (+1 -1)
➕ csrc/hybrid_ep/buffer/internode_doca.cuh (+182 -0)
➕ csrc/hybrid_ep/buffer/internode_nixl.cu (+208 -0)
➕ csrc/hybrid_ep/buffer/internode_nixl.cuh (+41 -0)
➕ csrc/hybrid_ep/buffer/nixl_connector.cu (+934 -0)
➕ csrc/hybrid_ep/buffer/nixl_connector.h (+146 -0)
📝 csrc/hybrid_ep/coordinator.cuh (+75 -0)
📝 csrc/hybrid_ep/executor/executor.cu (+19 -5)
📝 csrc/hybrid_ep/hybrid_ep.cu (+6 -6)
📝 csrc/hybrid_ep/hybrid_ep.cuh (+6 -1)
📝 csrc/hybrid_ep/jit/compiler.cu (+38 -3)
➕ docs/Dockerfile.nixl (+164 -0)
📝 docs/README_Hybrid-EP.md (+105 -2)
📝 setup.py (+91 -44)

📄 Description

Pull Request: NIXL Integration for Hybrid-EP Multi-Node Communication

Summary

This PR integrates NIXL (NVIDIA Inter-node eXchange Library) as an alternative inter-node communication backend for Hybrid-EP, alongside the existing DOCA/RDMA path. NIXL uses UCX for GPU-to-GPU communication and does not require the NCCL submodule or DOCA SDK, simplifying deployment in environments where DOCA is unavailable.

Motivation

  • DOCA build complexity: The DOCA path requires the NCCL submodule, DOCA SDK, and doca_gpunetio_device.h, which often fails with "No rule to make target" when DOCA is not installed.
  • NIXL availability: NIXL provides a lighter-weight inter-node path using UCX and etcd, suitable for clusters without DOCA.
  • Build flexibility: Users can choose NIXL or DOCA at build time via USE_NIXL=1.

Changes

1. NIXL Coordinator and Connector

  • NIXLCoordinator (internode_nixl.cuh / internode_nixl.cu): Implements allocate_buffers(), destroy(), grow_buffer_config(), and update_config() using the NIXL connector.
  • HybridEP_NIXLConnector (nixl_connector.h / nixl_connector.cu): Ported from hybrid-ep; manages NIXL agent lifecycle, buffer registration, peer exchange via etcd, UCX wireup, and GPU context building.
  • Backend integration (hybrid_ep_backend.cuh, nixl_types.h, nixl_warp.cuh): Added USE_NIXL-guarded paths for dispatch_gpu_nixl_ctx and combine_gpu_nixl_ctx in dispatch/combine kernels.

2. Build System (setup.py)

  • Dual-path build: When HYBRID_EP_MULTINODE=1:
    • NIXL path (USE_NIXL=1): Compiles internode_nixl.cu and nixl_connector.cu; links against NIXL, UCX, mlx5, ibverbs; skips NCCL/DOCA build.
    • DOCA path (USE_NIXL=0): Compiles internode.cu; builds NCCL submodule; uses DOCA objects for JIT.
  • .use_nixl fallback: If USE_NIXL is not set but .use_nixl exists in the project root, the NIXL path is used (helps when pip build isolation does not propagate env vars).
  • NCCL build failure handling: When the DOCA path fails (e.g. missing doca_gpunetio_device.h), a clear error suggests using USE_NIXL=1 or creating .use_nixl.
  • Build logging: Prints use_nixl=... and -> NIXL path or -> DOCA path during build.

3. JIT Compiler (compiler.cu)

  • USE_NIXL support: When built with USE_NIXL, JIT-compiled inter-node kernels use NIXL libs instead of DOCA objects.
  • Runtime logging: Prints [HybridEP JIT] Inter-node: using NIXL or using DOCA/RDMA when the JIT compiler initializes.

4. Runtime Logging (hybrid_ep.cu)

  • Init: Prints [HybridEP] Inter-node: using NIXL (node_rank=X local_rank=Y) or using DOCA/RDMA when multinode coordinators are initialized.
  • allocate_buffers: Prints [HybridEP] NIXL allocate_buffers (num_of_nodes=X) or DOCA allocate_buffers for debugging.

5. Documentation

  • README_Hybrid-EP.md:
    • NIXL vs DOCA installation instructions.
    • Troubleshooting for doca_gpunetio_device.h error.
    • Two-node hybrid_ep_test configuration (WORLD_SIZE, RANK, MASTER_ADDR, etc.).
    • Single-machine simulation with NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN.

6. Other Fixes

  • JIT race fix: Process-specific JIT cache dir (.../jit/proc-{pid}) to avoid races when multiple processes compile concurrently.
  • Device API v2 / FP8: Flag semantics, stride alignment, and GPU context struct updates for NIXL compatibility.

Build Instructions

NIXL path (recommended when DOCA is unavailable):

export HYBRID_EP_MULTINODE=1
export USE_NIXL=1
export NIXL_HOME=/usr/local/nixl  # Adjust if needed
export TORCH_CUDA_ARCH_LIST="9.0 10.0"
pip install .

DOCA path (when DOCA SDK is available):

export HYBRID_EP_MULTINODE=1
export RDMA_CORE_HOME=/path/to/rdma-core
pip install .

Fallback when env does not propagate:

touch .use_nixl
HYBRID_EP_MULTINODE=1 pip install .

Testing

  • Single-node: python tests/test_hybrid_ep.py --num-processes 4 (unchanged; NIXL not used).
  • Multi-node: Set WORLD_SIZE=2, RANK=0 or 1, MASTER_ADDR, MASTER_PORT on each node; run python tests/test_hybrid_ep.py --num-processes 4 on both.
  • Single-machine simulation: NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN=2 python tests/test_hybrid_ep.py --num-processes 4 to simulate 2 nodes on 4 GPUs.

Dependencies

  • NIXL path: NIXL library, UCX, mlx5, ibverbs. Optional: etcd (NIXL_ETCD_ENDPOINTS) for multi-node.
  • DOCA path: NCCL submodule, DOCA SDK, rdma-core (unchanged).

Backward Compatibility

  • USE_NIXL=0 (default) preserves the DOCA path; no behavior change for existing DOCA users.
  • NIXL code is fully guarded by #ifdef USE_NIXL; no NIXL symbols are linked when using DOCA.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/591 **Author:** [@QiaoK](https://github.com/QiaoK) **Created:** 3/12/2026 **Status:** ✅ Merged **Merged:** 4/16/2026 **Merged by:** [@jershi425](https://github.com/jershi425) **Base:** `hybrid-ep` ← **Head:** `hybrid-ep-nixl2` --- ### 📝 Commits (10+) - [`8bc735b`](https://github.com/deepseek-ai/DeepEP/commit/8bc735b9c022e3bc0615be06e71ebc609b0e2b47) add nixl connector stubs - [`a441b04`](https://github.com/deepseek-ai/DeepEP/commit/a441b04cdd0ba9a6240a05116ce8f00d93d427d4) JIT: Fix potential race condition from multiple processes modifying the same file path - [`d053b69`](https://github.com/deepseek-ai/DeepEP/commit/d053b69ce11583910c0d0e61ba243adae3998f4e) NIXL: Copy connector and warp code from hybrid-ep - [`cd6d4df`](https://github.com/deepseek-ai/DeepEP/commit/cd6d4dfd16b870e3b5d9cbf8be23d1cca52cab3b) NIXL: Implement NIXLCoordinator - [`87af414`](https://github.com/deepseek-ai/DeepEP/commit/87af4141fb29cdf718d50aabade5f93925c3911d) NIXL: Wire NIXLCoordinator into DeepEP - [`bf71d60`](https://github.com/deepseek-ai/DeepEP/commit/bf71d60ef78f62f1627ced2e7ff92ddb4280f3d5) NIXL: Add USE_NIXL build path in setup.py - [`c10ad9f`](https://github.com/deepseek-ai/DeepEP/commit/c10ad9fa89cd8021b53fd11a9bd57bd9f4dc805b) NIXL: Add JIT compiler support for USE_NIXL build - [`dbac297`](https://github.com/deepseek-ai/DeepEP/commit/dbac29775eb4db719c5cb104084425d733eb7365) NIXL: Fix Device API v2 compat, FP8 stride mismatch, and JIT build - [`33e0158`](https://github.com/deepseek-ai/DeepEP/commit/33e0158a701271a3af7ae47efe5771b858db9e18) refactor: decouple internode backends and unify coordinator - [`6ddeb4c`](https://github.com/deepseek-ai/DeepEP/commit/6ddeb4cd31d24c14cdb5a4083a087fbc4044f8d9) Consolidate NIXL warp, fix param_t ABI ### 📊 Changes **18 files changed** (+2286 additions, -433 deletions) <details> <summary>View changed files</summary> ➕ `.use_nixl` (+0 -0) 📝 `README.md` (+10 -0) 📝 `csrc/hybrid_ep/backend/hybrid_ep_backend.cuh` (+252 -142) 📝 `csrc/hybrid_ep/buffer/internode.cuh` (+8 -229) 📝 `csrc/hybrid_ep/buffer/internode_doca.cu` (+1 -1) ➕ `csrc/hybrid_ep/buffer/internode_doca.cuh` (+182 -0) ➕ `csrc/hybrid_ep/buffer/internode_nixl.cu` (+208 -0) ➕ `csrc/hybrid_ep/buffer/internode_nixl.cuh` (+41 -0) ➕ `csrc/hybrid_ep/buffer/nixl_connector.cu` (+934 -0) ➕ `csrc/hybrid_ep/buffer/nixl_connector.h` (+146 -0) 📝 `csrc/hybrid_ep/coordinator.cuh` (+75 -0) 📝 `csrc/hybrid_ep/executor/executor.cu` (+19 -5) 📝 `csrc/hybrid_ep/hybrid_ep.cu` (+6 -6) 📝 `csrc/hybrid_ep/hybrid_ep.cuh` (+6 -1) 📝 `csrc/hybrid_ep/jit/compiler.cu` (+38 -3) ➕ `docs/Dockerfile.nixl` (+164 -0) 📝 `docs/README_Hybrid-EP.md` (+105 -2) 📝 `setup.py` (+91 -44) </details> ### 📄 Description # Pull Request: NIXL Integration for Hybrid-EP Multi-Node Communication ## Summary This PR integrates **NIXL** (NVIDIA Inter-node eXchange Library) as an alternative inter-node communication backend for Hybrid-EP, alongside the existing DOCA/RDMA path. NIXL uses UCX for GPU-to-GPU communication and does not require the NCCL submodule or DOCA SDK, simplifying deployment in environments where DOCA is unavailable. ## Motivation - **DOCA build complexity**: The DOCA path requires the NCCL submodule, DOCA SDK, and `doca_gpunetio_device.h`, which often fails with "No rule to make target" when DOCA is not installed. - **NIXL availability**: NIXL provides a lighter-weight inter-node path using UCX and etcd, suitable for clusters without DOCA. - **Build flexibility**: Users can choose NIXL or DOCA at build time via `USE_NIXL=1`. ## Changes ### 1. NIXL Coordinator and Connector - **`NIXLCoordinator`** (`internode_nixl.cuh` / `internode_nixl.cu`): Implements `allocate_buffers()`, `destroy()`, `grow_buffer_config()`, and `update_config()` using the NIXL connector. - **`HybridEP_NIXLConnector`** (`nixl_connector.h` / `nixl_connector.cu`): Ported from hybrid-ep; manages NIXL agent lifecycle, buffer registration, peer exchange via etcd, UCX wireup, and GPU context building. - **Backend integration** (`hybrid_ep_backend.cuh`, `nixl_types.h`, `nixl_warp.cuh`): Added `USE_NIXL`-guarded paths for `dispatch_gpu_nixl_ctx` and `combine_gpu_nixl_ctx` in dispatch/combine kernels. ### 2. Build System (`setup.py`) - **Dual-path build**: When `HYBRID_EP_MULTINODE=1`: - **NIXL path** (`USE_NIXL=1`): Compiles `internode_nixl.cu` and `nixl_connector.cu`; links against NIXL, UCX, mlx5, ibverbs; skips NCCL/DOCA build. - **DOCA path** (`USE_NIXL=0`): Compiles `internode.cu`; builds NCCL submodule; uses DOCA objects for JIT. - **`.use_nixl` fallback**: If `USE_NIXL` is not set but `.use_nixl` exists in the project root, the NIXL path is used (helps when pip build isolation does not propagate env vars). - **NCCL build failure handling**: When the DOCA path fails (e.g. missing `doca_gpunetio_device.h`), a clear error suggests using `USE_NIXL=1` or creating `.use_nixl`. - **Build logging**: Prints `use_nixl=...` and `-> NIXL path` or `-> DOCA path` during build. ### 3. JIT Compiler (`compiler.cu`) - **USE_NIXL support**: When built with `USE_NIXL`, JIT-compiled inter-node kernels use NIXL libs instead of DOCA objects. - **Runtime logging**: Prints `[HybridEP JIT] Inter-node: using NIXL` or `using DOCA/RDMA` when the JIT compiler initializes. ### 4. Runtime Logging (`hybrid_ep.cu`) - **Init**: Prints `[HybridEP] Inter-node: using NIXL (node_rank=X local_rank=Y)` or `using DOCA/RDMA` when multinode coordinators are initialized. - **allocate_buffers**: Prints `[HybridEP] NIXL allocate_buffers (num_of_nodes=X)` or `DOCA allocate_buffers` for debugging. ### 5. Documentation - **README_Hybrid-EP.md**: - NIXL vs DOCA installation instructions. - Troubleshooting for `doca_gpunetio_device.h` error. - Two-node `hybrid_ep_test` configuration (`WORLD_SIZE`, `RANK`, `MASTER_ADDR`, etc.). - Single-machine simulation with `NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN`. ### 6. Other Fixes - **JIT race fix**: Process-specific JIT cache dir (`.../jit/proc-{pid}`) to avoid races when multiple processes compile concurrently. - **Device API v2 / FP8**: Flag semantics, stride alignment, and GPU context struct updates for NIXL compatibility. ## Build Instructions **NIXL path (recommended when DOCA is unavailable):** ```bash export HYBRID_EP_MULTINODE=1 export USE_NIXL=1 export NIXL_HOME=/usr/local/nixl # Adjust if needed export TORCH_CUDA_ARCH_LIST="9.0 10.0" pip install . ``` **DOCA path (when DOCA SDK is available):** ```bash export HYBRID_EP_MULTINODE=1 export RDMA_CORE_HOME=/path/to/rdma-core pip install . ``` **Fallback when env does not propagate:** ```bash touch .use_nixl HYBRID_EP_MULTINODE=1 pip install . ``` ## Testing - **Single-node**: `python tests/test_hybrid_ep.py --num-processes 4` (unchanged; NIXL not used). - **Multi-node**: Set `WORLD_SIZE=2`, `RANK=0` or `1`, `MASTER_ADDR`, `MASTER_PORT` on each node; run `python tests/test_hybrid_ep.py --num-processes 4` on both. - **Single-machine simulation**: `NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN=2 python tests/test_hybrid_ep.py --num-processes 4` to simulate 2 nodes on 4 GPUs. ## Dependencies - **NIXL path**: NIXL library, UCX, mlx5, ibverbs. Optional: etcd (NIXL_ETCD_ENDPOINTS) for multi-node. - **DOCA path**: NCCL submodule, DOCA SDK, rdma-core (unchanged). ## Backward Compatibility - `USE_NIXL=0` (default) preserves the DOCA path; no behavior change for existing DOCA users. - NIXL code is fully guarded by `#ifdef USE_NIXL`; no NIXL symbols are linked when using DOCA. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-27 21:04:52 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#650
No description provided.