[PR #748] [CLOSED] [HybridEP] Report RDMA memory registration failures #755

New issue

Closed

opened 2026-10-27 21:05:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/748
Author: @janbernloehr
Created: 9/11/2026
Status: ❌ Closed

Base: main ← Head: fix/hybridep-rdma-registration-errors

📝 Commits (10+)

c9f647d Add HybridEP (#420)
6ad6896 add document of Hybrid-EP (#428)
476ad70 [Feature] Support normal mode for non-nvlink environment (#375)
e713f52 fix issue with align, recover format for csrc/kernels/internode.cu and (#431)
c5a3e9b Add Jit compiler (#433)
f4f2c10 add permute/unpermute, add sync free, refactor (#439)
85fba86 Jershi/fp4 intranode ready (#339)
0ed15b5 fix bug on token drop (#443)
1242eb4 Decouple the dispatch and buffer allocation on hybrid-ep (#448)
3f601f7 Improve Jit (#457)

📊 Changes

65 files changed (+21907 additions, -216 deletions)

View changed files

➕ .gitmodules (+4 -0)
📝 README.md (+13 -0)
📝 csrc/config.hpp (+26 -0)
📝 csrc/deep_ep.cpp (+620 -52)
📝 csrc/deep_ep.hpp (+69 -7)
➕ csrc/hybrid_ep/allocator/allocator.cu (+212 -0)
➕ csrc/hybrid_ep/allocator/allocator.cuh (+63 -0)
➕ csrc/hybrid_ep/backend/NCCL_LICENSE.txt (+38 -0)
➕ csrc/hybrid_ep/backend/hybrid_ep_backend.cuh (+6112 -0)
➕ csrc/hybrid_ep/backend/ibvcore.h (+271 -0)
➕ csrc/hybrid_ep/backend/topo_detection.cuh (+1657 -0)
➕ csrc/hybrid_ep/buffer/internode.cuh (+15 -0)
➕ csrc/hybrid_ep/buffer/internode_doca.cu (+971 -0)
➕ csrc/hybrid_ep/buffer/internode_doca.cuh (+190 -0)
➕ csrc/hybrid_ep/buffer/internode_nixl.cu (+272 -0)
➕ csrc/hybrid_ep/buffer/internode_nixl.cuh (+57 -0)
➕ csrc/hybrid_ep/buffer/intranode.cu (+453 -0)
➕ csrc/hybrid_ep/buffer/intranode.cuh (+90 -0)
➕ csrc/hybrid_ep/buffer/nixl_connector.cu (+934 -0)
➕ csrc/hybrid_ep/buffer/nixl_connector.h (+146 -0)

...and 45 more files

📄 Description

Motivation

The HybridEP DOCA backend currently assumes that every ibv_reg_mr call succeeds. If GPU memory registration fails, the returned null pointer is dereferenced later while constructing remote-memory metadata. This produces a generic segmentation fault and loses the original errno, making failures such as unavailable peer-memory support, an incompatible HCA, or insufficient memlock limits difficult to diagnose.

Changes

Add a checked wrapper for ibv_reg_mr.
Route all 14 dispatch and combine memory-registration sites through it.
On failure, report the buffer name, address, size, access flags, IB device, errno, and RLIMIT_MEMLOCK values before terminating.
Include guidance about the DOCA backend's current legacy ibv_reg_mr path and the absence of a DMA-BUF registration fallback.
Flush the diagnostic before aborting instead of allowing a later null-pointer dereference.

This change improves error reporting only; it does not add DMA-BUF registration support.

Performance impact

There are no changes to dispatch, combine, or training hot paths. The wrapper is inline and is used only during one-time RDMA buffer initialization. On success, its only additional work is a null check after the existing ibv_reg_mr call.

Testing

git diff --check
Verified that all 14 HybridEP DOCA registration sites use the checked helper and that the helper contains the only remaining direct ibv_reg_mr call in internode_doca.cu.
Built the patched HybridEP wheel successfully in an uncached amd64 MBridge image with CUDA/DOCA, NVSHMEM 3.7.2, and rdma-core 5321d809e095d6dd32a1355a5d4aa2ebbda4ba74.
Ran an 8-node, 64-rank Nemotron 3 Super BF16 HybridEP workload on B200 nodes that provide DMA-BUF rather than nvidia_peermem. The new diagnostic consistently reported ibv_reg_mr failure for combine.rdma_intra_node_red_token (117,440,512 bytes) on the rank-selected HCA with errno=14 (Bad address) and unlimited soft/hard RLIMIT_MEMLOCK, then aborted before the null memory-region pointer could be dereferenced.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/748 **Author:** [@janbernloehr](https://github.com/janbernloehr) **Created:** 9/11/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/hybridep-rdma-registration-errors` --- ### 📝 Commits (10+) - [`c9f647d`](https://github.com/deepseek-ai/DeepEP/commit/c9f647dd4108539e32d4375c932d0d44279d45ba) Add HybridEP (#420) - [`6ad6896`](https://github.com/deepseek-ai/DeepEP/commit/6ad6896ddcc4f02531a008851006af1e191d9cd4) add document of Hybrid-EP (#428) - [`476ad70`](https://github.com/deepseek-ai/DeepEP/commit/476ad704133e04942ce113ec3ffdd028abb2fa5e) [Feature] Support normal mode for non-nvlink environment (#375) - [`e713f52`](https://github.com/deepseek-ai/DeepEP/commit/e713f5296157e6e0d7d465b1f2d13383a8791d93) fix issue with align, recover format for csrc/kernels/internode.cu and (#431) - [`c5a3e9b`](https://github.com/deepseek-ai/DeepEP/commit/c5a3e9b790caedecfe362bc206c186185d8bfaab) Add Jit compiler (#433) - [`f4f2c10`](https://github.com/deepseek-ai/DeepEP/commit/f4f2c10aa57d3155a1320b0c26a6e70d5746d204) add permute/unpermute, add sync free, refactor (#439) - [`85fba86`](https://github.com/deepseek-ai/DeepEP/commit/85fba86d7f6e3e6e04d19af3cfd7929622facdff) Jershi/fp4 intranode ready (#339) - [`0ed15b5`](https://github.com/deepseek-ai/DeepEP/commit/0ed15b59f0f1d61504cc7e29d79e8c219b8b0e0a) fix bug on token drop (#443) - [`1242eb4`](https://github.com/deepseek-ai/DeepEP/commit/1242eb4fc1410ac37bac81003298a6d5be4922d7) Decouple the dispatch and buffer allocation on hybrid-ep (#448) - [`3f601f7`](https://github.com/deepseek-ai/DeepEP/commit/3f601f7ac1c062c46502646ff04c535013bfca00) Improve Jit (#457) ### 📊 Changes **65 files changed** (+21907 additions, -216 deletions) <details> <summary>View changed files</summary> ➕ `.gitmodules` (+4 -0) 📝 `README.md` (+13 -0) 📝 `csrc/config.hpp` (+26 -0) 📝 `csrc/deep_ep.cpp` (+620 -52) 📝 `csrc/deep_ep.hpp` (+69 -7) ➕ `csrc/hybrid_ep/allocator/allocator.cu` (+212 -0) ➕ `csrc/hybrid_ep/allocator/allocator.cuh` (+63 -0) ➕ `csrc/hybrid_ep/backend/NCCL_LICENSE.txt` (+38 -0) ➕ `csrc/hybrid_ep/backend/hybrid_ep_backend.cuh` (+6112 -0) ➕ `csrc/hybrid_ep/backend/ibvcore.h` (+271 -0) ➕ `csrc/hybrid_ep/backend/topo_detection.cuh` (+1657 -0) ➕ `csrc/hybrid_ep/buffer/internode.cuh` (+15 -0) ➕ `csrc/hybrid_ep/buffer/internode_doca.cu` (+971 -0) ➕ `csrc/hybrid_ep/buffer/internode_doca.cuh` (+190 -0) ➕ `csrc/hybrid_ep/buffer/internode_nixl.cu` (+272 -0) ➕ `csrc/hybrid_ep/buffer/internode_nixl.cuh` (+57 -0) ➕ `csrc/hybrid_ep/buffer/intranode.cu` (+453 -0) ➕ `csrc/hybrid_ep/buffer/intranode.cuh` (+90 -0) ➕ `csrc/hybrid_ep/buffer/nixl_connector.cu` (+934 -0) ➕ `csrc/hybrid_ep/buffer/nixl_connector.h` (+146 -0) _...and 45 more files_ </details> ### 📄 Description ## Motivation The HybridEP DOCA backend currently assumes that every `ibv_reg_mr` call succeeds. If GPU memory registration fails, the returned null pointer is dereferenced later while constructing remote-memory metadata. This produces a generic segmentation fault and loses the original `errno`, making failures such as unavailable peer-memory support, an incompatible HCA, or insufficient memlock limits difficult to diagnose. ## Changes - Add a checked wrapper for `ibv_reg_mr`. - Route all 14 dispatch and combine memory-registration sites through it. - On failure, report the buffer name, address, size, access flags, IB device, `errno`, and `RLIMIT_MEMLOCK` values before terminating. - Include guidance about the DOCA backend's current legacy `ibv_reg_mr` path and the absence of a DMA-BUF registration fallback. - Flush the diagnostic before aborting instead of allowing a later null-pointer dereference. This change improves error reporting only; it does not add DMA-BUF registration support. ## Performance impact There are no changes to dispatch, combine, or training hot paths. The wrapper is inline and is used only during one-time RDMA buffer initialization. On success, its only additional work is a null check after the existing `ibv_reg_mr` call. ## Testing - `git diff --check` - Verified that all 14 HybridEP DOCA registration sites use the checked helper and that the helper contains the only remaining direct `ibv_reg_mr` call in `internode_doca.cu`. - Built the patched HybridEP wheel successfully in an uncached amd64 MBridge image with CUDA/DOCA, NVSHMEM 3.7.2, and rdma-core `5321d809e095d6dd32a1355a5d4aa2ebbda4ba74`. - Ran an 8-node, 64-rank Nemotron 3 Super BF16 HybridEP workload on B200 nodes that provide DMA-BUF rather than `nvidia_peermem`. The new diagnostic consistently reported `ibv_reg_mr` failure for `combine.rdma_intra_node_red_token` (117,440,512 bytes) on the rank-selected HCA with `errno=14 (Bad address)` and unlimited soft/hard `RLIMIT_MEMLOCK`, then aborted before the null memory-region pointer could be dereferenced. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:58 +00:00