[PR #749] [CLOSED] [HybridEP] Report RDMA memory registration failures #753

New issue

Closed

opened 2026-10-27 21:05:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/749
Author: @janbernloehr
Created: 9/11/2026
Status: ❌ Closed

Base: hybrid-ep ← Head: fix/hybridep-rdma-registration-errors

📝 Commits (1)

7e1d572 fix(hybrid-ep): report RDMA registration failures

📊 Changes

1 file changed (+136 additions, -30 deletions)

View changed files

📝 csrc/hybrid_ep/buffer/internode_doca.cu (+136 -30)

📄 Description

Motivation

The HybridEP DOCA backend currently assumes that every ibv_reg_mr call succeeds. If GPU memory registration fails, the returned null pointer is dereferenced later while constructing remote-memory metadata. This produces a generic segmentation fault and loses the original errno, making failures such as unavailable peer-memory support, an incompatible HCA, or insufficient memlock limits difficult to diagnose.

Changes

Add a checked wrapper for ibv_reg_mr.
Route all 14 dispatch and combine memory-registration sites through it.
On failure, report the buffer name, address, size, access flags, IB device, errno, and RLIMIT_MEMLOCK values before terminating.
Include guidance about the DOCA backend's current legacy ibv_reg_mr path and the absence of a DMA-BUF registration fallback.
Flush the diagnostic before aborting instead of allowing a later null-pointer dereference.

This change improves error reporting only; it does not add DMA-BUF registration support.

Performance impact

There are no changes to dispatch, combine, or training hot paths. The wrapper is inline and is used only during one-time RDMA buffer initialization. On success, its only additional work is a null check after the existing ibv_reg_mr call.

Testing

git diff --check
Verified that all 14 HybridEP DOCA registration sites use the checked helper and that the helper contains the only remaining direct ibv_reg_mr call in internode_doca.cu.
Built the patched HybridEP wheel successfully in an uncached amd64 MBridge image with CUDA/DOCA, NVSHMEM 3.7.2, and rdma-core 5321d809e095d6dd32a1355a5d4aa2ebbda4ba74.
Ran an 8-node, 64-rank Nemotron 3 Super BF16 HybridEP workload on B200 nodes that provide DMA-BUF rather than nvidia_peermem. The new diagnostic consistently reported ibv_reg_mr failure for combine.rdma_intra_node_red_token (117,440,512 bytes) on the rank-selected HCA with errno=14 (Bad address) and unlimited soft/hard RLIMIT_MEMLOCK, then aborted before the null memory-region pointer could be dereferenced.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/749 **Author:** [@janbernloehr](https://github.com/janbernloehr) **Created:** 9/11/2026 **Status:** ❌ Closed **Base:** `hybrid-ep` ← **Head:** `fix/hybridep-rdma-registration-errors` --- ### 📝 Commits (1) - [`7e1d572`](https://github.com/deepseek-ai/DeepEP/commit/7e1d572af9e98293552af821cdcc43a26d159487) fix(hybrid-ep): report RDMA registration failures ### 📊 Changes **1 file changed** (+136 additions, -30 deletions) <details> <summary>View changed files</summary> 📝 `csrc/hybrid_ep/buffer/internode_doca.cu` (+136 -30) </details> ### 📄 Description ## Motivation The HybridEP DOCA backend currently assumes that every `ibv_reg_mr` call succeeds. If GPU memory registration fails, the returned null pointer is dereferenced later while constructing remote-memory metadata. This produces a generic segmentation fault and loses the original `errno`, making failures such as unavailable peer-memory support, an incompatible HCA, or insufficient memlock limits difficult to diagnose. ## Changes - Add a checked wrapper for `ibv_reg_mr`. - Route all 14 dispatch and combine memory-registration sites through it. - On failure, report the buffer name, address, size, access flags, IB device, `errno`, and `RLIMIT_MEMLOCK` values before terminating. - Include guidance about the DOCA backend's current legacy `ibv_reg_mr` path and the absence of a DMA-BUF registration fallback. - Flush the diagnostic before aborting instead of allowing a later null-pointer dereference. This change improves error reporting only; it does not add DMA-BUF registration support. ## Performance impact There are no changes to dispatch, combine, or training hot paths. The wrapper is inline and is used only during one-time RDMA buffer initialization. On success, its only additional work is a null check after the existing `ibv_reg_mr` call. ## Testing - `git diff --check` - Verified that all 14 HybridEP DOCA registration sites use the checked helper and that the helper contains the only remaining direct `ibv_reg_mr` call in `internode_doca.cu`. - Built the patched HybridEP wheel successfully in an uncached amd64 MBridge image with CUDA/DOCA, NVSHMEM 3.7.2, and rdma-core `5321d809e095d6dd32a1355a5d4aa2ebbda4ba74`. - Ran an 8-node, 64-rank Nemotron 3 Super BF16 HybridEP workload on B200 nodes that provide DMA-BUF rather than `nvidia_peermem`. The new diagnostic consistently reported `ibv_reg_mr` failure for `combine.rdma_intra_node_red_token` (117,440,512 bytes) on the rank-selected HCA with `errno=14 (Bad address)` and unlimited soft/hard `RLIMIT_MEMLOCK`, then aborted before the null memory-region pointer could be dereferenced. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:54 +00:00