[GH-ISSUE #509] Hang in CPU-assisted IBGDA mode #328

New issue

Open

opened 2026-10-26 09:58:41 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:58:41 +00:00

Owner

Originally created by @YangZhou1997 on GitHub (Dec 3, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/509

Hi DeepEP developers!

I am trying to run DeepEP under CPU-assisted IBGDA mode following https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md#22-install-gdrcopy-and-load-the-gdrdrv-kernel-module. gdrcopy_sanity and nvshmrun -n 2 ./shmem_put_bw work. But both LL and normal kernel test will hang after:

WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path.
WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path.
WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path.
Buffer initialized.

The servers are standard Nebius cloud H100+CX7 IB 2-node testbed, using pip install nvidia-nvshmem-cu12 (should be version 3.4.5 iirc). Any thoughts or suggestions on this? My goal is basically to see the performance of LL and normal kernels with CPU-assisted IBGDA.

Correct me if I was wrong: I saw some issues mentioned that the latest DeepEP does not support IBRC, but I guess it should support CPU-assisted IBGDA, which should have the same interface as native IBGDA.

Another side question is: what is the security vulnerability specifically about when using native IBGDA? What is the root cause, and what are the security implications, especially in public cloud? I saw this blog https://developer.nvidia.com/blog/enhancing-application-portability-and-compatibility-across-new-platforms-using-nvidia-magnum-io-nvshmem-3-0/#cpu-assisted_infiniband_gpu_direct_async%C2%A0, but it does not dive deep into it... and I could not find a more detailed description about it

Best,
Yang

Originally created by @YangZhou1997 on GitHub (Dec 3, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/509 Hi DeepEP developers! I am trying to run DeepEP under CPU-assisted IBGDA mode following https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md#22-install-gdrcopy-and-load-the-gdrdrv-kernel-module. `gdrcopy_sanity` and `nvshmrun -n 2 ./shmem_put_bw` work. But both LL and normal kernel test will hang after: ``` WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path. WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path. WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path. Buffer initialized. ``` The servers are standard Nebius cloud H100+CX7 IB 2-node testbed, using `pip install nvidia-nvshmem-cu12` (should be version 3.4.5 iirc). Any thoughts or suggestions on this? My goal is basically to see the performance of LL and normal kernels with CPU-assisted IBGDA. Correct me if I was wrong: I saw some issues mentioned that the latest DeepEP does not support IBRC, but I guess it should support CPU-assisted IBGDA, which should have the same interface as native IBGDA. Another side question is: what is the security vulnerability specifically about when using native IBGDA? What is the root cause, and what are the security implications, especially in public cloud? I saw this blog https://developer.nvidia.com/blog/enhancing-application-portability-and-compatibility-across-new-platforms-using-nvidia-magnum-io-nvshmem-3-0/#cpu-assisted_infiniband_gpu_direct_async%C2%A0, but it does not dive deep into it... and I could not find a more detailed description about it Best, Yang

bsakaguchi commented

2026-10-26 09:58:42 +00:00

Author

Owner

@AshOfCat commented on GitHub (Dec 8, 2025):

Has your problem been resolved? I'm encountering the same issue.

@AshOfCat commented on GitHub (Dec 8, 2025): Has your problem been resolved? I'm encountering the same issue.

bsakaguchi commented

2026-10-26 09:58:42 +00:00

Author

Owner

@YangZhou1997 commented on GitHub (Dec 8, 2025):

Unfortunately no. Btw, I am working on UCCL ep that supports scenarios where ibgda cannot be used: https://github.com/uccl-project/uccl/tree/main/ep. It provides drop-in compatibility for deepep.

@YangZhou1997 commented on GitHub (Dec 8, 2025): Unfortunately no. Btw, I am working on UCCL ep that supports scenarios where ibgda cannot be used: https://github.com/uccl-project/uccl/tree/main/ep. It provides drop-in compatibility for deepep.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#328

No description provided.

Rows
Columns