[GH-ISSUE #649] DeepEP v2 hybrid mode runs out of QPs with the recommended QPS configuration #384

Closed
opened 2026-10-26 09:59:49 +00:00 by bsakaguchi · 1 comment
Owner

Originally created by @Cinderella0709 on GitHub (May 27, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/649

Hi,

When running DeepEP v2 in hybrid mode with the recommended QPS configuration from the following code:

github.com/deepseek-ai/DeepEP@723716fc11/deep_ep/buffers/elastic.py (L206)

we encounter the following error:

Image

It seems that the process may be exhausting RDMA/NIC resources while creating QPs.

Could this issue be related to host-side RDMA driver or firmware configurations, such as BFREG/UAR limits or other mlx5 resource constraints?

Any suggestions or insights would be greatly appreciated. Thanks in advance!

Originally created by @Cinderella0709 on GitHub (May 27, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/649 Hi, When running DeepEP v2 in hybrid mode with the recommended QPS configuration from the following code: https://github.com/deepseek-ai/DeepEP/blob/723716fc11ee44cbd87035fe9c3a360feca33528/deep_ep/buffers/elastic.py#L206 we encounter the following error: <img width="805" height="187" alt="Image" src="https://github.com/user-attachments/assets/268be46c-1684-456d-9b2c-c7955bca239c" /> It seems that the process may be exhausting RDMA/NIC resources while creating QPs. Could this issue be related to host-side RDMA driver or firmware configurations, such as BFREG/UAR limits or other mlx5 resource constraints? Any suggestions or insights would be greatly appreciated. Thanks in advance!
Author
Owner

@sphish commented on GitHub (Jul 7, 2026):

NCCL Gin created too many QPs, and most of them are unnecessary. NCCL team will fix this issue in the future.

<!-- gh-comment-id:4901492064 --> @sphish commented on GitHub (Jul 7, 2026): NCCL Gin created too many QPs, and most of them are unnecessary. NCCL team will fix this issue in the future.
Sign in to join this conversation.
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#384
No description provided.