deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #415] low_latency_test traffic in dual-port networks is unbalanced #280

New issue
Closed
opened 2026-10-26 09:57:53 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-26 09:57:53 +00:00
Owner
Copy link

Originally created by @PhdShi on GitHub (Sep 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/415

After the test_low_latency test, it was found that the low latency IBGDA qp created through mlx5dv_devx_obj_create would only go through one of the ports in the dual-port network, even though we created num_experts/nranks qps.
For instance, for 2 H20 servers with nranks=16 and 288 experts, by modifying the topk tensor to make it that Rank 0 data will only be sent to Rank 8, only one eth port will be used.
Is it as expected? Thanks.

Image
Originally created by @PhdShi on GitHub (Sep 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/415 After the test_low_latency test, it was found that the low latency IBGDA qp created through mlx5dv_devx_obj_create would only go through one of the ports in the dual-port network, even though we created num_experts/nranks qps. For instance, for 2 H20 servers with nranks=16 and 288 experts, by modifying the topk tensor to make it that Rank 0 data will only be sent to Rank 8, only one eth port will be used. Is it as expected? Thanks. <img width="1016" height="85" alt="Image" src="https://github.com/user-attachments/assets/db5b043e-fb74-4e56-afc2-5486552e140c" />
bsakaguchi commented 2026-10-26 09:57:55 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Sep 18, 2025):

Hello, what command did you use to check the real-time bandwidth of the device? It seems that this is not in line with expectations.
@PhdShi

<!-- gh-comment-id:3305347892 --> @alpha-baby commented on GitHub (Sep 18, 2025): Hello, what command did you use to check the real-time bandwidth of the device? It seems that this is not in line with expectations. @PhdShi
bsakaguchi commented 2026-10-26 09:57:55 +00:00
Author
Owner
Copy link

@PhdShi commented on GitHub (Sep 18, 2025):

Hello, what command did you use to check the real-time bandwidth of the device? It seems that this is not in line with expectations. @PhdShi

rdma_monitor

<!-- gh-comment-id:3305616171 --> @PhdShi commented on GitHub (Sep 18, 2025): > Hello, what command did you use to check the real-time bandwidth of the device? It seems that this is not in line with expectations. [@PhdShi](https://github.com/PhdShi) rdma_monitor
bsakaguchi commented 2026-10-26 09:57:56 +00:00
Author
Owner
Copy link

@PhdShi commented on GitHub (Sep 19, 2025):

fixed by modifying the udp src port of nvshmem IBGDA

<!-- gh-comment-id:3310418417 --> @PhdShi commented on GitHub (Sep 19, 2025): fixed by modifying the udp src port of nvshmem IBGDA
bsakaguchi commented 2026-10-26 09:57:56 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Sep 19, 2025):

fixed by modifying the udp src port of nvshmem IBGDA

How did you modify it? I understand that this udp-src port should not be a fixed value by default.

<!-- gh-comment-id:3310718987 --> @alpha-baby commented on GitHub (Sep 19, 2025): > fixed by modifying the udp src port of nvshmem IBGDA How did you modify it? I understand that this udp-src port should not be a fixed value by default.
bsakaguchi commented 2026-10-26 09:57:56 +00:00
Author
Owner
Copy link

@PhdShi commented on GitHub (Sep 19, 2025):

fixed by modifying the udp src port of nvshmem IBGDA

How did you modify it? I understand that this udp-src port should not be a fixed value by default.

I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created

<!-- gh-comment-id:3310775907 --> @PhdShi commented on GitHub (Sep 19, 2025): > > fixed by modifying the udp src port of nvshmem IBGDA > > How did you modify it? I understand that this udp-src port should not be a fixed value by default. I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created
bsakaguchi commented 2026-10-26 09:57:56 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Sep 19, 2025):

fixed by modifying the udp src port of nvshmem IBGDA

How did you modify it? I understand that this udp-src port should not be a fixed value by default.

I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created

I didn't find the magic number related to 49152 in the nvshmem source code. how do you find this magic number of 49152. tcpdump ?

how do you modify udp-src port ？Configure an environment variable with NVSHMEM_ as the prefix?

<!-- gh-comment-id:3310822191 --> @alpha-baby commented on GitHub (Sep 19, 2025): > > > fixed by modifying the udp src port of nvshmem IBGDA > > > > > > How did you modify it? I understand that this udp-src port should not be a fixed value by default. > > I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created I didn't find the magic number related to 49152 in the nvshmem source code. how do you find this magic number of 49152. tcpdump ? how do you modify udp-src port ？Configure an environment variable with NVSHMEM_ as the prefix?
bsakaguchi commented 2026-10-26 09:57:56 +00:00
Author
Owner
Copy link

@PhdShi commented on GitHub (Sep 19, 2025):

fixed by modifying the udp src port of nvshmem IBGDA

How did you modify it? I understand that this udp-src port should not be a fixed value by default.

I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created

I didn't find the magic number related to 49152 in the nvshmem source code. how do you find this magic number of 49152. tcpdump ?

how do you modify udp-src port ？Configure an environment variable with NVSHMEM_ as the prefix?

modify nvshmem source code.
#define IBGDA_ROCE_V2_UDP_SPORT_BASE 0xC000

<!-- gh-comment-id:3310838791 --> @PhdShi commented on GitHub (Sep 19, 2025): > > > > fixed by modifying the udp src port of nvshmem IBGDA > > > > > > > > > How did you modify it? I understand that this udp-src port should not be a fixed value by default. > > > > > > I think your understanding is correct. However, nvshmem change this value to 49152 after IBGDA RC is created > > I didn't find the magic number related to 49152 in the nvshmem source code. how do you find this magic number of 49152. tcpdump ? > > how do you modify udp-src port ？Configure an environment variable with NVSHMEM_ as the prefix? modify nvshmem source code. #define IBGDA_ROCE_V2_UDP_SPORT_BASE 0xC000
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#280
No description provided.