mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #221] What circumstances should low_latency_mode be set to false #172
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#172
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @pengchengneo on GitHub (Jun 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/221
When initializing the buffer, I need to pass in low_latency_mode.
If it is true, low_latency mode is enabled. However, I found that in almost all cases, the performance of setting it to true is better than setting it to false. because we can only use ranks'0 IB when this parameter is false
I would like to ask under what circumstances should this parameter be set to false?
@LyricZhao Thank You!
@LyricZhao commented on GitHub (Jun 18, 2025):
For training and inference prefill, this parameter is
Falseto save GPU memory.It is not expected, they should be same performance, can you please share more information (GPU, EP size, normal or low-latency kernel)?
@pengchengneo commented on GitHub (Jun 18, 2025):
During the prefill phase of inference, if this parameter is set to False, all traffic will be forwarded through IB NIC 0, which will cause a bandwidth bottleneck for the NIC and affect end-to-end throughput.
If it is set to true, the traffic between the NICs is balanced, so that I can get a larger bandwidth and the overall throughput will be higher.
My experimental environment is 48A800, each node is equipped with 4 IB network cards
@LyricZhao
@sphish commented on GitHub (Jun 18, 2025):
For the 8-GPU 4-NIC configuration, NVSHMEM sometimes cannot handle this topology correctly. You may need to manually set
NVSHMEM_HCA_LISTandNVSHMEM_HCA_PE_MAPPINGto map each GPU to the appropriate NIC.@pengchengneo commented on GitHub (Jun 18, 2025):
@sphish Thanks for Ur Reply,
I had setted these two environment variables before raising this issue, but there is still an imbalance in the network card traffic in normal mode.
However, after I set low_latency_mode to true, the NIC traffic becomes balanced and a higher throughput can be achieved.
So I have some doubts here.
@sphish commented on GitHub (Jun 18, 2025):
When you set
low_latency_mode = False, each rank requires a different environment variable setting. It is recommended to also setNVSHMEM_DEBUG=INFOto verify that your GPU is selecting the correct network interface.@pengchengneo commented on GitHub (Jun 18, 2025):
I see, it means that when low_latency_mode = False, I need to set different 'NVSHMEM_HCA_LIST' for each GPU rank ?
if I do like this, will the IB NICs become balanced ? @sphish thanks!
@sphish commented on GitHub (Jun 19, 2025):
I think so.
@alpha-baby commented on GitHub (Jun 19, 2025):
@pc-neo
you can show your GPU and NIC topo, command:
nvidia-smi topo -m.you can show all env you configed.
@Tiktus commented on GitHub (Oct 1, 2025):
@sphish I'm also facing this same challenge when running the internode tests with 2 nodes with 8 x H200. Each node has 8 400GB IB NICs. The scenarios that work with
low_latency_mode = Falseare:NVSHMEM_HCA_PE_MAPPING=1, then all ranks use device_id 0 (mlx5_0) - very poor performance (6 GB/s RDMA perf)NVSHMEM_HCA_PE_MAPPING=0, NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1,then each rank uses all devices which is not optimal - slightly better (10 GB/s RDMA perf)In comparison, when testing internode test with
--test-ll-compatibility, theHCA_PE_MAPPING=1works correctly withNVSHMEM_HCA_PE_MAPPING="mlx5_0:1:1,mlx5_1:1:1,mlx5_2:1:1,mlx5_3:1:1,mlx5_4:1:1,mlx5_5:1:1,mlx5_6:1:1,mlx5_7:1:1"and the performance is +50 GB/s.Ask
Can you explain how I should set the NVSHMEM_HCA_LIST or NVSHMEM_HCA_PE_MAPPING for each GPU rank separately when not using the low-latency options? Does this mean modifying the source code on how the devices are allocated?