[GH-ISSUE #72] Question about interface of low_latency_dispatch #67

New issue

Open

opened 2026-10-26 09:53:39 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 09:53:39 +00:00

Owner

Originally created by @JimmyWang1850 on GitHub (Mar 13, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/72

I'm a bit confused about the low_latency_dispatch interface. Why doesn't the interface support passing topk_weights? If the dispatch interface cannot return topk_weights, then it is necessary to

recalculate based on the obtained hidden stats
or use the dispatch interface again to transmit and obtain topk_weights.
However, based on our actual tests, the transmission time for all_gather during the decode stage is very short, so adding recalculations or additional transmissions does not offer any advantage. Or is it related to your experts' interface?

Originally created by @JimmyWang1850 on GitHub (Mar 13, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/72 I'm a bit confused about the low_latency_dispatch interface. Why doesn't the interface **support passing topk_weights**? If the dispatch interface cannot return topk_weights, then it is necessary to 1. recalculate based on the obtained hidden stats 2. or use the dispatch interface again to transmit and obtain topk_weights. However, based on our actual tests, the transmission time for all_gather during the decode stage is very short, so adding recalculations or additional transmissions does not offer any advantage. Or is it related to your experts' interface?

bsakaguchi commented

2026-10-26 09:53:40 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 13, 2025):

Low-latency API is designed for inference, and if the MoE part does not have local reduction before combine, then it is not needed.

And as the low-latency combine is at one level (normal kernels are at 2 levels: RDMA and NVLink domain two reductions), so topk_weights is only used at combine after receiving the data.

@LyricZhao commented on GitHub (Mar 13, 2025): Low-latency API is designed for inference, and if the MoE part does not have local reduction before combine, then it is not needed. And as the low-latency combine is at one level (normal kernels are at 2 levels: RDMA and NVLink domain two reductions), so `topk_weights` is only **used at combine** after receiving the data.

bsakaguchi commented

2026-10-26 09:53:41 +00:00

Author

Owner

@wujingyue commented on GitHub (Feb 2, 2026):

Follow-up question:

if the MoE part does not have local reduction before combine, then it is not needed.

I got that. But why does intranode_dispatch also produce topk_weights? Aren't intranode_dispatch and intranode_combine "one level" (i.e. via nvlink only)?

github.com/deepseek-ai/DeepEP@29d31c0957/deep_ep/buffer.py (L393)

@wujingyue commented on GitHub (Feb 2, 2026): Follow-up question: > if the MoE part does not have local reduction before combine, then it is not needed. I got that. But why does `intranode_dispatch` **also** produce topk_weights? Aren't `intranode_dispatch` and `intranode_combine` "one level" (i.e. via nvlink only)? https://github.com/deepseek-ai/DeepEP/blob/29d31c095796f3c8ece47ee9cdcc167051bbeed9/deep_ep/buffer.py#L393

bsakaguchi commented

2026-10-26 09:53:41 +00:00

Author

Owner

@wujingyue commented on GitHub (Feb 2, 2026):

as the low-latency combine is at one level, so topk_weights is only used at combine after receiving the data.

Actually, I'm not even sure about this any more. IIUC, performing the multiplication of top-k weights on the sender side is likely more optimal. It enables in-fabric reduction and minimizes the receiver buffer footprint, both of which are critical for minimizing inference latency.

For your reference, github.com/deepseek-ai/DeepEP@29d31c0957/csrc/kernels/internode_ll.cu (L1078) assumes topk_weights is in the original sequence order.

@wujingyue commented on GitHub (Feb 2, 2026): > as the low-latency combine is at one level, so topk_weights is only used at combine after receiving the data. Actually, I'm not even sure about this any more. IIUC, performing the multiplication of top-k weights on the sender side is likely more optimal. It enables in-fabric reduction and minimizes the receiver buffer footprint, both of which are critical for minimizing inference latency. For your reference, https://github.com/deepseek-ai/DeepEP/blob/29d31c095796f3c8ece47ee9cdcc167051bbeed9/csrc/kernels/internode_ll.cu#L1078 assumes `topk_weights` is in the original sequence order.

bsakaguchi referenced this issue

2026-10-26 10:00:38 +00:00

[PR #67] [MERGED] Update NVSHMEM to v3.2.5. #429