[GH-ISSUE #505] [Question] Question on using num_worst_tokens #326

New issue

Open

opened 2026-10-26 09:58:39 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:58:39 +00:00

Owner

Originally created by @yuhyao on GitHub (Nov 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/505

Hi DeepSeek team,

I noticed that num_worst_tokens is included in the arguments of dispatch, which makes sense for CUDA Graph support. However, it seems that the values written to moe_recv_counter and moe_recv_expert_counter are no longer directly accessible. From the code, it looks like these counters can be recomputed from recv_topk_idx, since padded entries are set to -1.

The prefill trace also doesn’t show any H2D memcpy after the dispatch kernel, so I assume you are recomputing those counters inside another kernel?

Would it be possible to write these counters into CUDA tensors directly? For example, open-source projects like SGLang simply copy num_recv_tokens_per_expert_list to device memory. Having these values available on CUDA would make it easier to switch to CPU-async mode and may also help avoid redundant memory accesses.

Thanks!

Originally created by @yuhyao on GitHub (Nov 28, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/505 Hi DeepSeek team, I noticed that `num_worst_tokens` is included in the arguments of dispatch, which makes sense for CUDA Graph support. However, it seems that the values written to `moe_recv_counter` and `moe_recv_expert_counter` are no longer directly accessible. From the code, it looks like these counters can be recomputed from `recv_topk_idx`, since padded entries are set to `-1`. The [prefill trace](https://github.com/deepseek-ai/profile-data/blob/main/prefill.json) also doesn’t show any H2D memcpy after the dispatch kernel, so I assume you are recomputing those counters inside another kernel? Would it be possible to write these counters into CUDA tensors directly? For example, open-source projects like SGLang simply copy `num_recv_tokens_per_expert_list` to device memory. Having these values available on CUDA would make it easier to switch to CPU-async mode and may also help avoid redundant memory accesses. Thanks!

bsakaguchi commented

2026-10-26 09:58:40 +00:00

Author

Owner

@LyricZhao commented on GitHub (Dec 5, 2025):

We don't have CUDA graph in prefill, and we use num_recv_tokens_per_expert_list just for backward. For contiguous GEMM's input, we have a get_fused_mapping kernel to do the compute from topk_idx.

We are releasing V2 soon, by then, the dispatched results will automatically expand tokens into experts and return a layout for the GEMM as well.

@LyricZhao commented on GitHub (Dec 5, 2025): We don't have CUDA graph in prefill, and we use `num_recv_tokens_per_expert_list` just for backward. For contiguous GEMM's input, we have a `get_fused_mapping` kernel to do the compute from `topk_idx`. We are releasing V2 soon, by then, the dispatched results will automatically expand tokens into experts and return a layout for the GEMM as well.

bsakaguchi commented

2026-10-26 09:58:41 +00:00

Author

Owner

@yuhyao commented on GitHub (Dec 8, 2025):

We don't have CUDA graph in prefill, and we use num_recv_tokens_per_expert_list just for backward. For contiguous GEMM's input, we have a get_fused_mapping kernel to do the compute from topk_idx.

We are releasing V2 soon, by then, the dispatched results will automatically expand tokens into experts and return a layout for the GEMM as well.

Thanks for the reply, and thanks for the exciting news about DeepEP v2!
May I ask about the estimated release timeline? Is it expected in a few weeks, or more likely a few months?

@yuhyao commented on GitHub (Dec 8, 2025): > We don't have CUDA graph in prefill, and we use `num_recv_tokens_per_expert_list` just for backward. For contiguous GEMM's input, we have a `get_fused_mapping` kernel to do the compute from `topk_idx`. > > We are releasing V2 soon, by then, the dispatched results will automatically expand tokens into experts and return a layout for the GEMM as well. Thanks for the reply, and thanks for the exciting news about DeepEP v2! May I ask about the estimated release timeline? Is it expected in a few weeks, or more likely a few months?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#326

No description provided.

Rows
Columns