[GH-ISSUE #301] How does DeepEP prevent send queue overflows? #214

New issue

Closed

opened 2026-10-26 09:56:45 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:56:45 +00:00

Owner

Originally created by @itayalroy on GitHub (Jul 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/301

Hi guys, It doesn't look like dispatch/combine kernels (both in ll mode & regular mode) take send queue occupancy into account:

LL mode has no flow control at all, it just submits all tokens on their corresponding QPs immediately
Regular mode has this head/tail flow control but tail-head is allowed to be up to 256 (while it looks like qp size in nvshmem is hardcoded to 128), also there are atomic writes that consume wqes as well

How does DeepEP prevent send queue overflow?

Originally created by @itayalroy on GitHub (Jul 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/301 Hi guys, It doesn't look like dispatch/combine kernels (both in ll mode & regular mode) take send queue occupancy into account: - LL mode has no flow control at all, it just submits all tokens on their corresponding QPs immediately - Regular mode has this head/tail flow control but `tail-head` is allowed to be up to 256 (while it looks like qp size in nvshmem is hardcoded to 128), also there are atomic writes that consume wqes as well How does DeepEP prevent send queue overflow?

bsakaguchi closed this issue

2026-10-26 09:56:45 +00:00

bsakaguchi commented

2026-10-26 09:56:48 +00:00

Author

Owner

@sphish commented on GitHub (Jul 15, 2025):

The QP size in NVSHMEM is not hardcoded, it can be set by the env var NVSHMEM_QP_DEPTH. In DeepEP, the variable is set to 1024 by default.

For performance reasons, we have removed send queue polling. At the same time, we ensure that the number of inflight send wrs does not exceed the QP size to prevent overflow.

For the normal kernels, the maximum number of inflight send wrs is num_tokens / num_channels / send_chunk_size * 2.
For the low latency kernel, the maximum number of inflight send wrs is num_tokens + 1

For the vast majority of practical cases, the QP size of 1024 is sufficient. If higher capacity is required, you can manually adjust it.

@sphish commented on GitHub (Jul 15, 2025): The QP size in NVSHMEM is not hardcoded, it can be set by the env var `NVSHMEM_QP_DEPTH`. In DeepEP, the variable is set to 1024 by default. For performance reasons, we have removed send queue polling. At the same time, we ensure that the number of inflight send wrs does not exceed the QP size to prevent overflow. - For the normal kernels, the maximum number of inflight send wrs is `num_tokens / num_channels / send_chunk_size * 2`. - For the low latency kernel, the maximum number of inflight send wrs is `num_tokens + 1` For the vast majority of practical cases, the QP size of 1024 is sufficient. If higher capacity is required, you can manually adjust it.

bsakaguchi referenced this issue

2026-10-26 10:01:06 +00:00

[PR #214] [MERGED] Remove the low-latency usage flag #468