deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #165] Setting deepep to low_latency mode fails to start model serving #144

New issue
Open
opened 2026-10-26 09:55:11 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:55:11 +00:00
Owner
Copy link

Originally created by @ch-tiger1 on GitHub (May 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/165

When I set the deepep mode to auto and normal, the stress test is normal, but when I compare the low_latency mode, the following error occurs. RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank' Has anyone encountered this before? What should I do to solve it?

[2025-05-14 03:58:33 DP0 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 116, in forward_thread_func
    self.forward_thread_func_()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 147, in forward_thread_func_
    logits_output, next_token_ids = self.worker.forward_batch_generation(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 181, in forward_batch_generation
    logits_output = self.model_runner.forward(forward_batch)
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1122, in forward
    return self._forward_raw(forward_batch, skip_attn_backend_init)
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1155, in _forward_raw
    return self.forward_extend(
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1064, in forward_extend
    return self.model.forward(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 2099, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1916, in forward
    hidden_states, residual = layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1462, in forward
    return self.forward_ffn_with_scattered_input(
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1617, in forward_ffn_with_scattered_input
    hidden_states = self.mlp(hidden_states, forward_batch.forward_mode)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 355, in forward
    return self.forward_deepep(hidden_states, forward_mode)
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 386, in forward_deepep
    self._forward_deepep_dispatch_a(
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 466, in _forward_deepep_dispatch_a
    chosen_deepep_dispatcher.dispatch_a(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 713, in dispatch_a
    inner_state = self._get_impl(forward_mode).dispatch_a(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 519, in dispatch_a
    hidden_states, masked_m, event, hook = self._dispatch_core(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 603, in _dispatch_core
    buffer.low_latency_dispatch(
  File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 487, in low_latency_dispatch
    self.runtime.low_latency_dispatch(x, topk_idx,
RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank'

[2025-05-14 03:58:33] Child process unexpectedly failed with an exit code 131. pid=255
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=1401
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2194
[2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2198

cmd

export SGL_ENABLE_JIT_DEEPGEMM=1
python3 -m sglang.launch_server \
        --model $model \
        --trust-remote-code \
        --tp 8 \
        --dp 8 \
        --enable-dp-attention \
        --enable-deepep-moe \
        --deepep-mode low_latency \
        --enable-metrics
Originally created by @ch-tiger1 on GitHub (May 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/165 When I set the deepep mode to auto and normal, the stress test is normal, but when I compare the low_latency mode, the following error occurs. `RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank' `Has anyone encountered this before? What should I do to solve it? ``` [2025-05-14 03:58:33 DP0 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 116, in forward_thread_func self.forward_thread_func_() File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 147, in forward_thread_func_ logits_output, next_token_ids = self.worker.forward_batch_generation( File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 181, in forward_batch_generation logits_output = self.model_runner.forward(forward_batch) File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1122, in forward return self._forward_raw(forward_batch, skip_attn_backend_init) File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1155, in _forward_raw return self.forward_extend( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1064, in forward_extend return self.model.forward( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 2099, in forward hidden_states = self.model(input_ids, positions, forward_batch, input_embeds) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1916, in forward hidden_states, residual = layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1462, in forward return self.forward_ffn_with_scattered_input( File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1617, in forward_ffn_with_scattered_input hidden_states = self.mlp(hidden_states, forward_batch.forward_mode) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 355, in forward return self.forward_deepep(hidden_states, forward_mode) File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 386, in forward_deepep self._forward_deepep_dispatch_a( File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 466, in _forward_deepep_dispatch_a chosen_deepep_dispatcher.dispatch_a( File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 713, in dispatch_a inner_state = self._get_impl(forward_mode).dispatch_a( File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 519, in dispatch_a hidden_states, masked_m, event, hook = self._dispatch_core( File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/ep_moe/token_dispatcher.py", line 603, in _dispatch_core buffer.low_latency_dispatch( File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 487, in low_latency_dispatch self.runtime.low_latency_dispatch(x, topk_idx, RuntimeError: Failed: Assertion error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:1025 'x.size(0) == topk_idx.size(0) and x.size(0) <= num_max_dispatch_tokens_per_rank' [2025-05-14 03:58:33] Child process unexpectedly failed with an exit code 131. pid=255 [2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=1401 [2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2194 [2025-05-14 03:58:34] Child process unexpectedly failed with an exit code 9. pid=2198 ``` cmd ``` export SGL_ENABLE_JIT_DEEPGEMM=1 python3 -m sglang.launch_server \ --model $model \ --trust-remote-code \ --tp 8 \ --dp 8 \ --enable-dp-attention \ --enable-deepep-moe \ --deepep-mode low_latency \ --enable-metrics ```
bsakaguchi commented 2026-10-26 09:55:12 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (May 14, 2025):

Maybe you should increase this value num_max_dispatch_tokens_per_rank

<!-- gh-comment-id:2879107230 --> @alpha-baby commented on GitHub (May 14, 2025): Maybe you should increase this value `num_max_dispatch_tokens_per_rank`
bsakaguchi commented 2026-10-26 09:55:12 +00:00
Author
Owner
Copy link

@ch-tiger1 commented on GitHub (May 14, 2025):

Maybe you should increase this value num_max_dispatch_tokens_per_rank

Thanks. Is there any standard I can refer to? What is the appropriate setting? I see that the default value of this variable is 128, and the maximum value is no more than 256

<!-- gh-comment-id:2879415047 --> @ch-tiger1 commented on GitHub (May 14, 2025): > Maybe you should increase this value `num_max_dispatch_tokens_per_rank` Thanks. Is there any standard I can refer to? What is the appropriate setting? I see that the default value of this variable is 128, and the maximum value is no more than 256
bsakaguchi commented 2026-10-26 09:55:12 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (May 14, 2025):

num_max_dispatch_tokens_per_rank means the maximum tokens to send in a single batch (must be consistent across all ranks).

So the "appropriate setting" would be the batch size per dispatch of SGLang, I think you should file a bug to the SGLang team :)

<!-- gh-comment-id:2879447503 --> @LyricZhao commented on GitHub (May 14, 2025): `num_max_dispatch_tokens_per_rank ` means the maximum tokens to send in a single batch (must be consistent across all ranks). So the "appropriate setting" would be the batch size per dispatch of SGLang, I think you should file a bug to the SGLang team :)
bsakaguchi commented 2026-10-26 09:55:12 +00:00
Author
Owner
Copy link

@ch-tiger1 commented on GitHub (May 14, 2025):

num_max_dispatch_tokens_per_rank means the maximum tokens to send in a single batch (must be consistent across all ranks).

So the "appropriate setting" would be the batch size per dispatch of SGLang, I think you should file a bug to the SGLang team :)

Yes, I don't know the specific reason at the moment, so I also added my question under this issue

<!-- gh-comment-id:2879514141 --> @ch-tiger1 commented on GitHub (May 14, 2025): > `num_max_dispatch_tokens_per_rank ` means the maximum tokens to send in a single batch (must be consistent across all ranks). > > So the "appropriate setting" would be the batch size per dispatch of SGLang, I think you should file a bug to the SGLang team :) Yes, I don't know the specific reason at the moment, so I also added my question under [this issue](https://github.com/sgl-project/sglang/issues/6017#issuecomment-2878599034)
bsakaguchi commented 2026-10-26 09:55:12 +00:00
Author
Owner
Copy link

@zheng1 commented on GitHub (Oct 22, 2025):

SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256

allowed me to successfully skip the issue. Everything works fine now.

<!-- gh-comment-id:3432995879 --> @zheng1 commented on GitHub (Oct 22, 2025): ``` SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 ``` allowed me to successfully skip the issue. Everything works fine now.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#144
No description provided.