[GH-ISSUE #604] Hybrid-EP has hang and precision issues in SFT training with variable sequence lengths #369

New issue

Open

opened 2026-10-26 09:59:15 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:59:15 +00:00

Owner

Originally created by @fengchen-98 on GitHub (Apr 17, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/604

hello DeepEp and HybridEp team
I found some bugs about HybridEp

During SFT training, each rank may have a different max_num_of_tokens_per_rank in HybridEPBuffer.__init__(). In update_template_config calls, the num_of_tokens_per_rank in the hidden state is also different. This causes some ranks to wait at an all_gather while others skip it, resulting in a hang.

the hang place is bool HybridEPBuffer::update_buffer. some ranks need_reallocate=false so skip the all_gather in allgather_obj.update(buffer_config);

In dispatch_with_permute, metadata_preprocessing receives a routing_map, but the length of routing_map is different. This leads to different input lengths for the all_gather_into_tensor operation in Executor::allgather_routing_map, causing precision issues.

Fix:

I try to use max_num_of_tokens_per_rank=next_power_of_two(max_num_of_tokens_per_rank) so that the length is always max_length.
Fix precision issue by padding routing_map to max_num_of_tokens_per_rank.

pad_rows = config.max_num_of_tokens_per_rank - num_of_tokens_per_rank
routing_map = torch.nn.functional.pad(routing_map, (0, 0, 0, pad_rows), value=False)

Training normally after these two fixes

Originally created by @fengchen-98 on GitHub (Apr 17, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/604 hello DeepEp and HybridEp team I found some bugs about HybridEp 1. During SFT training, each rank may have a different `max_num_of_tokens_per_rank ` in `HybridEPBuffer.__init__().` In `update_template_config` calls, the `num_of_tokens_per_rank` in the hidden state is also different. This causes some ranks to wait at an all_gather while others skip it, resulting in a hang. the hang place is `bool HybridEPBuffer::update_buffer`. some ranks `need_reallocate=false` so skip the all_gather in `allgather_obj.update(buffer_config);` 2. In `dispatch_with_permute`, `metadata_preprocessing` receives a `routing_map`, but the length of `routing_map` is different. This leads to different input lengths for the `all_gather_into_tensor` operation in `Executor::allgather_routing_map`, causing precision issues. Fix: 1. I try to use `max_num_of_tokens_per_rank=next_power_of_two(max_num_of_tokens_per_rank)` so that the length is always max_length. 2. Fix precision issue by padding `routing_map` to `max_num_of_tokens_per_rank`. `pad_rows = config.max_num_of_tokens_per_rank - num_of_tokens_per_rank` `routing_map = torch.nn.functional.pad(routing_map, (0, 0, 0, pad_rows), value=False)` Training normally after these two fixes

bsakaguchi commented

2026-10-26 09:59:16 +00:00

Author

Owner

@Autumn1998 commented on GitHub (May 9, 2026):

Hi, thank you for the information. For this situation where the number of tokens from attn by each rank is inconsistent, there is currently no official support. The only workaround is through an approach similar to padding. Sorry about that!

@Autumn1998 commented on GitHub (May 9, 2026): Hi, thank you for the information. For this situation where the number of tokens from attn by each rank is inconsistent, there is currently no official support. The only workaround is through an approach similar to padding. Sorry about that!

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#369

No description provided.

Rows
Columns