[GH-ISSUE #593] HIddensize alignment can be different in e8m0/non e8m0 mode? #365

New issue

Closed

opened 2026-10-26 09:59:12 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:59:12 +00:00

Owner

Originally created by @MARD1NO on GitHub (Mar 18, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/593

There is some alignment check in low latency dispatch

if (use_fp8) {
        // TODO: support unaligned cases
        EP_HOST_ASSERT(hidden % 512 == 0);
        if (not use_ue8m0) {
            packed_recv_x_scales = torch::empty({num_local_experts, hidden / 128, num_ranks * num_max_dispatch_tokens_per_rank},
                                                torch::dtype(torch::kFloat32).device(torch::kCUDA));
        } else {
            EP_HOST_ASSERT(round_scale);
            packed_recv_x_scales = torch::empty({num_local_experts, hidden / 512, num_ranks * num_max_dispatch_tokens_per_rank},
                                                torch::dtype(torch::kInt).device(torch::kCUDA));
        }
        packed_recv_x_scales = torch::transpose(packed_recv_x_scales.value(), 1, 2);
        packed_recv_x_scales_ptr = packed_recv_x_scales->data_ptr();
    }

I think EP_HOST_ASSERT(hidden % 512 == 0); can only be checked in use_ue8m0 branch ?...

Originally created by @MARD1NO on GitHub (Mar 18, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/593 There is some alignment check in low latency dispatch ``` if (use_fp8) { // TODO: support unaligned cases EP_HOST_ASSERT(hidden % 512 == 0); if (not use_ue8m0) { packed_recv_x_scales = torch::empty({num_local_experts, hidden / 128, num_ranks * num_max_dispatch_tokens_per_rank}, torch::dtype(torch::kFloat32).device(torch::kCUDA)); } else { EP_HOST_ASSERT(round_scale); packed_recv_x_scales = torch::empty({num_local_experts, hidden / 512, num_ranks * num_max_dispatch_tokens_per_rank}, torch::dtype(torch::kInt).device(torch::kCUDA)); } packed_recv_x_scales = torch::transpose(packed_recv_x_scales.value(), 1, 2); packed_recv_x_scales_ptr = packed_recv_x_scales->data_ptr(); } ``` I think `EP_HOST_ASSERT(hidden % 512 == 0);` can only be checked in `use_ue8m0` branch ?...

bsakaguchi closed this issue

2026-10-26 09:59:12 +00:00

bsakaguchi commented

2026-10-26 09:59:13 +00:00

Author

Owner

@thepointer1982 commented on GitHub (Mar 20, 2026):

@thepointer1982 commented on GitHub (Mar 20, 2026): Hu

bsakaguchi referenced this issue

2026-10-27 21:03:46 +00:00

[PR #365] [MERGED] Init buffer with mpi4py.MPI.Comm #536