smxx_layout.hpp:136 'mn % 4 == 0 and num_groups == 1' #90

New issue

Closed

opened 2026-10-26 21:12:13 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:12:13 +00:00

Owner

Originally created by @yewentao256 on GitHub (Aug 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/153

Follow up with previous issue https://github.com/deepseek-ai/DeepGEMM/issues/140

(I don't have the permission to reopen, so creating a new one)

Tested with ~/vllm/tests/kernels/moe$ pytest -x test_batched_deepgemm.py , and the issue is still there

../../../vllm/model_executor/layers/fused_moe/modular_kernel.py:492: in _do_fused_experts
    self.fused_experts.apply(
../../../vllm/model_executor/layers/fused_moe/batched_deep_gemm_moe.py:295: in apply
    fp8_m_grouped_gemm_nt_masked((a2q, a2q_scale), (w2, w2_scale), output,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = ((tensor([[[ 2.4000e+01,  1.2000e+02,  7.5000e-01,  ..., -1.1000e+01,
          -3.5000e+00, -8.0000e+01],
         [ ...8, 517, 488, 509, 507, 503, 503, 537, 556, 526, 507, 505,
        493, 526], device='cuda:0', dtype=torch.int32), 1024)
kwargs = {}

    def fp8_m_grouped_gemm_nt_masked(*args, **kwargs):
        _lazy_init()
        if _grouped_masked_impl is None:
            return _missing(*args, **kwargs)
>       return _grouped_masked_impl(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       RuntimeError: Failed: Assertion error csrc/jit_kernels/impls/smxx_layout.hpp:122 'mn % 4 == 0 and num_groups == 1'

But if we use

cc416ee4faf0533a9263c2de814e5565f56ca1cc (HEAD)
Author: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
Date:   Thu Jun 12 16:10:00 2025 +0800

    Update layout.py

Now it works.

(vllm-user-6) vllm-user-6@centralia:~/vllm/tests/kernels/moe$ pytest -x test_batched_deepgemm.py 
========================= test session starts ==========================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /data/vllm-community-homes/vllm-user-6/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 32 items                                                     

test_batched_deepgemm.py ................................        [100%]

========================= 32 passed in 51.74s =========================

Originally created by @yewentao256 on GitHub (Aug 4, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/153 Follow up with previous issue https://github.com/deepseek-ai/DeepGEMM/issues/140 (I don't have the permission to reopen, so creating a new one) Tested with `~/vllm/tests/kernels/moe$ pytest -x test_batched_deepgemm.py `, and the issue is still there ```bash ../../../vllm/model_executor/layers/fused_moe/modular_kernel.py:492: in _do_fused_experts self.fused_experts.apply( ../../../vllm/model_executor/layers/fused_moe/batched_deep_gemm_moe.py:295: in apply fp8_m_grouped_gemm_nt_masked((a2q, a2q_scale), (w2, w2_scale), output, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ((tensor([[[ 2.4000e+01, 1.2000e+02, 7.5000e-01, ..., -1.1000e+01, -3.5000e+00, -8.0000e+01], [ ...8, 517, 488, 509, 507, 503, 503, 537, 556, 526, 507, 505, 493, 526], device='cuda:0', dtype=torch.int32), 1024) kwargs = {} def fp8_m_grouped_gemm_nt_masked(*args, **kwargs): _lazy_init() if _grouped_masked_impl is None: return _missing(*args, **kwargs) > return _grouped_masked_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E RuntimeError: Failed: Assertion error csrc/jit_kernels/impls/smxx_layout.hpp:122 'mn % 4 == 0 and num_groups == 1' ``` But if we use ```bash cc416ee4faf0533a9263c2de814e5565f56ca1cc (HEAD) Author: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com> Date: Thu Jun 12 16:10:00 2025 +0800 Update layout.py ``` Now it works. ```bash (vllm-user-6) vllm-user-6@centralia:~/vllm/tests/kernels/moe$ pytest -x test_batched_deepgemm.py ========================= test session starts ========================== platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 rootdir: /data/vllm-community-homes/vllm-user-6/vllm configfile: pyproject.toml plugins: anyio-4.9.0 collected 32 items test_batched_deepgemm.py ................................ [100%] ========================= 32 passed in 51.74s ========================= ```

bsakaguchi closed this issue

2026-10-26 21:12:13 +00:00

bsakaguchi commented

2026-10-26 21:12:15 +00:00

Author

Owner

@LJC00118 commented on GitHub (Aug 5, 2025):

After #154, I believe it is fixed. Thanks!

@LJC00118 commented on GitHub (Aug 5, 2025): After #154, I believe it is fixed. Thanks!

bsakaguchi commented

2026-10-26 21:12:15 +00:00

Author

Owner

@yewentao256 commented on GitHub (Aug 5, 2025):

(vllm-user-6) vllm-user-6@centralia:~/vllm/tests/kernels/moe$ pytest test_batched_deepgemm.py 
========================================== test session starts ==========================================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /data/vllm-community-homes/vllm-user-6/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 32 items                                                                                      

test_batched_deepgemm.py ................................                                         [100%]

========================================== 32 passed in 35.55s ==========================================

Tested and fixed, thanks!

@yewentao256 commented on GitHub (Aug 5, 2025): ```bash (vllm-user-6) vllm-user-6@centralia:~/vllm/tests/kernels/moe$ pytest test_batched_deepgemm.py ========================================== test session starts ========================================== platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 rootdir: /data/vllm-community-homes/vllm-user-6/vllm configfile: pyproject.toml plugins: anyio-4.9.0 collected 32 items test_batched_deepgemm.py ................................ [100%] ========================================== 32 passed in 35.55s ========================================== ``` Tested and fixed, thanks!

bsakaguchi referenced this issue

2026-10-26 21:15:13 +00:00

[PR #90] [MERGED] Indivisible TMA #242