[GH-ISSUE #338] [RFC] Support FP4 (E2M1) activation dispatch in MegaMoE #166

New issue

Open

opened 2026-10-26 21:14:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:14:04 +00:00

Owner

Originally created by @wuudiz on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/338

Currently fp8_fp4_mega_moe accepts FP8 activations only. The
use_fp8_dispatch
parameter is plumbed through get_symm_buffer_for_mega_moe / SymmBuffer
but has
no effect -- buffer sizing and the kernel always assume FP8.
Switching to FP4 activations (E2M1) would halve the symmetric buffer input
footprint
and potentially enable the MXF4 mainloop (K=64 dense vs K=32 padded in
mxf8f6f4).
I ran an end-to-end proof-of-concept (w4a4 vs w4a8) using an external
mega_moe_pre_dispatch kernel for E2M1 packing + FP4-aware buffer allocation,
with the rest of the pipeline unchanged. GSM8K (1319 samples, greedy) results:
• DeepSeek-V4-Flash: w4a8 93.86% → w4a4 94.39% (delta +0.53%)
• DeepSeek-V4-Pro: w4a8 91.81% → w4a4 91.13% (delta −0.68%)
Both within noise -- no meaningful accuracy degradation from FP4 activation
packing.
Would it be feasible to wire use_fp8_dispatch=False through to the buffer
layout
and kernel?
Happy to help test or contribute if there's a preferred direction.

Originally created by @wuudiz on GitHub (May 19, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/338 Currently fp8_fp4_mega_moe accepts FP8 activations only. The use_fp8_dispatch parameter is plumbed through get_symm_buffer_for_mega_moe / SymmBuffer but has no effect -- buffer sizing and the kernel always assume FP8. Switching to FP4 activations (E2M1) would halve the symmetric buffer input footprint and potentially enable the MXF4 mainloop (K=64 dense vs K=32 padded in mxf8f6f4). I ran an end-to-end proof-of-concept (w4a4 vs w4a8) using an external mega_moe_pre_dispatch kernel for E2M1 packing + FP4-aware buffer allocation, with the rest of the pipeline unchanged. GSM8K (1319 samples, greedy) results: • DeepSeek-V4-Flash: w4a8 93.86% → w4a4 94.39% (delta +0.53%) • DeepSeek-V4-Pro: w4a8 91.81% → w4a4 91.13% (delta −0.68%) Both within noise -- no meaningful accuracy degradation from FP4 activation packing. Would it be feasible to wire use_fp8_dispatch=False through to the buffer layout and kernel? Happy to help test or contribute if there's a preferred direction.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepGEMM#166

No description provided.

Rows
Columns