mirror of
https://github.com/deepseek-ai/DeepGEMM.git
synced 2026-11-05 00:47:26 +00:00
[GH-ISSUE #98] m_grouped_gemm_fp8_fp8_bf16_nt_contiguous's requiresment of 128 padding for each expert causes computation waste #55
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepGEMM#55
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xutizhou on GitHub (May 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/98
I noticed that the number of input tokens must be aligned to BLOCK_M (default 128) code, which can lead to up to 20% wasted computation when the workload is light. Would it be possible to relax this alignment requirement to 64?
@LyricZhao commented on GitHub (May 14, 2025):
We are working on a M-computation-skipping optimization for such cases, which does not require changing the alignment but can achieve similar effect. @yukuai26
@xutizhou commented on GitHub (May 14, 2025):
I am looking forward to it. Is there any expected timeline?
@LyricZhao commented on GitHub (May 14, 2025):
I guess we will finish it in the next week (before 5.24).
@yukuai26 commented on GitHub (May 20, 2025):
@xutizhou You're suggesting that when each expert receives a very small number of tokens, the BLOCK_M = 128 setting leads to computational waste. In such cases, where M (the token count per expert) is small and the workload is memory-bound, computational waste may not significantly impact performance. Could you clarify what scenarios you expect to see performance improvements in this optimization?
@xutizhou commented on GitHub (May 20, 2025):
Hi @yukuai26 , when I deploy deepseek in one h20 node, and total input tokens are 8192 for a better ttft, then the 128 padding will cause almost to 20% waste
@yukuai26 commented on GitHub (May 21, 2025):
Hi @xutizhou If M = 8192, assuming 20% computational waste, does that mean you used 50 groups (where each group has a 50% probability of wasting computation for 64 tokens)? The expected tokens per group would then be 160. Could you clarify the size of your weights? If they are not extremely small, this scenario should generally be memory-bound rather than compute-bound.Therefore, the computational waste should not significantly impact performance to this extent.
Additionally, how exactly was the 20% waste calculated?
@xutizhou commented on GitHub (May 21, 2025):
Hi @yukuai26, sorry for not clarifying. M is the number of total tokens of 256 groups in one node.
@yukuai26 commented on GitHub (May 22, 2025):
Hi @xutizhou Your M is 8192. Could you specify how many experts are selected and what the exact N/K values are for each expert? (Since if N and K are not particularly small, the kernel is actually memory-bound rather than compute-bound.)
We skipped some m-related computations in this MR -> https://github.com/deepseek-ai/DeepGEMM/pull/103. We would like to merge this pull request only after observing significant improvements for your specific shape. Could you test the optimization effects of this MR on your shape?
@xutizhou commented on GitHub (May 26, 2025):
Hi @yukuai26, I selected 8 experts in my setup. I tested your MR using my configuration with num_groups = 32, expected_m_per_group = 256, k = 7168, and n = 4096. I observed a performance drop of over 20% compared to the original grouped GEMM (without skipping). Even factoring in the reduced redundant calculations, the overall performance still decreased.
@LyricZhao commented on GitHub (May 27, 2025):
Hi, @xutizhou. #103 is already merged, and we've added your test case in
test_core.py.https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/include/deep_gemm/fp8_gemm.cuh#L353
On H800 (CUDA 12.9), the PR brings about 1-3% boost:
1-3% is reasonable for H800, as such shapes are memory-bound (reducing computation will not help with performance).
We don't have H20 on hands, and this PR also contains some refactor on tests, so we recommend you:
not scheduler.is_computation_valid(m_block_idx, math_wg_idx * WGMMA::M)intofalsein line https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/include/deep_gemm/fp8_gemm.cuh#L353.So that you can compare the performance speedup with skipping on H20 with your shapes.
As communicating in issues is somehow inefficient, you can contact me via WeChat (ID: LyricZ_THU) if you have further questions, thanks!
@xutizhou commented on GitHub (May 27, 2025):
Thank you for your thorough experiment. I have also tested auto skipping and no-skipping in H20 and observed a 1% to 3% performance improvement when enabling auto skipping. However, since H2O is compute-bound, is this performance gain in line with expectations?