mirror of
https://github.com/deepseek-ai/DeepGEMM.git
synced 2026-11-05 00:47:26 +00:00
[GH-ISSUE #35] performance on H800 does not meet expectations for small batchsize #23
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepGEMM#23
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Planck-a on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/35
I tested gemm on H800 SXM,k=7168,n=2112. I was confused that when bsz increased from 1 to 64, the gemm time was decreasing as m increased. This phenomenon also occurred in 64-128, which seemed to be related to the number of SMs. It seems that there is only optimized for bsz=64/128, is there any suggestion how to optimize other bsz like 1-32 ,thx
@soundOfDestiny commented on GitHub (Mar 4, 2025):
This is because TMA load is faster for m=132 than for m=68 due to boundary check.
You can pad your input tensor (LHS and LHS scale) to get better performance.
For a quick test, you could change shape_m in make_2d_tma_a_desc in deep_gemm/include/deep_gemm/fp8_gemm.cuh to a padded size, and you would observe that it becomes faster for m=68.
However, we won't do that due to the risk of illegal read.
@Planck-a commented on GitHub (Mar 4, 2025):
@soundOfDestiny thx,but my question is why m=64 is faster than m=4 ? Is this reasonable?
@soundOfDestiny commented on GitHub (Mar 4, 2025):
It's the same reason and you can do the quick test mentioned above:
You could replace
shape_minmake_2d_tma_a_descindeep_gemm/include/deep_gemm/fp8_gemm.cuhwith 64, and you would observe that it becomes faster for m=4.