[GH-ISSUE #290] [QUESTION]Why does fp8_mqa_logits use four MATH WGs for computation? #146

New issue

Closed

opened 2026-10-26 21:13:40 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:13:40 +00:00

Owner

Originally created by @pengwubj on GitHub (Mar 3, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/290

Why does fp8_mqa_logits use four MATH WGs for computation? In typical Hopper implementations, only two MATH WGs are generally used. This is also the case in the DeepGEMM 1d1d and 1d2d implementations.

Is this design choice related to the KV block dimension being 256? If we change the KV block size to 128, would it impact performance?

Originally created by @pengwubj on GitHub (Mar 3, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/290 Why does fp8_mqa_logits use four MATH WGs for computation? In typical Hopper implementations, only two MATH WGs are generally used. This is also the case in the DeepGEMM 1d1d and 1d2d implementations. Is this design choice related to the KV block dimension being 256? If we change the KV block size to 128, would it impact performance?

bsakaguchi closed this issue

2026-10-26 21:13:40 +00:00

bsakaguchi commented

2026-10-26 21:13:41 +00:00

Author

Owner

@ajwise9 commented on GitHub (Mar 22, 2026):

In simple terms just because FP8 WGMMA on H100 delivers ~2× the throughput of BF16.

source

Also, with 2 MATH WGs (the standard ping-pong schedule), the tensor cores in each WG are underutilized because the per-WG tile doesn't generate enough independent WGMMA instructions to fully hide the ~16-cycle WGMMA latency. Going to 4 WGs doubles the in-flight WGMMA wavefronts, keeping the tensor core pipeline saturated.

Is it tied to block_N=256? What if you drop to 128?
Yes, directly. 256 gives 8× k32 WGMMA steps per KV iteration, so enough to keep 4 WGs fed. At 128 you'd halve that, starve the WGs, and want to drop back to 2. Expect ~10–20% give or take regression.

@ajwise9 commented on GitHub (Mar 22, 2026): In simple terms just because FP8 WGMMA on H100 delivers ~2× the throughput of BF16. <img width="1075" height="287" alt="Image" src="https://github.com/user-attachments/assets/2e59de40-6c12-480d-9fc6-5f809bc4bcc6" /> [source](https://gemslabs.io/reports ) Also, with 2 MATH WGs (the standard ping-pong schedule), the tensor cores in each WG are underutilized because the per-WG tile doesn't generate enough independent WGMMA instructions to fully hide the ~16-cycle WGMMA latency. Going to 4 WGs doubles the in-flight WGMMA wavefronts, keeping the tensor core pipeline saturated. **Is it tied to block_N=256? What if you drop to 128?** Yes, directly. 256 gives 8× k32 WGMMA steps per KV iteration, so enough to keep 4 WGs fed. At 128 you'd halve that, starve the WGs, and want to drop back to 2. Expect ~10–20% give or take regression.

bsakaguchi commented

2026-10-26 21:13:42 +00:00

Author

Owner

@pengwubj commented on GitHub (May 22, 2026):

Thanks a lot !

@pengwubj commented on GitHub (May 22, 2026): Thanks a lot !

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepGEMM#146

No description provided.

Rows
Columns