[PR #62] [CLOSED] [Performance] Minor speed up 2% on some shapes #223

New issue

Closed

opened 2026-10-26 21:15:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:15:03 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/62
Author: @fzyzcjy
Created: 3/13/2025
Status: ❌ Closed

Base: main ← Head: patch-1

📝 Commits (1)

cf64055 Update fp8_gemm.cuh

📊 Changes

1 file changed (+113 additions, -30 deletions)

View changed files

📝 deep_gemm/include/deep_gemm/fp8_gemm.cuh (+113 -30)

📄 Description

Hi, thanks for the fast kernel! When checking why 4096,32768,512 shape is somehow slow, there seems to be a very tiny optimization that we may overlap store-to-smem with gemm. Thus I made a hacky prototype (as is shown in the PR). Results show that it improves ~2% on a few shapes, thus we may conditionally compile to enable them there.

I wonder whether the direction of PR looks acceptable? If so I will polish the code (cleanup messy code, extract functions, conditionally enable, etc).

Experiments: Since this is a prototype, I only did some very naive experiments. In order to make numbers more exact, each run in test_core.py is repeated by 200 times. Then, the mean and std are computed. In addition, the bandwidth delta divided by std is shown in the last column as a very vanilla statistics information.

Results (two green rows are improvements - 1.72% and 2.35% each; two small green boxes are potentially improvements or just noises or too small to be interesting):

P.S. I am playing with other very naive optimizations as well, e.g. this (will need do to optimizations later before knowing whether it speeds up or not).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/62 **Author:** [@fzyzcjy](https://github.com/fzyzcjy) **Created:** 3/13/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `patch-1` --- ### 📝 Commits (1) - [`cf64055`](https://github.com/deepseek-ai/DeepGEMM/commit/cf640558afbf6c0d506f652388b923807350b3de) Update fp8_gemm.cuh ### 📊 Changes **1 file changed** (+113 additions, -30 deletions) <details> <summary>View changed files</summary> 📝 `deep_gemm/include/deep_gemm/fp8_gemm.cuh` (+113 -30) </details> ### 📄 Description Hi, thanks for the fast kernel! When checking why 4096,32768,512 shape is somehow slow, there seems to be a very tiny optimization that we may overlap store-to-smem with gemm. Thus I made a hacky prototype (as is shown in the PR). Results show that it improves ~2% on a few shapes, thus we may conditionally compile to enable them there. I wonder whether the direction of PR looks acceptable? If so I will polish the code (cleanup messy code, extract functions, conditionally enable, etc). Experiments: Since this is a prototype, I only did some very naive experiments. In order to make numbers more exact, each run in test_core.py is repeated by 200 times. Then, the mean and std are computed. In addition, the bandwidth delta divided by std is shown in the last column as a very vanilla statistics information. Results (two green rows are improvements - 1.72% and 2.35% each; two small green boxes are potentially improvements or just noises or too small to be interesting): ![Pasted image 20250313210040](https://github.com/user-attachments/assets/f991b7d8-bec1-4a6d-9023-c5f16c7634ea) P.S. I am playing with other very naive optimizations as well, e.g. [this](https://gist.github.com/fzyzcjy/fc0c0014a24e3c501209c38861745cf4) (will need do to optimizations later before knowing whether it speeds up or not). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:15:03 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepGEMM#223

No description provided.

Rows
Columns