[PR #326] [WIP] support sm90_fp4_mqa_logits #365

New issue

Open

opened 2026-10-26 21:16:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:16:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/326
Author: @laixinn
Created: 5/1/2026
Status: 🔄 Open

Base: main ← Head: sm90_fp4_mqa_logits

📝 Commits (4)

4650dfb sm90_fp4_mqa_logits v1
779fcd0 opt v1: 20% flops improved
81cc13d opt v2: 75% of the fp8, via ncu profiling
c981a8d rm install.sh change

📊 Changes

6 files changed (+739 additions, -14 deletions)

View changed files

📝 csrc/apis/attention.hpp (+3 -0)
📝 csrc/jit_kernels/impls/runtime_utils.hpp (+1 -0)
📝 csrc/jit_kernels/impls/smxx_fp8_fp4_mqa_logits.hpp (+183 -0)
➕ deep_gemm/include/deep_gemm/impls/sm90_fp4_mqa_logits.cuh (+551 -0)
➖ install.sh (+0 -13)
📝 tests/test_attention.py (+1 -1)

📄 Description

This PR dequants w4a4 to w8a8 to support the FP4 Indexer in SM90, supposing to be a part of sgl-project/sglang#23602.
Currently, this operation brings 40% latency drawbacks.
cc @AniZpZ

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/326 **Author:** [@laixinn](https://github.com/laixinn) **Created:** 5/1/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `sm90_fp4_mqa_logits` --- ### 📝 Commits (4) - [`4650dfb`](https://github.com/deepseek-ai/DeepGEMM/commit/4650dfb7109e8439a8b5e1ae30be02b0117d908e) sm90_fp4_mqa_logits v1 - [`779fcd0`](https://github.com/deepseek-ai/DeepGEMM/commit/779fcd08bc1c75d40d6da14e0642d6ad61ec3926) opt v1: 20% flops improved - [`81cc13d`](https://github.com/deepseek-ai/DeepGEMM/commit/81cc13d17d13d48326b6dd403afafe4f92b4c018) opt v2: 75% of the fp8, via ncu profiling - [`c981a8d`](https://github.com/deepseek-ai/DeepGEMM/commit/c981a8d74f4eab6dedb0015d564e2570d7919852) rm install.sh change ### 📊 Changes **6 files changed** (+739 additions, -14 deletions) <details> <summary>View changed files</summary> 📝 `csrc/apis/attention.hpp` (+3 -0) 📝 `csrc/jit_kernels/impls/runtime_utils.hpp` (+1 -0) 📝 `csrc/jit_kernels/impls/smxx_fp8_fp4_mqa_logits.hpp` (+183 -0) ➕ `deep_gemm/include/deep_gemm/impls/sm90_fp4_mqa_logits.cuh` (+551 -0) ➖ `install.sh` (+0 -13) 📝 `tests/test_attention.py` (+1 -1) </details> ### 📄 Description This PR dequants w4a4 to w8a8 to support the FP4 Indexer in SM90, supposing to be a part of sgl-project/sglang#23602. Currently, this operation brings 40% latency drawbacks. cc @AniZpZ --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 21:16:51 +00:00

bsakaguchi referenced this issue

2026-10-26 21:17:13 +00:00

[PR #375] Fix non-deterministic results in k_grouped_fp8_gemm_nt_contiguous #397