deepseek-ai/DeepGEMM

Fork 0

mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00

30 #467

New issue

Closed

opened 2026-10-26 21:18:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:18:12 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/462
Author: @LyricZhao
Created: 9/30/2026
Status: ✅ Merged
Merged: 9/30/2026
Merged by: @LyricZhao

Base: main ← Head: 0930-release

📝 Commits (1)

c7849af Public release 26/09/30

📊 Changes

86 files changed (+4670 additions, -4023 deletions)

View changed files

📝 README.md (+17 -9)
📝 csrc/apis/attention.hpp (+228 -377)
📝 csrc/apis/config.hpp (+3 -0)
📝 csrc/apis/einsum.hpp (+68 -32)
➕ csrc/apis/epilogue_class.hpp (+27 -0)
📝 csrc/apis/gemm.hpp (+110 -56)
➕ csrc/apis/locality_domain.hpp (+162 -0)
📝 csrc/apis/mega_gate.hpp (+58 -96)
📝 csrc/apis/mega_moe.hpp (+27 -30)
📝 csrc/indexing/main.cu (+3 -0)
📝 csrc/jit_kernels/heuristics/mega_gate.hpp (+130 -177)
📝 csrc/jit_kernels/heuristics/mega_moe.hpp (+25 -12)
➖ csrc/jit_kernels/impls/epilogue.hpp (+0 -45)
➕ csrc/jit_kernels/impls/epilogue_class.hpp (+129 -0)
📝 csrc/jit_kernels/impls/runtime_utils.hpp (+128 -2)
📝 csrc/jit_kernels/impls/sm100_bf16_gemm.hpp (+31 -19)
📝 csrc/jit_kernels/impls/sm100_bf16_mega_gate.hpp (+32 -63)
📝 csrc/jit_kernels/impls/sm100_bf16_mega_moe.hpp (+15 -27)
📝 csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp (+32 -27)
📝 csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp (+22 -36)

...and 66 more files

📄 Description

This update introduces new features, performance improvements, and important API changes:

New features

Locality-aware Mega MoE: optional weight localization and domain-aware scheduling, with helpers for localized allocation (via MLOPart) and SM locality discovery. Saving 100W power and boosting 5-10% GPU frequency for heavy workload.
New deep_gemm.epilogue classes: Identity, Alpha, FP8Quantization, and opt-in BF16StochasticRounding for direct BF16 output on SM100.
BF16 batched einsums now support accumulation and FP32 output for bhr,hdr->bhd and bhd,hdr->bhr.
Sparse Indexer now supports head counts divisible by 4, from 4 through 32, for both contiguous and paged MXFP4/MXFP8 inputs.
Support for padded K strides in packed UE8M0 scaling factors.

Performance improvements

Improved Lightning Indexer kernels for V4.1, with better scheduling, KV pipelines, and head reduction for prefill and paged decoding.
Mega MoE epilogue prefetch and shared memory optimizations enable deeper pipelines for FP8xFP4 prefill.

Fixes

Fixed Sparse Indexer memory bounds and strengthened paged-cache and metadata validation.
Fixed Mega MoE combine reduction order for bitwise agreement with the shared-expert reference, and improved locality probing and fallback behavior.
Fixed runtime shutdown ordering for cuBLASLt resources and a missing FP8 header in Mega mHC.
Removed the Mega mHC test baseline's dependency on external tile-kernels code.

Breaking changes

The non-paged fp8_fp4_mqa_logits API now requires a positive max_seqlen_k and returns compressed logits. The dense non-paged and paged MQA APIs no longer accept clean_logits or logits_dtype; callers must mask entries beyond each row's valid KV span.
Dense MQA on SM100 now requires MXFP8/MXFP4 inputs with packed UE8M0 scales and BF16 weights, and returns BF16 logits. SM90 retains non-MX FP8 inputs with FP32 weights and logits. Weight rows must be 16-byte aligned.
Paged MQA on SM100 now requires the variable-length interface: pass indices and use next_n=1.
Removed the legacy fp8_mqa_logits, fp8_paged_mqa_logits, and fp8_gemm_nt_skip_head_mid APIs. Use the fp8_fp4_*mqa_logits APIs for MQA workloads.
Mega Gate now supports only scoring_func="sqrtsoftplus", which is also the default. get_bf16_mega_gate_config has been removed.

DeepGEMM Ascend is now available in a separate repository.

Contributors

Locality-aware Mega MoE and SM probing: @Yongqi-Zhuo @LyricZhao
Lightning Indexer, Sparse Indexer, and Mega MoE correctness: @zheanxu
GEMM epilogue classes, stochastic rounding, and BF16 einsum extensions: @xyf007
Mega MoE epilogue prefetch and shared-memory optimizations: @THUTwenTy
Packed scaling-factor strides: @xay5421

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/462 **Author:** [@LyricZhao](https://github.com/LyricZhao) **Created:** 9/30/2026 **Status:** ✅ Merged **Merged:** 9/30/2026 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `0930-release` --- ### 📝 Commits (1) - [`c7849af`](https://github.com/deepseek-ai/DeepGEMM/commit/c7849af3eeb532b37c44983cf8951d6ddad9c457) Public release 26/09/30 ### 📊 Changes **86 files changed** (+4670 additions, -4023 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+17 -9) 📝 `csrc/apis/attention.hpp` (+228 -377) 📝 `csrc/apis/config.hpp` (+3 -0) 📝 `csrc/apis/einsum.hpp` (+68 -32) ➕ `csrc/apis/epilogue_class.hpp` (+27 -0) 📝 `csrc/apis/gemm.hpp` (+110 -56) ➕ `csrc/apis/locality_domain.hpp` (+162 -0) 📝 `csrc/apis/mega_gate.hpp` (+58 -96) 📝 `csrc/apis/mega_moe.hpp` (+27 -30) 📝 `csrc/indexing/main.cu` (+3 -0) 📝 `csrc/jit_kernels/heuristics/mega_gate.hpp` (+130 -177) 📝 `csrc/jit_kernels/heuristics/mega_moe.hpp` (+25 -12) ➖ `csrc/jit_kernels/impls/epilogue.hpp` (+0 -45) ➕ `csrc/jit_kernels/impls/epilogue_class.hpp` (+129 -0) 📝 `csrc/jit_kernels/impls/runtime_utils.hpp` (+128 -2) 📝 `csrc/jit_kernels/impls/sm100_bf16_gemm.hpp` (+31 -19) 📝 `csrc/jit_kernels/impls/sm100_bf16_mega_gate.hpp` (+32 -63) 📝 `csrc/jit_kernels/impls/sm100_bf16_mega_moe.hpp` (+15 -27) 📝 `csrc/jit_kernels/impls/sm100_fp8_fp4_gemm_1d1d.hpp` (+32 -27) 📝 `csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp` (+22 -36) _...and 66 more files_ </details> ### 📄 Description This update introduces new features, performance improvements, and important API changes: ### New features - Locality-aware Mega MoE: optional weight localization and domain-aware scheduling, with helpers for localized allocation (via MLOPart) and SM locality discovery. Saving 100W power and boosting 5-10% GPU frequency for heavy workload. - New `deep_gemm.epilogue` classes: `Identity`, `Alpha`, `FP8Quantization`, and opt-in `BF16StochasticRounding` for direct BF16 output on SM100. - BF16 batched einsums now support accumulation and FP32 output for `bhr,hdr->bhd` and `bhd,hdr->bhr`. - Sparse Indexer now supports head counts divisible by 4, from 4 through 32, for both contiguous and paged MXFP4/MXFP8 inputs. - Support for padded K strides in packed UE8M0 scaling factors. ### Performance improvements - Improved Lightning Indexer kernels for V4.1, with better scheduling, KV pipelines, and head reduction for prefill and paged decoding. - Mega MoE epilogue prefetch and shared memory optimizations enable deeper pipelines for FP8xFP4 prefill. ### Fixes - Fixed Sparse Indexer memory bounds and strengthened paged-cache and metadata validation. - Fixed Mega MoE combine reduction order for bitwise agreement with the shared-expert reference, and improved locality probing and fallback behavior. - Fixed runtime shutdown ordering for cuBLASLt resources and a missing FP8 header in Mega mHC. - Removed the Mega mHC test baseline's dependency on external tile-kernels code. ### Breaking changes - The non-paged `fp8_fp4_mqa_logits` API now requires a positive `max_seqlen_k` and returns compressed logits. The dense non-paged and paged MQA APIs no longer accept `clean_logits` or `logits_dtype`; callers must mask entries beyond each row's valid KV span. - Dense MQA on SM100 now requires MXFP8/MXFP4 inputs with packed UE8M0 scales and BF16 weights, and returns BF16 logits. SM90 retains non-MX FP8 inputs with FP32 weights and logits. Weight rows must be 16-byte aligned. - Paged MQA on SM100 now requires the variable-length interface: pass `indices` and use `next_n=1`. - Removed the legacy `fp8_mqa_logits`, `fp8_paged_mqa_logits`, and `fp8_gemm_nt_skip_head_mid` APIs. Use the `fp8_fp4_*mqa_logits` APIs for MQA workloads. - Mega Gate now supports only `scoring_func="sqrtsoftplus"`, which is also the default. `get_bf16_mega_gate_config` has been removed. ### Related release - [DeepGEMM Ascend](https://github.com/deepseek-ai/DeepGEMM-Ascend/) is now available in a separate repository. ### Contributors - Locality-aware Mega MoE and SM probing: @Yongqi-Zhuo @LyricZhao - Lightning Indexer, Sparse Indexer, and Mega MoE correctness: @zheanxu - GEMM epilogue classes, stochastic rounding, and BF16 einsum extensions: @xyf007 - Mega MoE epilogue prefetch and shared-memory optimizations: @THUTwenTy - Packed scaling-factor strides: @xay5421 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:18:12 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepGEMM#467

No description provided.

Rows
Columns