fixes #357

New issue

Closed

opened 2026-10-26 21:16:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:16:44 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/304
Author: @LyricZhao
Created: 4/16/2026
Status: ✅ Merged
Merged: 4/17/2026
Merged by: @LyricZhao

Base: main ← Head: public-release-260416

📝 Commits (7)

7dbe077 Merge with private repo
37ab81a Update README
d01623b Update README
0c72e11 Update README
2d6fd1c Add PyTorch requirements
9f4bed6 Fix sync scopes for MQA logits (#256)
211d267 Update README

📊 Changes

109 files changed (+12135 additions, -3253 deletions)

View changed files

📝 CMakeLists.txt (+4 -5)
📝 README.md (+76 -44)
📝 csrc/apis/attention.hpp (+279 -128)
📝 csrc/apis/einsum.hpp (+35 -34)
📝 csrc/apis/gemm.hpp (+68 -65)
📝 csrc/apis/hyperconnection.hpp (+7 -7)
📝 csrc/apis/layout.hpp (+46 -23)
➕ csrc/apis/mega.hpp (+216 -0)
📝 csrc/apis/runtime.hpp (+20 -0)
📝 csrc/indexing/main.cu (+6 -1)
📝 csrc/jit/cache.hpp (+1 -1)
📝 csrc/jit/compiler.hpp (+70 -61)
📝 csrc/jit/device_runtime.hpp (+46 -7)
📝 csrc/jit/handle.hpp (+72 -18)
➕ csrc/jit/include_parser.hpp (+80 -0)
📝 csrc/jit/kernel_runtime.hpp (+77 -30)
📝 csrc/jit_kernels/heuristics/common.hpp (+34 -319)
➕ csrc/jit_kernels/heuristics/config.hpp (+171 -0)
➕ csrc/jit_kernels/heuristics/mega_moe.hpp (+211 -0)
➕ csrc/jit_kernels/heuristics/runtime.hpp (+62 -0)

...and 80 more files

📄 Description

New features

Mega MoE, fusing & overlapping dispatch/linear 1/SwiGLU/linear 2/combine into a single mega-kernel, overlapping NVLink communication and tensor core computation
- Performance number: https://github.com/deepseek-ai/DeepGEMM/pull/316
- Only FP8 x FP4 MoE is supported
- Requires PyTorch >= 2.9
FP4 Indexer (MQA logits) with larger MTP support
FP8 x FP4 GEMM
PDL
Refactors on GEMM heuristics
Faster JIT compilation
GEMM optimizations (dynamic swap A/B, much faster MoE GEMM)
DeepEPv2 MoE GEMM layout

Bug fixes

JIT may crash on distributed FS
Some kernel hangs and IMA

Contributors

Mega MoE: @LyricZhao @zheanxu @bucket-xv @RayWang96 @interestingLSY @kurisu6912 @xay5421 @yukuai26
FP4 Indexer: @zheanxu @xay5421 @interestingLSY @kurisu6912
GEMM, PDL, JIT and bug fixes: @zheanxu @bucket-xv @xay5421 @yukuai26 @LyricZhao

Additional notes

Mega MoE is still under development and optimizations, stay tuned and optimization ideas are welcome!
Disclaimer: this release is only related to DeepGEMM's development, has nothing to do with internal model release.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/304 **Author:** [@LyricZhao](https://github.com/LyricZhao) **Created:** 4/16/2026 **Status:** ✅ Merged **Merged:** 4/17/2026 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `public-release-260416` --- ### 📝 Commits (7) - [`7dbe077`](https://github.com/deepseek-ai/DeepGEMM/commit/7dbe0772e47dccc5eb11912c42bba4633daf87d4) Merge with private repo - [`37ab81a`](https://github.com/deepseek-ai/DeepGEMM/commit/37ab81a35369b9d16cb44c98321638d07f9af2be) Update README - [`d01623b`](https://github.com/deepseek-ai/DeepGEMM/commit/d01623bcc4d1d36b47c4cbbf7e599dc7ce216678) Update README - [`0c72e11`](https://github.com/deepseek-ai/DeepGEMM/commit/0c72e115420fe6dd0456b12d52a86b4a4a326a79) Update README - [`2d6fd1c`](https://github.com/deepseek-ai/DeepGEMM/commit/2d6fd1c5728e682e3c7072846540524308516c1d) Add PyTorch requirements - [`9f4bed6`](https://github.com/deepseek-ai/DeepGEMM/commit/9f4bed637a3d7dd802400a4b4505702b0e7ce136) Fix sync scopes for MQA logits (#256) - [`211d267`](https://github.com/deepseek-ai/DeepGEMM/commit/211d2678d9d178de7a77fbfefe7310e0f4f8821c) Update README ### 📊 Changes **109 files changed** (+12135 additions, -3253 deletions) <details> <summary>View changed files</summary> 📝 `CMakeLists.txt` (+4 -5) 📝 `README.md` (+76 -44) 📝 `csrc/apis/attention.hpp` (+279 -128) 📝 `csrc/apis/einsum.hpp` (+35 -34) 📝 `csrc/apis/gemm.hpp` (+68 -65) 📝 `csrc/apis/hyperconnection.hpp` (+7 -7) 📝 `csrc/apis/layout.hpp` (+46 -23) ➕ `csrc/apis/mega.hpp` (+216 -0) 📝 `csrc/apis/runtime.hpp` (+20 -0) 📝 `csrc/indexing/main.cu` (+6 -1) 📝 `csrc/jit/cache.hpp` (+1 -1) 📝 `csrc/jit/compiler.hpp` (+70 -61) 📝 `csrc/jit/device_runtime.hpp` (+46 -7) 📝 `csrc/jit/handle.hpp` (+72 -18) ➕ `csrc/jit/include_parser.hpp` (+80 -0) 📝 `csrc/jit/kernel_runtime.hpp` (+77 -30) 📝 `csrc/jit_kernels/heuristics/common.hpp` (+34 -319) ➕ `csrc/jit_kernels/heuristics/config.hpp` (+171 -0) ➕ `csrc/jit_kernels/heuristics/mega_moe.hpp` (+211 -0) ➕ `csrc/jit_kernels/heuristics/runtime.hpp` (+62 -0) _...and 80 more files_ </details> ### 📄 Description ### New features - Mega MoE, fusing & overlapping dispatch/linear 1/SwiGLU/linear 2/combine into a single mega-kernel, overlapping NVLink communication and tensor core computation - Performance number: https://github.com/deepseek-ai/DeepGEMM/pull/316 - Only FP8 x FP4 MoE is supported - Requires PyTorch >= 2.9 - FP4 Indexer (MQA logits) with larger MTP support - FP8 x FP4 GEMM - PDL - Refactors on GEMM heuristics - Faster JIT compilation - GEMM optimizations (**dynamic** swap A/B, much faster MoE GEMM) - DeepEPv2 MoE GEMM layout ### Bug fixes - JIT may crash on distributed FS - Some kernel hangs and IMA ### Contributors - Mega MoE: @LyricZhao @zheanxu @bucket-xv @RayWang96 @interestingLSY @kurisu6912 @xay5421 @yukuai26 - FP4 Indexer: @zheanxu @xay5421 @interestingLSY @kurisu6912 - GEMM, PDL, JIT and bug fixes: @zheanxu @bucket-xv @xay5421 @yukuai26 @LyricZhao ### Additional notes Mega MoE is still under development and optimizations, stay tuned and optimization ideas are welcome! **Disclaimer**: this release is only related to DeepGEMM's development, has nothing to do with internal model release. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:16:44 +00:00