[PR #396] [MERGED] feat: add SiTU support to FP8/FP4 Mega MoE #415

New issue

Closed

opened 2026-10-26 21:17:31 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:17:31 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/396
Author: @Barry-Delaney
Created: 7/30/2026
Status: ✅ Merged
Merged: 8/11/2026
Merged by: @RayWang96

Base: nv_dev ← Head: feat/mega-moe-situ

📝 Commits (2)

7f17040 feat: add SiTU support to FP8 FP4 MegaMoE on 2.6.1
d86a8f6 perf: refine SiTU activation handling

📊 Changes

5 files changed (+292 additions, -11 deletions)

View changed files

📝 csrc/apis/mega.hpp (+17 -4)
📝 csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp (+15 -2)
📝 deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh (+29 -1)
📝 deep_gemm/mega/__init__.py (+10 -4)
➕ tests/test_mega_moe_situ.py (+221 -0)

📄 Description

Summary

Add SiTU gated activation support to SM100 FP8/FP4 Mega MoE while preserving the default SwiGLU path.
Thread activation, situ_beta, and situ_linear_beta through symmetric-buffer validation, Python/C++ APIs, JIT specialization, and the routed/shared L1 epilogue.
Restrict SiTU to FP8xFP4, reject activation clamp and non-positive or non-finite beta values, and keep BF16 Mega MoE SwiGLU-only.

This is a semantic port of longlee0622/DeepGEMM@cc832217 onto the latest nv_dev. It uses the author refactor-ready follow-up 39d7cac and preserves the original authorship and sign-off.

Validation

Built the exact PR commit with ./develop.sh.
Passed the focused deterministic routed SiTU regression for both fast_math modes.
Verified high-beta SiTU converges to SwiGLU, each beta control changes the result independently, the two controls remain distinct, and every output is finite with a non-degenerate baseline.
Verified situ_beta=+inf is rejected by the API validation.
PTXAS comparison reported the same 56-byte stack frame and zero spill stores/loads for SiTU and SwiGLU.
git diff --check and Python py_compile passed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/396 **Author:** [@Barry-Delaney](https://github.com/Barry-Delaney) **Created:** 7/30/2026 **Status:** ✅ Merged **Merged:** 8/11/2026 **Merged by:** [@RayWang96](https://github.com/RayWang96) **Base:** `nv_dev` ← **Head:** `feat/mega-moe-situ` --- ### 📝 Commits (2) - [`7f17040`](https://github.com/deepseek-ai/DeepGEMM/commit/7f170404c3f36877bcaa2f42017b724bfd61dc74) feat: add SiTU support to FP8 FP4 MegaMoE on 2.6.1 - [`d86a8f6`](https://github.com/deepseek-ai/DeepGEMM/commit/d86a8f6d6f448d3f3e2b4f0b58ac8149bdc0ce36) perf: refine SiTU activation handling ### 📊 Changes **5 files changed** (+292 additions, -11 deletions) <details> <summary>View changed files</summary> 📝 `csrc/apis/mega.hpp` (+17 -4) 📝 `csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp` (+15 -2) 📝 `deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh` (+29 -1) 📝 `deep_gemm/mega/__init__.py` (+10 -4) ➕ `tests/test_mega_moe_situ.py` (+221 -0) </details> ### 📄 Description ## Summary - Add SiTU gated activation support to SM100 FP8/FP4 Mega MoE while preserving the default SwiGLU path. - Thread `activation`, `situ_beta`, and `situ_linear_beta` through symmetric-buffer validation, Python/C++ APIs, JIT specialization, and the routed/shared L1 epilogue. - Restrict SiTU to FP8xFP4, reject activation clamp and non-positive or non-finite beta values, and keep BF16 Mega MoE SwiGLU-only. This is a semantic port of [longlee0622/DeepGEMM@cc832217](https://github.com/longlee0622/DeepGEMM/commit/cc832217b65bbe693a9eaa64f1077f65daaee67c) onto the latest `nv_dev`. It uses the author refactor-ready follow-up [39d7cac](https://github.com/longlee0622/DeepGEMM/commit/39d7cac2170b12768a324d5013eeb2a4c81c82fa) and preserves the original authorship and sign-off. ## Validation - Built the exact PR commit with `./develop.sh`. - Passed the focused deterministic routed SiTU regression for both `fast_math` modes. - Verified high-beta SiTU converges to SwiGLU, each beta control changes the result independently, the two controls remain distinct, and every output is finite with a non-degenerate baseline. - Verified `situ_beta=+inf` is rejected by the API validation. - PTXAS comparison reported the same 56-byte stack frame and zero spill stores/loads for SiTU and SwiGLU. - `git diff --check` and Python `py_compile` passed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:17:31 +00:00