[PR #112] [MERGED] Add more GPU architectures support #252

New issue

Closed

opened 2026-10-26 21:15:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:15:22 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/112
Author: @RayWang96
Created: 6/12/2025
Status: ✅ Merged
Merged: 7/18/2025
Merged by: @LyricZhao

Base: main ← Head: multi_arch_support

📝 Commits (5)

a437e0b Add more GPU architectures support
cc416ee Update layout.py
c52b1c1 Optimize performance, Add SM90 support, Add 1D2D SM100 support
2e937c4 Add fmtlib submodule at commit 553ec11
4c4ff2e Merge commit '03d0be3d2d' into multi_arch_support

📊 Changes

67 files changed (+5575 additions, -2954 deletions)

View changed files

📝 .gitmodules (+4 -1)
📝 CMakeLists.txt (+22 -33)
📝 README.md (+53 -115)
➕ csrc/indexing/main.cu (+13 -0)
➕ csrc/jit/cache.hpp (+31 -0)
➕ csrc/jit/compiler.hpp (+172 -0)
➕ csrc/jit/device_runtime.hpp (+50 -0)
➕ csrc/jit/kernel_runtime.hpp (+139 -0)
➕ csrc/jit_kernels/heuristics/common.hpp (+298 -0)
➕ csrc/jit_kernels/heuristics/sm100.hpp (+144 -0)
➕ csrc/jit_kernels/heuristics/sm90.hpp (+115 -0)
➕ csrc/jit_kernels/impls/runtime_utils.hpp (+173 -0)
➕ csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp (+351 -0)
➕ csrc/jit_kernels/impls/sm100_fp8_gemm_1d2d.hpp (+242 -0)
➕ csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp (+255 -0)
➕ csrc/jit_kernels/impls/smxx_layout.hpp (+199 -0)
➕ csrc/python_api.cpp (+402 -0)
➕ csrc/utils/exception.hpp (+58 -0)
➕ csrc/utils/format.hpp (+6 -0)
➕ csrc/utils/hash.hpp (+35 -0)

...and 47 more files

📄 Description

Hi DeepGEMM Team,

This PR is submitted by NVIDIA and introduces support for more GPU architectures.

Key Points

MXFP8 support
- Same 1x128 by 128x128 SF (scaling factor) recipe as original DeepGEMM, but using UE8M0 instead of FP32 for SF data format.
- Same SF input layout as original DeepGEMM.
Multi-architecture support
- Refactored the file structure for better extensibility, working on merging other GPU architectures (e.g. Ada) into DeepGEMM.
- Notice: code structural changes are significant, we'd like to align with you on a multi-architecture support strategy before proceeding further.

On-going and coming soon

Small M optimization (~128)
Very small M values (<64), with swap AB optimization
Masked grouped GEMM optimization

Long-term optimization

Other architecture support (e.g. Ada)
Add more input format support such as offset layout used by TRT-LLM

We are actively working on these optimizations. It aims to continuously provide support for NVIDIA's latest GPU hardware in DeepGEMM, ensuring the community can perform efficient and scalable training and inference with DeepSeek models for next-generation AI workloads.

Thanks!

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/112 **Author:** [@RayWang96](https://github.com/RayWang96) **Created:** 6/12/2025 **Status:** ✅ Merged **Merged:** 7/18/2025 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `multi_arch_support` --- ### 📝 Commits (5) - [`a437e0b`](https://github.com/deepseek-ai/DeepGEMM/commit/a437e0b1ca2744c83c2b301caf8df0584a28f910) Add more GPU architectures support - [`cc416ee`](https://github.com/deepseek-ai/DeepGEMM/commit/cc416ee4faf0533a9263c2de814e5565f56ca1cc) Update layout.py - [`c52b1c1`](https://github.com/deepseek-ai/DeepGEMM/commit/c52b1c1a78df3ea73526887b1360cb47c97b71e9) Optimize performance, Add SM90 support, Add 1D2D SM100 support - [`2e937c4`](https://github.com/deepseek-ai/DeepGEMM/commit/2e937c4656963c961265fd08a0c025c93127dd1f) Add fmtlib submodule at commit 553ec11 - [`4c4ff2e`](https://github.com/deepseek-ai/DeepGEMM/commit/4c4ff2e4ffcdbd8ddac3fb1f9c8caccec27f932a) Merge commit '03d0be3d2d03b6eed3c99d683c0620949a13a826' into multi_arch_support ### 📊 Changes **67 files changed** (+5575 additions, -2954 deletions) <details> <summary>View changed files</summary> 📝 `.gitmodules` (+4 -1) 📝 `CMakeLists.txt` (+22 -33) 📝 `README.md` (+53 -115) ➕ `csrc/indexing/main.cu` (+13 -0) ➕ `csrc/jit/cache.hpp` (+31 -0) ➕ `csrc/jit/compiler.hpp` (+172 -0) ➕ `csrc/jit/device_runtime.hpp` (+50 -0) ➕ `csrc/jit/kernel_runtime.hpp` (+139 -0) ➕ `csrc/jit_kernels/heuristics/common.hpp` (+298 -0) ➕ `csrc/jit_kernels/heuristics/sm100.hpp` (+144 -0) ➕ `csrc/jit_kernels/heuristics/sm90.hpp` (+115 -0) ➕ `csrc/jit_kernels/impls/runtime_utils.hpp` (+173 -0) ➕ `csrc/jit_kernels/impls/sm100_fp8_gemm_1d1d.hpp` (+351 -0) ➕ `csrc/jit_kernels/impls/sm100_fp8_gemm_1d2d.hpp` (+242 -0) ➕ `csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp` (+255 -0) ➕ `csrc/jit_kernels/impls/smxx_layout.hpp` (+199 -0) ➕ `csrc/python_api.cpp` (+402 -0) ➕ `csrc/utils/exception.hpp` (+58 -0) ➕ `csrc/utils/format.hpp` (+6 -0) ➕ `csrc/utils/hash.hpp` (+35 -0) _...and 47 more files_ </details> ### 📄 Description Hi DeepGEMM Team, This PR is submitted by NVIDIA and introduces support for more GPU architectures. ## Key Points - MXFP8 support - Same 1x128 by 128x128 SF (scaling factor) recipe as original DeepGEMM, but using [UE8M0](https://docs.nvidia.com/cuda/parallel-thread-execution/#alternate-floating-point-data-formats) instead of FP32 for SF data format. - Same SF input layout as original DeepGEMM. - Multi-architecture support - Refactored the file structure for better extensibility, working on merging other GPU architectures (e.g. Ada) into DeepGEMM. - Notice: code structural changes are significant, we'd like to align with you on a multi-architecture support strategy before proceeding further. ## On-going and coming soon - Small M optimization (~128) - Very small M values (<64), with swap AB optimization - Masked grouped GEMM optimization ## Long-term optimization - Other architecture support (e.g. Ada) - Add more input format support such as offset layout used by TRT-LLM We are actively working on these optimizations. It aims to continuously provide support for NVIDIA's latest GPU hardware in DeepGEMM, ensuring the community can perform efficient and scalable training and inference with DeepSeek models for next-generation AI workloads. Thanks! --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:15:22 +00:00