[PR #72] [MERGED] mxfp4 marlin kernel layerwise prefill #71

New issue

Closed

opened 2026-10-29 21:06:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:06:57 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/72
Author: @yyj6666667
Created: 8/2/2026
Status: ✅ Merged
Merged: 8/2/2026
Merged by: @yyj6666667

Base: main ← Head: codex/mxfp4-marlin-layerwise-official

📝 Commits (2)

8b741b2 feat(marlin): add DeepSeek V4 MXFP4 MoE backend
180c125 feat(kt): pipeline MXFP4 layerwise prefill

📊 Changes

12 files changed (+3484 additions, -162 deletions)

View changed files

📝 python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh (+93 -5)
📝 python/sglang/jit_kernel/csrc/gemm/marlin_moe/marlin_template.h (+46 -61)
📝 python/sglang/jit_kernel/gptq_marlin_repack.py (+46 -2)
📝 python/sglang/jit_kernel/moe_wna16_marlin.py (+36 -9)
➕ python/sglang/jit_kernel/tests/test_v4_marlin_moe.py (+154 -0)
📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+1070 -48)
📝 python/sglang/srt/layers/quantization/mxfp4_deepseek.py (+146 -37)
➕ python/sglang/srt/layers/quantization/v4_marlin_moe.py (+501 -0)
📝 python/sglang/srt/model_executor/model_runner.py (+22 -0)
➕ test/manual/quant/bench_v4_mxfp4_marlin.py (+104 -0)
➕ test/manual/quant/run_v4_mxfp4_layerwise_e2e.py (+201 -0)
➕ test/unit/test_mxfp4_layerwise_prefill.py (+1065 -0)

📄 Description

GSM8K 200题准确率96.5%, 与官方保持一致。
单卡layerwise prefill 800tps+，已经更新到官网

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/72 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 8/2/2026 **Status:** ✅ Merged **Merged:** 8/2/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `codex/mxfp4-marlin-layerwise-official` --- ### 📝 Commits (2) - [`8b741b2`](https://github.com/kvcache-ai/sglang/commit/8b741b221cdcfb10f7ef26c9aaf0b0e35dded091) feat(marlin): add DeepSeek V4 MXFP4 MoE backend - [`180c125`](https://github.com/kvcache-ai/sglang/commit/180c125b1c32ac159e69826753e72ddc95c1b4cf) feat(kt): pipeline MXFP4 layerwise prefill ### 📊 Changes **12 files changed** (+3484 additions, -162 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh` (+93 -5) 📝 `python/sglang/jit_kernel/csrc/gemm/marlin_moe/marlin_template.h` (+46 -61) 📝 `python/sglang/jit_kernel/gptq_marlin_repack.py` (+46 -2) 📝 `python/sglang/jit_kernel/moe_wna16_marlin.py` (+36 -9) ➕ `python/sglang/jit_kernel/tests/test_v4_marlin_moe.py` (+154 -0) 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+1070 -48) 📝 `python/sglang/srt/layers/quantization/mxfp4_deepseek.py` (+146 -37) ➕ `python/sglang/srt/layers/quantization/v4_marlin_moe.py` (+501 -0) 📝 `python/sglang/srt/model_executor/model_runner.py` (+22 -0) ➕ `test/manual/quant/bench_v4_mxfp4_marlin.py` (+104 -0) ➕ `test/manual/quant/run_v4_mxfp4_layerwise_e2e.py` (+201 -0) ➕ `test/unit/test_mxfp4_layerwise_prefill.py` (+1065 -0) </details> ### 📄 Description GSM8K 200题准确率96.5%, 与官方保持一致。单卡layerwise prefill 800tps+，已经更新到官网 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:06:57 +00:00