[PR #84] feat(dsv4): add SGLANG_V4_MARLIN_PARTIAL for partial-expert MoE #84

New issue

Open

opened 2026-10-29 21:07:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/84
Author: @05yuki
Created: 8/27/2026
Status: 🔄 Open

Base: main ← Head: feat/dsv4-marlin-partial-experts

📝 Commits (1)

d2effcb feat(dsv4): add SGLANG_V4_MARLIN_PARTIAL for partial-expert MoE

📊 Changes

2 files changed (+9 additions, -1 deletions)

View changed files

📝 python/sglang/srt/environ.py (+2 -0)
📝 python/sglang/srt/layers/quantization/mxfp4_deepseek.py (+7 -1)

📄 Description

When KTransformers keeps only a subset of experts resident on the GPU,
the resident partial-expert image is deliberately excluded from the
Marlin W4A16 path and always runs through the triton_kernels matmul_ogs
fallback.

That fallback reserves a fixed device transient at prefill launch,
independent of batch size: measured at roughly 0.5 GB, and about 0.85 GB
in total at 2048-token chunks. A 16 GB card that is also holding a large
KV pool has no room for it, so every prompt long enough to trigger a
real prefill dies with CUDA OOM in the scheduler while short benchmark
prompts keep working. Marlin has no comparable reservation.

Add an opt-in escape hatch: SGLANG_V4_MARLIN_PARTIAL=1 lets the resident
partial-expert image take the Marlin path too. The default is unchanged,
so this cannot affect existing deployments.

Measured on 2x RTX 5070 Ti (SM120, 16 GB each) with DeepSeek-V4-Flash
MXFP4 and KT CPU experts, at a 1M-token pool and 256-token chunks:

before: any prompt beyond ~100 tokens OOMs at prefill
after: 3001-token prefill in 35.6 s, 12001-token in 141 s (~85 tok/s)
decode unchanged at 32.3 tok/s, output stays coherent

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/84 **Author:** [@05yuki](https://github.com/05yuki) **Created:** 8/27/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/dsv4-marlin-partial-experts` --- ### 📝 Commits (1) - [`d2effcb`](https://github.com/kvcache-ai/sglang/commit/d2effcb75222e39fc4dfd83768b6e2a969899cdb) feat(dsv4): add SGLANG_V4_MARLIN_PARTIAL for partial-expert MoE ### 📊 Changes **2 files changed** (+9 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/environ.py` (+2 -0) 📝 `python/sglang/srt/layers/quantization/mxfp4_deepseek.py` (+7 -1) </details> ### 📄 Description When KTransformers keeps only a subset of experts resident on the GPU, the resident partial-expert image is deliberately excluded from the Marlin W4A16 path and always runs through the triton_kernels matmul_ogs fallback. That fallback reserves a fixed device transient at prefill launch, independent of batch size: measured at roughly 0.5 GB, and about 0.85 GB in total at 2048-token chunks. A 16 GB card that is also holding a large KV pool has no room for it, so every prompt long enough to trigger a real prefill dies with CUDA OOM in the scheduler while short benchmark prompts keep working. Marlin has no comparable reservation. Add an opt-in escape hatch: SGLANG_V4_MARLIN_PARTIAL=1 lets the resident partial-expert image take the Marlin path too. The default is unchanged, so this cannot affect existing deployments. Measured on 2x RTX 5070 Ti (SM120, 16 GB each) with DeepSeek-V4-Flash MXFP4 and KT CPU experts, at a 1M-token pool and 256-token chunks: - before: any prompt beyond ~100 tokens OOMs at prefill - after: 3001-token prefill in 35.6 s, 12001-token in 141 s (~85 tok/s) - decode unchanged at 32.3 tok/s, output stays coherent --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:07:04 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#84

No description provided.

Rows
Columns