sglang-kt #1297

New issue

Open

opened 2026-10-29 14:50:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:50:56 +00:00

Owner

Originally created by @gdevenyi on GitHub (Aug 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2179

Reminder

I have read the above rules and searched the existing issues.

Description

Model: Qwen/Qwen3.8-Flash-Next (HF model_type: qwen4_exp), released 2026-08-26. Qwen4 architecture preview: 125B main model, 6B active/token, 48 layers, MoE with 512 routed experts + 1 shared, top-10 routing; plus 51B N-gram embedding table (PLE) and 4B MTP module.

Request: kt-kernel expert-offload support for qwen4_exp, analogous to the existing Qwen3-Next / Qwen3-Coder-Next sglang-kt integrations.

Why this model fits KTransformers particularly well:

Ultra-sparse routing (10/512) means a hotness-aware GPU-resident expert set (--kt-num-gpu-experts + --kt-enable-dynamic-expert-update) should capture most expert traffic — the skew is stronger than in Qwen3MoE.
The PLE table already has a host-offload story in vanilla SGLang; kt-kernel expert offload is the missing piece that makes single-GPU / small-VRAM serving of this model viable at FP8 expert precision instead of INT4 requantization.
Qwen's own model card recommends KTransformers for production serving of this model, but no qwen4_exp support exists yet.

Architecture deltas vs. Qwen3-Next that affect integration (from the model card / SGLang PRs #36497, #36585):

GDN layers carried over (sigmoid output gate instead of silu per llama.cpp PR #27742 notes); 36 GDN + 12 QSA sparse-attention layers
QSA: lightweight indexer selecting context at micro-block granularity (512 blocks / 2048-token budget)
Gated Residual: 4-branch residual stream, low-rank (r=320) read/write gates
PLE n-gram embedding input path at layer 2
MTP draft layers use QSA as well

The SGLang model graph is landing in sgl-project/sglang#36497 / #36585, so the injection target should exist shortly.

Hardware I can test on: dual Xeon Gold 6526Y (Emerald Rapids, AMX), 512 GB DDR5, 2× RTX 6000 Ada (48 GB, SM89) — and separately a Ryzen 9950X + RTX 5080 (SM120) consumer box. Happy to run benchmarks/validation on both AMX and non-AMX paths, single- and dual-GPU, for a multi-user serving workload (~8 concurrent).

Pull Request

No response

Originally created by @gdevenyi on GitHub (Aug 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2179 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) (HF model_type: qwen4_exp), released 2026-08-26. Qwen4 architecture preview: 125B main model, 6B active/token, 48 layers, MoE with 512 routed experts + 1 shared, top-10 routing; plus 51B N-gram embedding table (PLE) and 4B MTP module. Request: kt-kernel expert-offload support for qwen4_exp, analogous to the existing Qwen3-Next / Qwen3-Coder-Next sglang-kt integrations. Why this model fits KTransformers particularly well: - Ultra-sparse routing (10/512) means a hotness-aware GPU-resident expert set (--kt-num-gpu-experts + --kt-enable-dynamic-expert-update) should capture most expert traffic — the skew is stronger than in Qwen3MoE. - The PLE table already has a host-offload story in vanilla SGLang; kt-kernel expert offload is the missing piece that makes single-GPU / small-VRAM serving of this model viable at FP8 expert precision instead of INT4 requantization. - Qwen's own model card recommends KTransformers for production serving of this model, but no qwen4_exp support exists yet. Architecture deltas vs. Qwen3-Next that affect integration (from the model card / SGLang PRs #36497, #36585): - GDN layers carried over (sigmoid output gate instead of silu per llama.cpp PR #27742 notes); 36 GDN + 12 QSA sparse-attention layers - QSA: lightweight indexer selecting context at micro-block granularity (512 blocks / 2048-token budget) - Gated Residual: 4-branch residual stream, low-rank (r=320) read/write gates - PLE n-gram embedding input path at layer 2 - MTP draft layers use QSA as well The SGLang model graph is landing in sgl-project/sglang#36497 / #36585, so the injection target should exist shortly. Hardware I can test on: dual Xeon Gold 6526Y (Emerald Rapids, AMX), 512 GB DDR5, 2× RTX 6000 Ada (48 GB, SM89) — and separately a Ryzen 9950X + RTX 5080 (SM120) consumer box. Happy to run benchmarks/validation on both AMX and non-AMX paths, single- and dual-GPU, for a multi-user serving workload (~8 concurrent). ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:50:56 +00:00