[PR #82] [MERGED] fix(lora): support all-target KT MoE adapters #81

New issue

Closed

opened 2026-10-29 21:07:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/82
Author: @yyj6666667
Created: 8/27/2026
Status: ✅ Merged
Merged: 8/31/2026
Merged by: @yyj6666667

Base: main ← Head: fix/kt-lora-all-model-serving

📝 Commits (6)

9a4de96 fix(lora): enforce KT composite adapter contract
9405729 feat(lora): support all-target Qwen MoE adapters
adf32b0 feat(lora): serve DeepSeek adapters with native FP8 experts
45058c3 fix(lora): avoid registry refs for static KT adapters
6820dfc fix(lora): harden mixed-shape Qwen3.5 execution
5047631 fix(lora): preserve DeepSeek deltas in optimized paths

📊 Changes

24 files changed (+2930 additions, -235 deletions)

View changed files

📝 python/sglang/srt/entrypoints/grpc_server.py (+29 -0)
📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+135 -5)
📝 python/sglang/srt/lora/layers.py (+58 -0)
📝 python/sglang/srt/lora/lora_manager.py (+275 -10)
📝 python/sglang/srt/lora/mem_pool.py (+112 -114)
📝 python/sglang/srt/lora/triton_ops/chunked_sgmv_expand.py (+10 -1)
📝 python/sglang/srt/lora/triton_ops/chunked_sgmv_shrink.py (+8 -1)
📝 python/sglang/srt/lora/utils.py (+30 -8)
📝 python/sglang/srt/managers/tokenizer_communicator_mixin.py (+57 -0)
📝 python/sglang/srt/managers/tokenizer_manager.py (+80 -4)
📝 python/sglang/srt/model_executor/cuda_graph_runner.py (+1 -3)
📝 python/sglang/srt/model_executor/model_runner.py (+1 -1)
📝 python/sglang/srt/model_executor/piecewise_cuda_graph_runner.py (+2 -4)
📝 python/sglang/srt/models/deepseek_common/deepseek_weight_loader.py (+4 -2)
📝 python/sglang/srt/models/deepseek_v2.py (+240 -17)
📝 python/sglang/srt/models/qwen2_moe.py (+4 -2)
📝 python/sglang/srt/models/qwen3_5.py (+14 -15)
📝 python/sglang/srt/models/qwen3_moe.py (+30 -0)
📝 python/sglang/srt/server_args.py (+284 -48)
📝 python/sglang/srt/utils/common.py (+14 -0)

...and 4 more files

📄 Description

What does this PR do?

This PR completes the static KT composite LoRA serving path for supported MoE models.

A merged adapter is loaded once, split into GPU non-expert and KT routed-expert tensors, and exposed under one immutable adapter ID.

Main changes

Enforce exact module matching, module-aware TP sharding, and full adapter-tensor consumption checks.
Reject unsupported static-path usage, including dynamic adapter replacement, mixed base/adapter batches, pipeline parallelism, and CUDA graph capture.
Add the missing model paths:
- Qwen3 router LoRA.
- Qwen3.5 router, shared-expert gate/MLP, attention, and GatedDeltaNet LoRA.
- DeepSeek router, shared experts, independent MLA LoRA pairs, and native block-FP8 routed experts.
Fix runtime issues found by real-model validation:
- Key cached CSGMV kernels by every compile-time shape parameter, preventing reuse across incompatible mixed-width projections.
- Apply DeepSeek kv_b_proj LoRA in MLA absorb paths without merging or requantizing the native FP8 base weight.
- Enable the shared backward BufferB required by KT FP8/INT8 SFT wrappers used for static expert LoRA serving.

Tests

65 passed across the common contract, request-policy, Qwen MoE, and DeepSeek suites, including a real CUDA regression for mixed-width CSGMV launches.

Real adapters were validated on sap4 with startup, generation, chat completion, request-policy checks, graceful shutdown, and TP1/TP2 comparison:

Model	Adapter coverage	TP1	TP2	TP parity
Qwen3-235B-A22B	Routed experts	Pass	Pass	Pass
Qwen3-30B-A3B	Router and routed experts	Pass	Pass	Pass
Qwen3.5-397B-A17B	Router, shared experts, attention, and GatedDeltaNet	Pass	Pass	Pass
DeepSeek-V3.1	All-target LoRA with native FP8 experts	Pass	Pass	Pass

TP parity is a tolerance-based distribution check under fixed prompts: exact input top-1 token IDs, bounded selected-token log-probability differences, top-20 overlap, a non-empty common output prefix, and successful generation. It does not claim bit-exact output across TP sizes. Qwen3.5 additionally produced identical TP1/TP2 greedy tokens, generation text, and chat text.

black --check, isort --check-only, ruff --select F401,F821, py_compile, and git diff --check pass for all changed Python files. The sap4 validation source matches the submitted files by SHA256.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/82 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 8/27/2026 **Status:** ✅ Merged **Merged:** 8/31/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `fix/kt-lora-all-model-serving` --- ### 📝 Commits (6) - [`9a4de96`](https://github.com/kvcache-ai/sglang/commit/9a4de96491a095c324280eb084ee3a805f931a6a) fix(lora): enforce KT composite adapter contract - [`9405729`](https://github.com/kvcache-ai/sglang/commit/9405729c991dc3f636797c0798e2a55a6faf422a) feat(lora): support all-target Qwen MoE adapters - [`adf32b0`](https://github.com/kvcache-ai/sglang/commit/adf32b0572832f27ce40271eb4c1b5aa634a2c90) feat(lora): serve DeepSeek adapters with native FP8 experts - [`45058c3`](https://github.com/kvcache-ai/sglang/commit/45058c36ce170991b528e7f200789fccc9659eab) fix(lora): avoid registry refs for static KT adapters - [`6820dfc`](https://github.com/kvcache-ai/sglang/commit/6820dfcf92dbb05fe53987b7cd49b9f96f28c3e0) fix(lora): harden mixed-shape Qwen3.5 execution - [`5047631`](https://github.com/kvcache-ai/sglang/commit/5047631283bf246181e075255f0f86d78313c4c0) fix(lora): preserve DeepSeek deltas in optimized paths ### 📊 Changes **24 files changed** (+2930 additions, -235 deletions) <details> <summary>View changed files</summary> 📝 `python/sglang/srt/entrypoints/grpc_server.py` (+29 -0) 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+135 -5) 📝 `python/sglang/srt/lora/layers.py` (+58 -0) 📝 `python/sglang/srt/lora/lora_manager.py` (+275 -10) 📝 `python/sglang/srt/lora/mem_pool.py` (+112 -114) 📝 `python/sglang/srt/lora/triton_ops/chunked_sgmv_expand.py` (+10 -1) 📝 `python/sglang/srt/lora/triton_ops/chunked_sgmv_shrink.py` (+8 -1) 📝 `python/sglang/srt/lora/utils.py` (+30 -8) 📝 `python/sglang/srt/managers/tokenizer_communicator_mixin.py` (+57 -0) 📝 `python/sglang/srt/managers/tokenizer_manager.py` (+80 -4) 📝 `python/sglang/srt/model_executor/cuda_graph_runner.py` (+1 -3) 📝 `python/sglang/srt/model_executor/model_runner.py` (+1 -1) 📝 `python/sglang/srt/model_executor/piecewise_cuda_graph_runner.py` (+2 -4) 📝 `python/sglang/srt/models/deepseek_common/deepseek_weight_loader.py` (+4 -2) 📝 `python/sglang/srt/models/deepseek_v2.py` (+240 -17) 📝 `python/sglang/srt/models/qwen2_moe.py` (+4 -2) 📝 `python/sglang/srt/models/qwen3_5.py` (+14 -15) 📝 `python/sglang/srt/models/qwen3_moe.py` (+30 -0) 📝 `python/sglang/srt/server_args.py` (+284 -48) 📝 `python/sglang/srt/utils/common.py` (+14 -0) _...and 4 more files_ </details> ### 📄 Description ## What does this PR do? This PR completes the static KT composite LoRA serving path for supported MoE models. A merged adapter is loaded once, split into GPU non-expert and KT routed-expert tensors, and exposed under one immutable adapter ID. ### Main changes - Enforce exact module matching, module-aware TP sharding, and full adapter-tensor consumption checks. - Reject unsupported static-path usage, including dynamic adapter replacement, mixed base/adapter batches, pipeline parallelism, and CUDA graph capture. - Add the missing model paths: - Qwen3 router LoRA. - Qwen3.5 router, shared-expert gate/MLP, attention, and GatedDeltaNet LoRA. - DeepSeek router, shared experts, independent MLA LoRA pairs, and native block-FP8 routed experts. - Fix runtime issues found by real-model validation: - Key cached CSGMV kernels by every compile-time shape parameter, preventing reuse across incompatible mixed-width projections. - Apply DeepSeek `kv_b_proj` LoRA in MLA absorb paths without merging or requantizing the native FP8 base weight. - Enable the shared backward BufferB required by KT FP8/INT8 SFT wrappers used for static expert LoRA serving. ## Tests `65 passed` across the common contract, request-policy, Qwen MoE, and DeepSeek suites, including a real CUDA regression for mixed-width CSGMV launches. Real adapters were validated on sap4 with startup, generation, chat completion, request-policy checks, graceful shutdown, and TP1/TP2 comparison: | Model | Adapter coverage | TP1 | TP2 | TP parity | |---|---|---:|---:|---:| | Qwen3-235B-A22B | Routed experts | Pass | Pass | Pass | | Qwen3-30B-A3B | Router and routed experts | Pass | Pass | Pass | | Qwen3.5-397B-A17B | Router, shared experts, attention, and GatedDeltaNet | Pass | Pass | Pass | | DeepSeek-V3.1 | All-target LoRA with native FP8 experts | Pass | Pass | Pass | TP parity is a tolerance-based distribution check under fixed prompts: exact input top-1 token IDs, bounded selected-token log-probability differences, top-20 overlap, a non-empty common output prefix, and successful generation. It does not claim bit-exact output across TP sizes. Qwen3.5 additionally produced identical TP1/TP2 greedy tokens, generation text, and chat text. `black --check`, `isort --check-only`, `ruff --select F401,F821`, `py_compile`, and `git diff --check` pass for all changed Python files. The sap4 validation source matches the submitted files by SHA256. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:07:04 +00:00