[PR #81] [MERGED] feat(glm5-next): support GLM-5.3-flash serving #83

New issue

Closed

opened 2026-10-29 21:07:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/81
Author: @yyj6666667
Created: 8/26/2026
Status: ✅ Merged
Merged: 8/26/2026
Merged by: @yyj6666667

Base: main ← Head: feat/glm5-next-video-tools-0808

📝 Commits (10+)

695c70d feat: add GLM-5-Next Session AB runtime
9206288 feat: complete GLM-5-Next Session C runtime
226c54b feat: prepare GLM-5-Next multimodal Session D
640132a fix: validate GLM-5-Next config with transformers-kt
f2eb981 fix: align GLM acceptance with runtime telemetry
ddbb164 refactor: use single-slot GLM layerwise prefill
f4c760f perf(moe): overlap GLM FP8 layerwise transport
429286c feat(glm5-next): support SM86 and SM89 consumer GPUs
487892c feat(glm5-next): support multiple images per request
e4121b2 fix(glm5-next): preserve multi-image boundaries

📊 Changes

88 files changed (+34713 additions, -268 deletions)

View changed files

➕ examples/chat_template/glm5_next_multimodal.jinja (+218 -0)
📝 python/pyproject.toml (+1 -1)
➕ python/sglang/jit_kernel/csrc/nsa/kpool_topk_transform.cuh (+462 -0)
➕ python/sglang/jit_kernel/kpool_topk_transform.py (+80 -0)
➕ python/sglang/kernels/ops/layernorm/mhc.py (+284 -0)
📝 python/sglang/srt/configs/__init__.py (+3 -0)
➕ python/sglang/srt/configs/glm5_next.py (+629 -0)
📝 python/sglang/srt/configs/model_config.py (+26 -3)
📝 python/sglang/srt/entrypoints/openai/protocol.py (+1 -0)
📝 python/sglang/srt/entrypoints/openai/serving_chat.py (+25 -12)
📝 python/sglang/srt/function_call/base_format_detector.py (+6 -0)
📝 python/sglang/srt/function_call/function_call_parser.py (+58 -1)
📝 python/sglang/srt/function_call/glm47_moe_detector.py (+85 -1)
📝 python/sglang/srt/layers/attention/attention_registry.py (+26 -0)
📝 python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py (+17 -0)
➕ python/sglang/srt/layers/attention/linear/glm5_next_kda_backend.py (+214 -0)
➕ python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda.py (+122 -0)
➕ python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda_ops.py (+536 -0)
➕ python/sglang/srt/layers/attention/nsa/glm5_next_indexer_logits.py (+353 -0)
➕ python/sglang/srt/layers/attention/nsa/glm5_next_indexer_triton.py (+386 -0)

...and 68 more files

📄 Description

Summary

add native GLM-5-Next / GLM-5.3-flash text and multimodal serving
implement the 34-layer KDA-style linear-attention path and 11-layer DSA path
add FP8 heterogeneous MoE execution, CUDA Graph decode, and Layerwise Prefill
support TP1, TP2, TP4, and TP8; Layerwise Prefill uses zero resident GPU experts
support SM86, SM89, and SM120 consumer GPUs
support text with up to eight images or text with one video, while rejecting mixed image/video requests
preserve GLM reasoning and tool-call parsing for OpenAI-compatible coding-agent clients

Dependency

This branch requires transformers-kt==5.6.0.post3 for the GLM image/video processors. The package-ready source is pinned on kvcache-ai/transformers at commit 3f97700; it has not been published to PyPI yet. Publication is intentionally handled by the coordinated KTransformers release workflow.

Validation

GLM unit/contract suite: 282 passed, 13 skipped locally; the remaining processor import test requires torchvision in the local test environment and is covered by the full runtime environment
end-to-end text serving and decode CUDA Graph validation on RTX 5090
TP1/TP2/TP4/TP8 Layerwise Prefill validation on RTX 5090
TP1/TP2/TP4 serving validation on RTX 4090
semantically correct image request on RTX 4090 TP4
multi-image, single-video, reasoning-parser, and tool-call-parser contract coverage
Qwen3-30B-A3B regression smoke with CUDA Graph enabled

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/81 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 8/26/2026 **Status:** ✅ Merged **Merged:** 8/26/2026 **Merged by:** [@yyj6666667](https://github.com/yyj6666667) **Base:** `main` ← **Head:** `feat/glm5-next-video-tools-0808` --- ### 📝 Commits (10+) - [`695c70d`](https://github.com/kvcache-ai/sglang/commit/695c70d98cf58b5cc316c09c742a09a353c30c7d) feat: add GLM-5-Next Session AB runtime - [`9206288`](https://github.com/kvcache-ai/sglang/commit/92062885634e27d807d46fa5389b9e1779e599eb) feat: complete GLM-5-Next Session C runtime - [`226c54b`](https://github.com/kvcache-ai/sglang/commit/226c54b3210744185fac2b253db7a8e50ea4464e) feat: prepare GLM-5-Next multimodal Session D - [`640132a`](https://github.com/kvcache-ai/sglang/commit/640132af61b3854ba0f77865967c0869097308ec) fix: validate GLM-5-Next config with transformers-kt - [`f2eb981`](https://github.com/kvcache-ai/sglang/commit/f2eb981a8c661678c3d19c7d30dc0b0ae5329e2f) fix: align GLM acceptance with runtime telemetry - [`ddbb164`](https://github.com/kvcache-ai/sglang/commit/ddbb164b959387fa86ae6c3240b4cc76b2d66367) refactor: use single-slot GLM layerwise prefill - [`f4c760f`](https://github.com/kvcache-ai/sglang/commit/f4c760f46c892cc001b0fe76860e8057d746085f) perf(moe): overlap GLM FP8 layerwise transport - [`429286c`](https://github.com/kvcache-ai/sglang/commit/429286cdd5e29e86b492ce5a93487bc773664330) feat(glm5-next): support SM86 and SM89 consumer GPUs - [`487892c`](https://github.com/kvcache-ai/sglang/commit/487892c97857bd7a3410c705b1f0e9a7095cf298) feat(glm5-next): support multiple images per request - [`e4121b2`](https://github.com/kvcache-ai/sglang/commit/e4121b2b3abede048d76ea47af9ac3c3dcdb2e6a) fix(glm5-next): preserve multi-image boundaries ### 📊 Changes **88 files changed** (+34713 additions, -268 deletions) <details> <summary>View changed files</summary> ➕ `examples/chat_template/glm5_next_multimodal.jinja` (+218 -0) 📝 `python/pyproject.toml` (+1 -1) ➕ `python/sglang/jit_kernel/csrc/nsa/kpool_topk_transform.cuh` (+462 -0) ➕ `python/sglang/jit_kernel/kpool_topk_transform.py` (+80 -0) ➕ `python/sglang/kernels/ops/layernorm/mhc.py` (+284 -0) 📝 `python/sglang/srt/configs/__init__.py` (+3 -0) ➕ `python/sglang/srt/configs/glm5_next.py` (+629 -0) 📝 `python/sglang/srt/configs/model_config.py` (+26 -3) 📝 `python/sglang/srt/entrypoints/openai/protocol.py` (+1 -0) 📝 `python/sglang/srt/entrypoints/openai/serving_chat.py` (+25 -12) 📝 `python/sglang/srt/function_call/base_format_detector.py` (+6 -0) 📝 `python/sglang/srt/function_call/function_call_parser.py` (+58 -1) 📝 `python/sglang/srt/function_call/glm47_moe_detector.py` (+85 -1) 📝 `python/sglang/srt/layers/attention/attention_registry.py` (+26 -0) 📝 `python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py` (+17 -0) ➕ `python/sglang/srt/layers/attention/linear/glm5_next_kda_backend.py` (+214 -0) ➕ `python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda.py` (+122 -0) ➕ `python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda_ops.py` (+536 -0) ➕ `python/sglang/srt/layers/attention/nsa/glm5_next_indexer_logits.py` (+353 -0) ➕ `python/sglang/srt/layers/attention/nsa/glm5_next_indexer_triton.py` (+386 -0) _...and 68 more files_ </details> ### 📄 Description ## Summary - add native GLM-5-Next / GLM-5.3-flash text and multimodal serving - implement the 34-layer KDA-style linear-attention path and 11-layer DSA path - add FP8 heterogeneous MoE execution, CUDA Graph decode, and Layerwise Prefill - support TP1, TP2, TP4, and TP8; Layerwise Prefill uses zero resident GPU experts - support SM86, SM89, and SM120 consumer GPUs - support text with up to eight images or text with one video, while rejecting mixed image/video requests - preserve GLM reasoning and tool-call parsing for OpenAI-compatible coding-agent clients ## Dependency This branch requires `transformers-kt==5.6.0.post3` for the GLM image/video processors. The package-ready source is pinned on `kvcache-ai/transformers` at commit `3f97700`; it has not been published to PyPI yet. Publication is intentionally handled by the coordinated KTransformers release workflow. ## Validation - GLM unit/contract suite: 282 passed, 13 skipped locally; the remaining processor import test requires torchvision in the local test environment and is covered by the full runtime environment - end-to-end text serving and decode CUDA Graph validation on RTX 5090 - TP1/TP2/TP4/TP8 Layerwise Prefill validation on RTX 5090 - TP1/TP2/TP4 serving validation on RTX 4090 - semantically correct image request on RTX 4090 TP4 - multi-image, single-video, reasoning-parser, and tool-call-parser contract coverage - Qwen3-30B-A3B regression smoke with CUDA Graph enabled --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:07:04 +00:00