[PR #80] [perf] Overlap GLM-5-Next FP8 layerwise prefill transport #80

New issue

Open

opened 2026-10-29 21:07:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:02 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/80
Author: @yyj6666667
Created: 8/16/2026
Status: 🔄 Open

Base: feat/glm5-next-0808 ← Head: glm5-next-session-ab

📝 Commits (6)

ddbb164 refactor: use single-slot GLM layerwise prefill
f4c760f perf(moe): overlap GLM FP8 layerwise transport
429286c feat(glm5-next): support SM86 and SM89 consumer GPUs
487892c feat(glm5-next): support multiple images per request
e4121b2 fix(glm5-next): preserve multi-image boundaries
845f7ff Merge pull request #2 from yyj6666667/glm5-next-multi-image

📊 Changes

33 files changed (+6147 additions, -2814 deletions)

View changed files

➕ examples/chat_template/glm5_next_multimodal.jinja (+28 -0)
📝 python/sglang/srt/configs/glm5_next.py (+56 -1)
📝 python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda_ops.py (+33 -9)
➕ python/sglang/srt/layers/attention/nsa/glm5_next_indexer_triton.py (+386 -0)
📝 python/sglang/srt/layers/attention/nsa/glm5_next_sparse_attention.py (+37 -4)
📝 python/sglang/srt/layers/attention/nsa/kpool_fp8_index.py (+598 -2)
📝 python/sglang/srt/layers/attention/nsa/kpool_plan.py (+31 -1)
📝 python/sglang/srt/layers/attention/nsa/nsa_indexer_kpool.py (+112 -37)
📝 python/sglang/srt/layers/attention/nsa_backend.py (+43 -11)
📝 python/sglang/srt/layers/moe/kt_ep_wrapper.py (+1392 -1543)
📝 python/sglang/srt/mem_cache/glm5_next_memory_pool.py (+16 -0)
📝 python/sglang/srt/mem_cache/memory_pool.py (+40 -2)
📝 python/sglang/srt/model_executor/model_runner.py (+0 -19)
📝 python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py (+11 -7)
📝 python/sglang/srt/multimodal/processors/glm5_next.py (+255 -27)
📝 python/sglang/srt/server_args.py (+57 -34)
➕ scripts/glm5_next_consumer_gpu_compile.py (+636 -0)
📝 scripts/glm5_next_session_c_final_acceptance.py (+255 -117)
📝 scripts/glm5_next_session_c_graph_acceptance.py (+1 -1)
📝 scripts/glm5_next_session_c_qa_gsm_graph_acceptance.py (+3 -3)

...and 13 more files

📄 Description

Summary

add an exact GLM-5-Next FP8/TP4 single-GPU-slot prefetch controller
synchronously prime layer 3, then load layer n+1 on one persistent per-rank worker after the current-stream consumed event
overlap the successor native writer/H2D transport with postprocess and the next attention layer
keep exactly two Host expert slots and one full-layer GPU slot
fail closed on incomplete layer registry, stale ordering, pointer rebinding, rank-local failure, or poisoned transport

The path is hard-gated to exact GLM-5-Next, block-FP8, TP=4, gpu_experts=0, raw Triton MoE. Legacy transport, MXFP4, BF16, and other models retain their existing paths.

Depends on kvcache-ai/ktransformers#2160.

Validation

focused GLM tests: 18 passed + 13 subtests
MXFP4 preservation tests: 27 passed
TP4 GPUs 0-3, CUDA Graph enabled, 30K prompt:
- chunk 8192 hot TTFT: 33.637s vs legacy 41.229s (-18.4%)
- chunk 16384 hot TTFT: 26.652s vs legacy 33.113s (-19.5%)
output token hashes match the corresponding legacy/native baselines exactly
natural 29,835-token Chinese prompt remained coherent:
- chunk 8192: 854.6 tok/s prefill
- chunk 16384: 1112.3 tok/s prefill
Qwen3-30B-A3B TP4 regression: 23/23 passed, including CUDA Graph batches 1/2/4 and deterministic top-256 checks

This PR is submitted for review only and must not be auto-merged.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/80 **Author:** [@yyj6666667](https://github.com/yyj6666667) **Created:** 8/16/2026 **Status:** 🔄 Open **Base:** `feat/glm5-next-0808` ← **Head:** `glm5-next-session-ab` --- ### 📝 Commits (6) - [`ddbb164`](https://github.com/kvcache-ai/sglang/commit/ddbb164b959387fa86ae6c3240b4cc76b2d66367) refactor: use single-slot GLM layerwise prefill - [`f4c760f`](https://github.com/kvcache-ai/sglang/commit/f4c760f46c892cc001b0fe76860e8057d746085f) perf(moe): overlap GLM FP8 layerwise transport - [`429286c`](https://github.com/kvcache-ai/sglang/commit/429286cdd5e29e86b492ce5a93487bc773664330) feat(glm5-next): support SM86 and SM89 consumer GPUs - [`487892c`](https://github.com/kvcache-ai/sglang/commit/487892c97857bd7a3410c705b1f0e9a7095cf298) feat(glm5-next): support multiple images per request - [`e4121b2`](https://github.com/kvcache-ai/sglang/commit/e4121b2b3abede048d76ea47af9ac3c3dcdb2e6a) fix(glm5-next): preserve multi-image boundaries - [`845f7ff`](https://github.com/kvcache-ai/sglang/commit/845f7ff76e79eb4383b7a84fc19d64e56380964c) Merge pull request #2 from yyj6666667/glm5-next-multi-image ### 📊 Changes **33 files changed** (+6147 additions, -2814 deletions) <details> <summary>View changed files</summary> ➕ `examples/chat_template/glm5_next_multimodal.jinja` (+28 -0) 📝 `python/sglang/srt/configs/glm5_next.py` (+56 -1) 📝 `python/sglang/srt/layers/attention/linear/kernels/glm5_next_kda_ops.py` (+33 -9) ➕ `python/sglang/srt/layers/attention/nsa/glm5_next_indexer_triton.py` (+386 -0) 📝 `python/sglang/srt/layers/attention/nsa/glm5_next_sparse_attention.py` (+37 -4) 📝 `python/sglang/srt/layers/attention/nsa/kpool_fp8_index.py` (+598 -2) 📝 `python/sglang/srt/layers/attention/nsa/kpool_plan.py` (+31 -1) 📝 `python/sglang/srt/layers/attention/nsa/nsa_indexer_kpool.py` (+112 -37) 📝 `python/sglang/srt/layers/attention/nsa_backend.py` (+43 -11) 📝 `python/sglang/srt/layers/moe/kt_ep_wrapper.py` (+1392 -1543) 📝 `python/sglang/srt/mem_cache/glm5_next_memory_pool.py` (+16 -0) 📝 `python/sglang/srt/mem_cache/memory_pool.py` (+40 -2) 📝 `python/sglang/srt/model_executor/model_runner.py` (+0 -19) 📝 `python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py` (+11 -7) 📝 `python/sglang/srt/multimodal/processors/glm5_next.py` (+255 -27) 📝 `python/sglang/srt/server_args.py` (+57 -34) ➕ `scripts/glm5_next_consumer_gpu_compile.py` (+636 -0) 📝 `scripts/glm5_next_session_c_final_acceptance.py` (+255 -117) 📝 `scripts/glm5_next_session_c_graph_acceptance.py` (+1 -1) 📝 `scripts/glm5_next_session_c_qa_gsm_graph_acceptance.py` (+3 -3) _...and 13 more files_ </details> ### 📄 Description ## Summary - add an exact GLM-5-Next FP8/TP4 single-GPU-slot prefetch controller - synchronously prime layer 3, then load layer n+1 on one persistent per-rank worker after the current-stream consumed event - overlap the successor native writer/H2D transport with postprocess and the next attention layer - keep exactly two Host expert slots and one full-layer GPU slot - fail closed on incomplete layer registry, stale ordering, pointer rebinding, rank-local failure, or poisoned transport The path is hard-gated to exact GLM-5-Next, block-FP8, TP=4, gpu_experts=0, raw Triton MoE. Legacy transport, MXFP4, BF16, and other models retain their existing paths. Depends on kvcache-ai/ktransformers#2160. ## Validation - focused GLM tests: 18 passed + 13 subtests - MXFP4 preservation tests: 27 passed - TP4 GPUs 0-3, CUDA Graph enabled, 30K prompt: - chunk 8192 hot TTFT: 33.637s vs legacy 41.229s (-18.4%) - chunk 16384 hot TTFT: 26.652s vs legacy 33.113s (-19.5%) - output token hashes match the corresponding legacy/native baselines exactly - natural 29,835-token Chinese prompt remained coherent: - chunk 8192: 854.6 tok/s prefill - chunk 16384: 1112.3 tok/s prefill - Qwen3-30B-A3B TP4 regression: 23/23 passed, including CUDA Graph batches 1/2/4 and deterministic top-256 checks This PR is submitted for review only and must not be auto-merged. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:07:02 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#80

No description provided.

Rows
Columns