[GH-ISSUE #2244] [Feature discussion] Full GLM-5.3 streamed FP8 GPU prefill on AVX2 / 2x32GiB SM120 #1309

New issue

Open

opened 2026-10-29 14:51:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:51:06 +00:00

Owner

Originally created by @f-wunderlich on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2244

Proposal: opt-in native FP8 streamed expert prefill for full GLM-5.3 on AVX2 hosts

We have deployed full zai-org/GLM-5.3, not GLM-5.3-Flash, on two EPYC 7443
CPUs (48 physical cores, AVX2, two NUMA nodes), 1 TiB RAM and two RTX PRO 4500
Blackwell GPUs (32 GiB each). Native FP8 CPU expert decode is usable for
non-interactive quality evaluation, but CPU expert prefill was approximately
3.6 input tokens/s. A small streamed FP8 GPU adapter made evaluation practical.

Would you be interested in integrating this as an opt-in feature, and should
the Python serving integration target kvcache-ai/sglang or sgl-project/sglang?
The former has issues disabled, hence this discussion here. We would like to
coordinate with existing work rather than maintain a separate model-specific
serving implementation.

Reference implementation and tests

https://gist.github.com/f-wunderlich/8eb87d678c20d05b87288c1396a31157

The gist contains the adapter, its prototype integration patch, seventeen
unit/control-flow tests and synthetic/real-checkpoint probes. The integration
patch unconditionally selects a model-specific wrapper in our dedicated image;
that is not proposed as merge-ready integration. An upstream version should
be explicitly opt-in and retain the existing path for other models/modes.

Runtime: SGLang revision 1aa0e962b206102b7c439a4a0c4981cfec6e87bc and KT revision
95009ea6856c0799e517e93cb12be5e8494bc7ce, with the existing SM120/AVX2 integration
patches from 05yuki/dsv41-flash-dual-epyc revision
6c98c9e1b84e5eb7a7d0e126dc6c2050ce174447. This is a pinned, patched runtime,
not a claim that the same patch applies unchanged to current main. The model
revision is aca966e4e02791568aa6a4ced368624b3d897f42.

Design

Subclass KTEPWrapperMethod; preserve native CPU decode, upstream attention,
shared experts, routed scaling, TP reduction and radix-cache ownership.
TP2, 48 CPU workers/two NUMA pools, zero resident or deferred GPU experts.
Only TP rank zero consumes one reusable 9.002 GiB pinned host/GPU layer slot.
Rank one contributes zero routed-expert output before the existing TP reduction.
Existing native submit_write_weight_scale_to_buffer reconstructs all 256
experts' gate/up/down FP8 weights and scales from CPU NUMA arenas. No additional
weight quantization or native extension rebuild was needed.
Complete native export before H2D. Synchronize the current CUDA stream before
reusing either slot, draining readers, transfers and delayed CPU callbacks.
No other TP rank reads these slot pointers. Fail closed after staging failure.
Reserve the slot before KV profiling. Use 8K serving chunks but tile expert
arithmetic at <=4K rows without repeating weight transfer.
Prefill >=256 tokens uses GPU W8A8; shorter continuations and decode use native
CPU W8A16. Full GLM uses unclamped SiLU, not Flash's clamp of ten.
Explicitly reject unsupported dimensions/activation/quantization, moved
experts, scheduling overlap, CUDA capture and decode batches at the threshold.

Measured results

Client-side streaming timings, including export/H2D and serving overhead:

Prompt tokens	Cached tokens	First token	Fresh input throughput
4,096	0	66.42 s	61.7 tok/s
4,096	4,032	18.19 s	-
16,384	0	139.17 s	117.7 tok/s
16,384	16,320	18.25 s	-

Decode remained approximately 3 tokens/s. These are not identical-length
CPU/GPU A/B samples: the historical CPU result used smaller fresh prompts.
Isolated real-layer A/B at 4K was 15.93 s CPU vs 0.96 s GPU including export/H2D;
the tiled 8K case was 32.60 s vs 2.44 s including first-call overhead.

Synthetic repeated-slot export matched every weight/scale byte. Independent
real-checkpoint checks matched gate/up/down weights/scales for experts
0, 127 and 255 of layer 3. GPU W8A8 output had approximately 3.8% relative mean
absolute error versus CPU W8A16 in random real-layer probes; this is not
bit-equivalence or a comprehensive numerical-quality qualification.

Fresh/cached reference-document replies and a tool-call/result LDAP task passed
bounded behavioral checks, including preserving the object class and other
attributes while ignoring tool-result injection. A subsequent user-run
overnight autonomous evaluation produced good results, according to the user;
this is anecdotal task evidence, not a public benchmark score.

GPU0 memory remains tight: PyTorch recovered from an allocation failure during
the second 8K chunk of a 16K prompt. The request and cache replay completed;
there was no fatal OOM or pod restart. 128K is configured/allocated but was not
qualified at full occupancy. Scheduling overlap, graphs, other TP sizes,
models and concurrent workloads are not qualified by this adapter.

Related work:

kvcache-ai/sglang#80: exact GLM-5-Next FP8/TP4 transport overlap.
kvcache-ai/sglang#98: generic full-GPU prefill slot memory reservation.
#2226 / kvcache-ai/sglang#102: TP staging-reader lifetime correctness.

Our contribution is the full-GLM AVX2/SM120 deployment path, bounded workspace
tiling and validation around it. We would appreciate guidance on whether to
extend the existing generic path or contribute a separate guarded transport.

Originally created by @f-wunderlich on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2244 ## Proposal: opt-in native FP8 streamed expert prefill for full GLM-5.3 on AVX2 hosts We have deployed **full `zai-org/GLM-5.3`**, not GLM-5.3-Flash, on two EPYC 7443 CPUs (48 physical cores, AVX2, two NUMA nodes), 1 TiB RAM and two RTX PRO 4500 Blackwell GPUs (32 GiB each). Native FP8 CPU expert decode is usable for non-interactive quality evaluation, but CPU expert prefill was approximately 3.6 input tokens/s. A small streamed FP8 GPU adapter made evaluation practical. Would you be interested in integrating this as an opt-in feature, and should the Python serving integration target `kvcache-ai/sglang` or `sgl-project/sglang`? The former has issues disabled, hence this discussion here. We would like to coordinate with existing work rather than maintain a separate model-specific serving implementation. ### Reference implementation and tests https://gist.github.com/f-wunderlich/8eb87d678c20d05b87288c1396a31157 The gist contains the adapter, its prototype integration patch, seventeen unit/control-flow tests and synthetic/real-checkpoint probes. The integration patch unconditionally selects a model-specific wrapper in our dedicated image; **that is not proposed as merge-ready integration**. An upstream version should be explicitly opt-in and retain the existing path for other models/modes. Runtime: SGLang revision `1aa0e962b206102b7c439a4a0c4981cfec6e87bc` and KT revision `95009ea6856c0799e517e93cb12be5e8494bc7ce`, with the existing SM120/AVX2 integration patches from `05yuki/dsv41-flash-dual-epyc` revision `6c98c9e1b84e5eb7a7d0e126dc6c2050ce174447`. This is a pinned, patched runtime, not a claim that the same patch applies unchanged to current main. The model revision is `aca966e4e02791568aa6a4ced368624b3d897f42`. ### Design - Subclass `KTEPWrapperMethod`; preserve native CPU decode, upstream attention, shared experts, routed scaling, TP reduction and radix-cache ownership. - TP2, 48 CPU workers/two NUMA pools, zero resident or deferred GPU experts. - Only TP rank zero consumes one reusable 9.002 GiB pinned host/GPU layer slot. Rank one contributes zero routed-expert output before the existing TP reduction. - Existing native `submit_write_weight_scale_to_buffer` reconstructs all 256 experts' gate/up/down FP8 weights and scales from CPU NUMA arenas. No additional weight quantization or native extension rebuild was needed. - Complete native export before H2D. Synchronize the current CUDA stream before reusing either slot, draining readers, transfers and delayed CPU callbacks. No other TP rank reads these slot pointers. Fail closed after staging failure. - Reserve the slot before KV profiling. Use 8K serving chunks but tile expert arithmetic at <=4K rows without repeating weight transfer. - Prefill >=256 tokens uses GPU W8A8; shorter continuations and decode use native CPU W8A16. Full GLM uses **unclamped SiLU**, not Flash's clamp of ten. - Explicitly reject unsupported dimensions/activation/quantization, moved experts, scheduling overlap, CUDA capture and decode batches at the threshold. ### Measured results Client-side streaming timings, including export/H2D and serving overhead: | Prompt tokens | Cached tokens | First token | Fresh input throughput | | --- | ---: | ---: | ---: | | 4,096 | 0 | 66.42 s | 61.7 tok/s | | 4,096 | 4,032 | 18.19 s | - | | 16,384 | 0 | 139.17 s | 117.7 tok/s | | 16,384 | 16,320 | 18.25 s | - | Decode remained approximately 3 tokens/s. These are not identical-length CPU/GPU A/B samples: the historical CPU result used smaller fresh prompts. Isolated real-layer A/B at 4K was 15.93 s CPU vs 0.96 s GPU including export/H2D; the tiled 8K case was 32.60 s vs 2.44 s including first-call overhead. Synthetic repeated-slot export matched every weight/scale byte. Independent real-checkpoint checks matched gate/up/down weights/scales for experts 0, 127 and 255 of layer 3. GPU W8A8 output had approximately 3.8% relative mean absolute error versus CPU W8A16 in random real-layer probes; this is **not** bit-equivalence or a comprehensive numerical-quality qualification. Fresh/cached reference-document replies and a tool-call/result LDAP task passed bounded behavioral checks, including preserving the object class and other attributes while ignoring tool-result injection. A subsequent user-run overnight autonomous evaluation produced good results, according to the user; this is anecdotal task evidence, not a public benchmark score. ### Limits and related work GPU0 memory remains tight: PyTorch recovered from an allocation failure during the second 8K chunk of a 16K prompt. The request and cache replay completed; there was no fatal OOM or pod restart. 128K is configured/allocated but was not qualified at full occupancy. Scheduling overlap, graphs, other TP sizes, models and concurrent workloads are not qualified by this adapter. Related work: - kvcache-ai/sglang#80: exact GLM-5-Next FP8/TP4 transport overlap. - kvcache-ai/sglang#98: generic full-GPU prefill slot memory reservation. - #2226 / kvcache-ai/sglang#102: TP staging-reader lifetime correctness. Our contribution is the full-GLM AVX2/SM120 deployment path, bounded workspace tiling and validation around it. We would appreciate guidance on whether to extend the existing generic path or contribute a separate guarded transport.