[GH-ISSUE #2245] [Performance] Exact AVX2 FP8 arithmetic conversion avoids costly LUT gathers #1311

New issue

Open

opened 2026-10-29 14:51:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:51:07 +00:00

Owner

Originally created by @f-wunderlich on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2245

Exact AVX2 FP8 conversion optimization

While profiling full GLM-5.3 on dual EPYC 7443 (Zen 3 / AVX2), we found a
substantial CPU cost in the vector E4M3 lookup-table conversion. We have a small
candidate patch replacing gathers with exact arithmetic:

https://gist.github.com/f-wunderlich/118bffc7de22aa001d9852947de162a3

This includes the patch and an exhaustive conversion test. It applies to
kt-kernel/operators/avx2/fp8_dequant.hpp; the current main branch still has the
gather implementation. Is this an optimization you would accept as a PR, and
are there additional target architectures or kernel tests you would want first?

Semantics and scope

Only fp8x8_to_fp32x8 changes. FP8 checkpoint bytes, block scales, BF16 inputs,
FMA/reduction order and scalar tails are unchanged.

Normal E4M3 values use mantissa/exponent bit arithmetic with adjusted FP32 bias.
Subnormals are integer mantissas times 2^-9, exactly representable in FP32.
Preserve signed zeros and the existing NaN-to-signed-zero policy.
No table gather or function-static LUT initialization check in the vector path.
This is not additional weight quantization or approximate conversion.

The C++ gate checks all 256 encodings in each vector lane, with mixed vectors
at all 32 byte offsets, comparing FP32 bits against the existing scalar LUT.
It passed during the matched native-extension build.

Real-checkpoint measurements

Two EPYC 7443 CPUs, 48 native workers in two NUMA pools, full GLM-5.3 native FP8.
Each probe loads one real layer and uses deterministic random BF16 inputs,
top-eight expert routes and normalized routing weights. Tiny CUDA staging buffers
are on the second GPU; no expert GPU slot is used by the probe. There is no
concurrent serving request during these measurements.

Layer / input rows	Original median	Arithmetic median	Full output hashes
3 / 1	4.015 ms	3.117 ms	identical
37 / 1	4.015 ms	3.082 ms	identical
77 / 1	4.006 ms	3.063 ms	identical
3 / 64	233.358 ms	113.202 ms	identical

Single-row probes use 32 warmups and 128 timed calls; the 64-row case uses
32 warmups and 32 timed calls. Hashes cover complete BF16 output tensors across
all timed calls, not just sample values or output sums. This gives approximately
23% lower single-token layer time and approximately 2x faster 64-token CPU tails.
It is not a claim of 2x whole-model decode.

A simplified AVX2 dot-product microbenchmark was approximately 3x faster on this
CPU, but memory traffic and framework overhead limit real-layer gains. We are
qualifying the candidate in whole-model serving separately.

Profile and revisions

A 60-second perf record -F 99 --call-graph fp of both serving TP ranks sampled
76.3% of CPU cycles in the KT extension. Four vgatherdps instructions in the
FP8 GEMM inner loop accounted for approximately 53% of sampled cycles. Aggregate
perf stat showed 3.54 GHz effective frequency; DRAM-event counts were similar
between sockets. This is CPU-cycle evidence, not an operator wall-time breakdown.

The extension is built from KT 95009ea6856c0799e517e93cb12be5e8494bc7ce plus
the matched SM120/AVX2 integration patch from 05yuki/dsv41-flash-dual-epyc
6c98c9e1b84e5eb7a7d0e126dc6c2050ce174447. The only candidate-vs-baseline source
change in that extension is the vector conversion patch. Native loader Python
wrappers are preserved. The checkpoint revision is
zai-org/GLM-5.3@aca966e4e02791568aa6a4ced368624b3d897f42.

Caveats: no cross-CPU-family performance qualification yet, no MXFP8 MoE runtime
qualification, and this does not change the scalar LUT's existing initialization
behavior. Related full-GLM deployment discussion: #2244.

Originally created by @f-wunderlich on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2245 ## Exact AVX2 FP8 conversion optimization While profiling full GLM-5.3 on dual EPYC 7443 (Zen 3 / AVX2), we found a substantial CPU cost in the vector E4M3 lookup-table conversion. We have a small candidate patch replacing gathers with exact arithmetic: https://gist.github.com/f-wunderlich/118bffc7de22aa001d9852947de162a3 This includes the patch and an exhaustive conversion test. It applies to `kt-kernel/operators/avx2/fp8_dequant.hpp`; the current main branch still has the gather implementation. Is this an optimization you would accept as a PR, and are there additional target architectures or kernel tests you would want first? ### Semantics and scope Only `fp8x8_to_fp32x8` changes. FP8 checkpoint bytes, block scales, BF16 inputs, FMA/reduction order and scalar tails are unchanged. - Normal E4M3 values use mantissa/exponent bit arithmetic with adjusted FP32 bias. - Subnormals are integer mantissas times 2^-9, exactly representable in FP32. - Preserve signed zeros and the existing NaN-to-signed-zero policy. - No table gather or function-static LUT initialization check in the vector path. - This is not additional weight quantization or approximate conversion. The C++ gate checks **all 256 encodings in each vector lane**, with mixed vectors at **all 32 byte offsets**, comparing FP32 bits against the existing scalar LUT. It passed during the matched native-extension build. ### Real-checkpoint measurements Two EPYC 7443 CPUs, 48 native workers in two NUMA pools, full GLM-5.3 native FP8. Each probe loads one real layer and uses deterministic random BF16 inputs, top-eight expert routes and normalized routing weights. Tiny CUDA staging buffers are on the second GPU; no expert GPU slot is used by the probe. There is no concurrent serving request during these measurements. | Layer / input rows | Original median | Arithmetic median | Full output hashes | | --- | ---: | ---: | --- | | 3 / 1 | 4.015 ms | 3.117 ms | identical | | 37 / 1 | 4.015 ms | 3.082 ms | identical | | 77 / 1 | 4.006 ms | 3.063 ms | identical | | 3 / 64 | 233.358 ms | 113.202 ms | identical | Single-row probes use 32 warmups and 128 timed calls; the 64-row case uses 32 warmups and 32 timed calls. Hashes cover complete BF16 output tensors across all timed calls, not just sample values or output sums. This gives approximately 23% lower single-token layer time and approximately 2x faster 64-token CPU tails. It is **not** a claim of 2x whole-model decode. A simplified AVX2 dot-product microbenchmark was approximately 3x faster on this CPU, but memory traffic and framework overhead limit real-layer gains. We are qualifying the candidate in whole-model serving separately. ### Profile and revisions A 60-second `perf record -F 99 --call-graph fp` of both serving TP ranks sampled 76.3% of CPU cycles in the KT extension. Four `vgatherdps` instructions in the FP8 GEMM inner loop accounted for approximately 53% of sampled cycles. Aggregate `perf stat` showed 3.54 GHz effective frequency; DRAM-event counts were similar between sockets. This is CPU-cycle evidence, not an operator wall-time breakdown. The extension is built from KT `95009ea6856c0799e517e93cb12be5e8494bc7ce` plus the matched SM120/AVX2 integration patch from `05yuki/dsv41-flash-dual-epyc` `6c98c9e1b84e5eb7a7d0e126dc6c2050ce174447`. The only candidate-vs-baseline source change in that extension is the vector conversion patch. Native loader Python wrappers are preserved. The checkpoint revision is `zai-org/GLM-5.3@aca966e4e02791568aa6a4ced368624b3d897f42`. Caveats: no cross-CPU-family performance qualification yet, no MXFP8 MoE runtime qualification, and this does not change the scalar LUT's existing initialization behavior. Related full-GLM deployment discussion: #2244.

bsakaguchi commented

2026-10-29 14:51:08 +00:00

Author

Owner

@f-wunderlich commented on GitHub (Oct 12, 2026):

Whole-model qualification is complete. The important distinction is that the
measured real-layer gain does not translate into a demonstrated serving decode
gain on our current non-graph stack.

All three tool-call/result, fresh-history and cached-history LDAP behavioral
gates passed, including preserving the class/other attributes and ignoring
tool-result injection.
All three complete 256-token, temperature-zero, max-effort reasoning streams
matched the old reference hash.
CPU-prefilled 225-token tool request: 63.61 -> 30.38 seconds to first token.
Tool continuation with 145 uncached tokens: 40.90 -> 18.74 seconds.
4K/16K cached replays with a 64-token CPU tail: 18.2 -> 8.9 seconds.
Large GPU prefill remained approximately unchanged: 67 seconds at 4K, 141
seconds at 16K. This converter change does not alter GPU expert arithmetic.
Fixed-output serving decode was 2.869/2.889/2.888 tok/s before versus
2.817/2.808/2.870 afterward: no whole-model decode acceleration demonstrated.

The CPU profile is an aggregate cycle profile, not a critical-path wall-time
timeline. Separate model reloads also changed memory layout/THP coverage; the
small serving decode-rate difference cannot confidently be assigned to the patch.
Short CPU prefill/continuation acceleration is the supported deployment benefit.

The native A/B probe is now included in the linked gist. The conversion mapping
gate also passed with -ffast-math, matching the native release build's setting.
No model precision, reasoning defaults, thread counts or serving scheduling flags
were changed for the rollout.

The 16K request recovered from the same allocator retry seen in the earlier
configuration and completed correctly with no pod restart. 128K remains configured,
not qualified at full occupancy. The image is left running for user evaluation;
decode execution-timeline/graph qualification is a separate next experiment.

@f-wunderlich commented on GitHub (Oct 12, 2026): Whole-model qualification is complete. The important distinction is that the measured real-layer gain does **not** translate into a demonstrated serving decode gain on our current non-graph stack. - All three tool-call/result, fresh-history and cached-history LDAP behavioral gates passed, including preserving the class/other attributes and ignoring tool-result injection. - All three complete 256-token, temperature-zero, `max`-effort reasoning streams matched the old reference hash. - CPU-prefilled 225-token tool request: **63.61 -> 30.38 seconds** to first token. - Tool continuation with 145 uncached tokens: **40.90 -> 18.74 seconds**. - 4K/16K cached replays with a 64-token CPU tail: **18.2 -> 8.9 seconds**. - Large GPU prefill remained approximately unchanged: **67 seconds at 4K**, **141 seconds at 16K**. This converter change does not alter GPU expert arithmetic. - Fixed-output serving decode was **2.869/2.889/2.888 tok/s before** versus **2.817/2.808/2.870 afterward**: no whole-model decode acceleration demonstrated. The CPU profile is an aggregate cycle profile, not a critical-path wall-time timeline. Separate model reloads also changed memory layout/THP coverage; the small serving decode-rate difference cannot confidently be assigned to the patch. Short CPU prefill/continuation acceleration is the supported deployment benefit. The native A/B probe is now included in the linked gist. The conversion mapping gate also passed with `-ffast-math`, matching the native release build's setting. No model precision, reasoning defaults, thread counts or serving scheduling flags were changed for the rollout. The 16K request recovered from the same allocator retry seen in the earlier configuration and completed correctly with no pod restart. 128K remains configured, not qualified at full occupancy. The image is left running for user evaluation; decode execution-timeline/graph qualification is a separate next experiment.