[GH-ISSUE #1883] ARM CPUs that do not support AVX512 (NVIDIA GH200)? #1174

New issue

Open

opened 2026-10-29 14:49:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:49:34 +00:00

Owner

Originally created by @ehfd on GitHub (Mar 11, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1883

Reminder

I have read the above rules and searched the existing issues.

System Info

KTransformers: github.com/kvcache-ai/ktransformers@7a4b9b0e87
SGLang: v0.5.6.post1
Platform: Kubernetes (Ubuntu 24.04 Server)
Python Version: 3.12
CPU: NVIDIA GH200 Superchip
GPU: NVIDIA GH200 GPU

Reproduction

[2026-03-11 11:42:47] Generated KT GPU experts masks using 'uniform' strategy: 89 MoE layers (out of 92 total layers) x 160 experts, total GPU experts in MoE layers = 0
[2026-03-11 11:42:47] Using CompressedTensorsW8A8Fp8MoE
[2026-03-11 11:42:47[] [KT[] Created shared staging buffer: 80.0 MiB (shape=torch.Size([8192, 5120]), dtype=torch.bfloat16)
[2026-03-11 11:42:47] Scheduler hit an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 363, in __init__
    self.init_model_worker()
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 559, in init_model_worker
    self.init_tp_model_worker()
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 517, in init_tp_model_worker
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__
    self._init_model_runner()
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 415, in __init__
    self.initialize(min_per_gpu_memory)
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize
    self.load_model()
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model
    self.model = self.loader.load_model(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_loader/loader.py", line 671, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_loader/loader.py", line 277, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 1031, in __init__
    self.model = Glm4MoeModel(
                 ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 920, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/common.py", line 648, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/common.py", line 650, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 922, in <lambda>
    lambda idx, prefix: Glm4MoeDecoderLayer(
                        ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 741, in __init__
    self.mlp = Glm4MoeSparseMoeBlock(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 375, in __init__
    self.experts = get_moe_impl_class(quant_config)(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 283, in __init__
    self.quant_method.create_weights(
  File "/usr/local/lib/python3.12/dist-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2095, in create_weights
    self.wrapper = KTMoEWrapper(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/kt_kernel/experts.py", line 106, in __new__
    return backend_cls(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/kt_kernel/utils/amx.py", line 350, in __init__
    raise RuntimeError(
RuntimeError: FP8 backend not available. Required ISA:
  - AVX512F + AVX512BW + AVX512_BF16 + AVX512_VBMI
Please recompile kt_kernel_ext with AVX512 + BF16 + VBMI enabled.

[2026-03-11 11:42:47] Received sigquit from a child process. It usually means the child failed.

Others

Are we able to work around AVX512 + BF16 + VBMI, which is not present on ARM Neoverse?

The GH200 has a high-bandwidth CPU RAM - VRAM interconnect, which makes it the perfect choice for leveraging KTransformers.

Originally created by @ehfd on GitHub (Mar 11, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1883 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info KTransformers: https://github.com/kvcache-ai/ktransformers/commit/7a4b9b0e87467ef1a23c5dd1690d92ca30ca3e26 SGLang: v0.5.6.post1 Platform: Kubernetes (Ubuntu 24.04 Server) Python Version: 3.12 CPU: NVIDIA GH200 Superchip GPU: NVIDIA GH200 GPU ### Reproduction ```text [2026-03-11 11:42:47] Generated KT GPU experts masks using 'uniform' strategy: 89 MoE layers (out of 92 total layers) x 160 experts, total GPU experts in MoE layers = 0 [2026-03-11 11:42:47] Using CompressedTensorsW8A8Fp8MoE [2026-03-11 11:42:47[] [KT[] Created shared staging buffer: 80.0 MiB (shape=torch.Size([8192, 5120]), dtype=torch.bfloat16) [2026-03-11 11:42:47] Scheduler hit an exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 363, in __init__ self.init_model_worker() File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 559, in init_model_worker self.init_tp_model_worker() File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/scheduler.py", line 517, in init_tp_model_worker self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__ self._init_model_runner() File "/usr/local/lib/python3.12/dist-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 415, in __init__ self.initialize(min_per_gpu_memory) File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize self.load_model() File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_loader/loader.py", line 671, in load_model model = _initialize_model( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/model_loader/loader.py", line 277, in _initialize_model return model_class(**kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 1031, in __init__ self.model = Glm4MoeModel( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 920, in __init__ self.layers, self.start_layer, self.end_layer = make_layers( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/common.py", line 648, in make_layers + get_offloader().wrap_modules( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules return list(all_modules_generator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/utils/common.py", line 650, in <genexpr> layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 922, in <lambda> lambda idx, prefix: Glm4MoeDecoderLayer( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 741, in __init__ self.mlp = Glm4MoeSparseMoeBlock( ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/models/glm4_moe.py", line 375, in __init__ self.experts = get_moe_impl_class(quant_config)( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 283, in __init__ self.quant_method.create_weights( File "/usr/local/lib/python3.12/dist-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2095, in create_weights self.wrapper = KTMoEWrapper( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/kt_kernel/experts.py", line 106, in __new__ return backend_cls( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/kt_kernel/utils/amx.py", line 350, in __init__ raise RuntimeError( RuntimeError: FP8 backend not available. Required ISA: - AVX512F + AVX512BW + AVX512_BF16 + AVX512_VBMI Please recompile kt_kernel_ext with AVX512 + BF16 + VBMI enabled. [2026-03-11 11:42:47] Received sigquit from a child process. It usually means the child failed. ``` ### Others Are we able to work around `AVX512 + BF16 + VBMI`, which is not present on ARM Neoverse? The GH200 has a high-bandwidth CPU RAM - VRAM interconnect, which makes it the perfect choice for leveraging KTransformers.

bsakaguchi added the

enhancement

label

2026-10-29 14:49:34 +00:00

bsakaguchi commented

2026-10-29 14:49:35 +00:00

Author

Owner

@ErvinXie commented on GitHub (Mar 13, 2026):

Hi @ehfd,

Thanks for the interest in running KTransformers on GH200 — we agree its NVLink C2C high-bandwidth interconnect makes it an excellent platform for CPU-GPU hybrid inference.

However, the current kt-kernel CPU inference backend is built entirely on x86 instruction sets (AVX512, AVX2, AMX). ARM Neoverse uses a completely different ISA (NEON/SVE/SVE2), so this is not a matter of recompiling with different flags — it would require porting the core CPU compute kernels to ARM.

Current status

x86 only: kt-kernel supports AVX512 (primary) and AVX2 (community-contributed, see #1863).
ARM support: Not currently available. There is no short-term plan, but it's on our radar as ARM server platforms (GH200, Grace) become more prevalent.

#1863 — AVX2 support for older x86 CPUs (EPYC 7402)
#1882 — Feature request for Ascend + Kunpeng (another ARM-based platform)

We'll keep this issue open as a feature request for ARM platform support.

This reply was co-authored by an AI agent and a human.

@ErvinXie commented on GitHub (Mar 13, 2026): Hi @ehfd, Thanks for the interest in running KTransformers on GH200 — we agree its NVLink C2C high-bandwidth interconnect makes it an excellent platform for CPU-GPU hybrid inference. However, the current `kt-kernel` CPU inference backend is **built entirely on x86 instruction sets** (AVX512, AVX2, AMX). ARM Neoverse uses a completely different ISA (NEON/SVE/SVE2), so this is not a matter of recompiling with different flags — it would require porting the core CPU compute kernels to ARM. ### Current status - **x86 only**: kt-kernel supports AVX512 (primary) and AVX2 (community-contributed, see #1863). - **ARM support**: Not currently available. There is no short-term plan, but it's on our radar as ARM server platforms (GH200, Grace) become more prevalent. ### Related - #1863 — AVX2 support for older x86 CPUs (EPYC 7402) - #1882 — Feature request for Ascend + Kunpeng (another ARM-based platform) We'll keep this issue open as a feature request for ARM platform support. --- *This reply was co-authored by an AI agent and a human.*