[GH-ISSUE #1692] DeepSeek v3.2 type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5') #1083

New issue

Closed

opened 2026-10-29 14:48:18 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:48:18 +00:00

Owner

Originally created by @mrgaolei on GitHub (Dec 9, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1692

Reminder

I have read the above rules and searched the existing issues.

System Info

CPU 6416H
RAM 768G
GPU 3090
Ubuntu 24.04
kt-kernel 0.4.3

Reproduction

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model /opt/ai-models/v3-2 \
  --kt-weight-path /opt/ai-models/v3-2-CPU \
  --kt-cpuinfer 35 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 0 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 2048 \
  --max-running-requests 8 \
  --max-total-tokens 20000 \
  --served-model-name DeepSeek-V3.2 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --kt-method AMXINT4

    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2691, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1390, in forward
    s = self.forward_prepare(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1437, in forward_prepare
    inner_state = self.forward_absorb_prepare(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1595, in forward_absorb_prepare
    fused_qkv_a_proj_out = self.fused_qkv_a_proj_with_mqa(hidden_states)[0]
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 254, in forward
    output = self.quant_method.apply(self, x, bias)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8.py", line 491, in apply
    return self.w8a8_block_fp8_linear(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8_utils.py", line 293, in triton_w8a8_block_fp8_linear
    q_input, x_scale = per_token_group_quant_fp8(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 292, in _per_token_group_quant_8bit_raw
    _per_token_group_quant_8bit[(M,)](
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/runtime/jit.py", line 594, in run
    kernel = self.compile(src, target=target, options=options.__dict__)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/compiler/compiler.py", line 339, in compile
    module = src.make_ir(options, codegen_fns, module_map, context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/compiler/compiler.py", line 83, in make_ir
    return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
triton.compiler.errors.CompilationError: at 1:0:
def _per_token_group_quant_8bit(
^
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")

[2025-12-09 23:54:06] Received sigquit from a child process. It usually means the child failed.

Others

Is kt-kernel does not support RTX 30xx GPU?

Originally created by @mrgaolei on GitHub (Dec 9, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1692 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info CPU 6416H RAM 768G GPU 3090 Ubuntu 24.04 kt-kernel 0.4.3 ### Reproduction ```bash python -m sglang.launch_server \ --host 0.0.0.0 \ --port 30000 \ --model /opt/ai-models/v3-2 \ --kt-weight-path /opt/ai-models/v3-2-CPU \ --kt-cpuinfer 35 \ --kt-threadpool-count 1 \ --kt-num-gpu-experts 0 \ --attention-backend triton \ --trust-remote-code \ --mem-fraction-static 0.98 \ --chunked-prefill-size 2048 \ --max-running-requests 8 \ --max-total-tokens 20000 \ --served-model-name DeepSeek-V3.2 \ --enable-mixed-chunk \ --tensor-parallel-size 1 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --kt-method AMXINT4 ``` ``` return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 2691, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1390, in forward s = self.forward_prepare( ^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1437, in forward_prepare inner_state = self.forward_absorb_prepare( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/models/deepseek_v2.py", line 1595, in forward_absorb_prepare fused_qkv_a_proj_out = self.fused_qkv_a_proj_with_mqa(hidden_states)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/linear.py", line 254, in forward output = self.quant_method.apply(self, x, bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8.py", line 491, in apply return self.w8a8_block_fp8_linear( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8_utils.py", line 293, in triton_w8a8_block_fp8_linear q_input, x_scale = per_token_group_quant_fp8( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 292, in _per_token_group_quant_8bit_raw _per_token_group_quant_8bit[(M,)]( File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/runtime/jit.py", line 390, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/runtime/jit.py", line 594, in run kernel = self.compile(src, target=target, options=options.__dict__) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/compiler/compiler.py", line 339, in compile module = src.make_ir(options, codegen_fns, module_map, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/anaconda3/envs/kt-kernel/lib/python3.11/site-packages/triton/compiler/compiler.py", line 83, in make_ir return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ triton.compiler.errors.CompilationError: at 1:0: def _per_token_group_quant_8bit( ^ ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')") [2025-12-09 23:54:06] Received sigquit from a child process. It usually means the child failed. ``` ### Others Is kt-kernel does not support RTX 30xx GPU?

bsakaguchi

2026-10-29 14:48:18 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 14:48:19 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 10, 2025):

Maybe this is because that 3090 does not support fp8? The GPU part is supported by SGLang, you can check sglang for more information about running fp8 on 3090.

@ErvinXie commented on GitHub (Dec 10, 2025): Maybe this is because that 3090 does not support fp8? The GPU part is supported by SGLang, you can check sglang for more information about running fp8 on 3090.