mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1501] [Bug] in support-qwen3next branch, transformers version has not high #1004
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1004
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @PPXGS on GitHub (Sep 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1501
检查清单
问题描述
when build ktransformers in support-qwen3next, it is successful. But the transformers version can not match the qwen-3 next models' version.
复现步骤
Afer building, the ktransformers version is 0.3.2+cu124torch26fancy, and transformers version is 4.51.3. but Qwen3-Next-80B-A3B-Instruct need transformers version is "transformers_version": "4.57.0.dev0".
when I run
python ktransformers/server/main.py \ --port 10021 \ --model_path /localnvme/application/common/models/Qwen/Qwen3-Next-80B-A3B-Instruct \ --model_name Qwen3NextForCausalLM \ --optimize_config_path /localnvme/application/zhangzn/ktransformers_v0.3.2/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Next-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --no-use_cuda_graph \ --backend_type balance_serveI get the error:
ImportError: cannot import name 'layer_type_validation' from 'transformers.configuration_utils' (/localnvme/application/zhangzn/anaconda3/envs/ktransformers_support-qwen3next/lib/python3.11/site-packages/transformers/configuration_utils.py)
环境信息
ktransformers 0.3.2+cu124torch26fancy
transformers 4.51.3
cuda 12.4
python 3.11
Ubuntu 20.04
GPU NVIDIA A800 ×8
@CYSTEV-chn commented on GitHub (Sep 16, 2025):
pip uninstall transformers -y
pip install git+https://github.com/huggingface/transformers.git
@mj520 commented on GitHub (Sep 22, 2025):
python ktransformers/server/main.py --port 10021 --model_path /root/Qwen.Qwen3-Next-80B-A3B-Instruct-FP8-Dynamic --gguf_path /root/Qwen.Qwen3-Next-80B-A3B-Instruct-FP8-Dynamic --model_name Qwen3NextForCausalLM --backend_type balance_serve
W0922 09:30:59.383000 462160 site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0922 09:30:59.383000 462160 site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
2025-09-22 09:30:59,385 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
flash_attn not found, flashinfer unit test needed it. If you are using balance serve, ignore this.
set start method
Connected to server at tcp://localhost:37793
W0922 09:31:07.075000 462268 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0922 09:31:07.075000 462268 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
2025-09-22 09:31:07,078 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
flash_attn not found, flashinfer unit test needed it. If you are using balance serve, ignore this.
start method already set to spawn
Connected to server at tcp://localhost:37793
args.architectures: Qwen3NextForCausalLM
The argument
trust_remote_codeis to be used with Auto classes. It has no effect here and is ignored.The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Injecting model as default
Injecting model.embed_tokens as default
......
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.balance_serve_attention . KQwen3NextAttention
Injecting model.layers.47.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3NextRMSNorm
Injecting model.layers.47.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3NextRMSNorm
Injecting model.layers.47.mlp as ktransformers.operators.experts . KQwen3NextSparseMoeBlockV2
Injecting model.layers.47.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.47.mlp.shared_expert as ktransformers.operators.mlp . KQwen2MoeMLP
Injecting model.layers.47.mlp.shared_expert.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_expert.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_expert.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_expert.act_fn as default
Injecting model.layers.47.mlp.shared_expert_gate as default
Injecting model.layers.47.input_layernorm as ktransformers.operators.layernorm . KQwen3NextRMSNorm
Injecting model.layers.47.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3NextRMSNorm
Injecting model.norm as ktransformers.operators.layernorm . KQwen3NextRMSNorm
Injecting model.rotary_emb as ktransformers.operators.RoPE . KQwen3MoeRotaryEmbedding
Injecting cache as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading model.embed_tokens.weight to cpu
loading model.layers.0.linear_attn.dt_bias to cuda
loading model.layers.0.linear_attn.A_log to cuda
loading model.layers.0.linear_attn.conv1d.weight to cuda:0
Process SpawnProcess-1:
Traceback (most recent call last):
File "/root/anaconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/root/anaconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 308, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 212, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, weights_loader, device=default_device)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
[Previous line repeated 1 more time]
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 944, in load
self.generate_linear.load(w=w)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 653, in load
marlin_q_w, marlin_s, g_idx, sort_indices, _ = marlin_quantize(
^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/ktransformers_ext/operators/custom_marlin/quantize/utils/marlin_utils.py", line 93, in marlin_quantize
q_w, s, g_idx, rand_perm = quantize_weights(w, num_bits, group_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/ktransformers_ext/operators/custom_marlin/quantize/utils/quant_utils.py", line 61, in quantize_weights
s = torch.max(torch.abs(w), 0, keepdim=True)[0]
^^^^^^^^^^^^
NotImplementedError: "abs_cuda" not implemented for 'Float8_e4m3fn'
@CYSTEV-chn commented on GitHub (Sep 22, 2025):
那就不知道了，只知道用dev版transform依赖可以加载，其他低版本的transform我也试过，连正常加载都做不到。
@harveyff commented on GitHub (Sep 29, 2025):
你先可以跑了吗？我用原版qwen3-next是可以跑的。但 https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 版本报错
@ligang0357-glitch commented on GitHub (Sep 30, 2025):
@harveyff 原版的跑的很慢基本不动，你那边可以正常跑吗？