mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1319] [Bug] #888
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#888
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @shenyan-008 on GitHub (May 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1319
检查清单
问题描述
ktransformers --model_path Qwen/Qwen3-30B-A3B --gguf_path ktransformers-0.3.1/Qwen3-30B-A3B --port 10002 --web True
no balance_serve
2025-05-18 14:21:05,607 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
Traceback (most recent call last):
File "/home/sy/miniconda3/envs/ktransformers2/bin/ktransformers", line 8, in
sys.exit(main())
^^^^^^
File "/home/sy/miniconda3/envs/ktransformers2/lib/python3.11/site-packages/ktransformers/server/main.py", line 109, in main
create_interface(config=cfg, default_args=cfg)
File "/home/sy/miniconda3/envs/ktransformers2/lib/python3.11/site-packages/ktransformers/server/utils/create_interface.py", line 30, in create_interface
GlobalInterface.interface = BackendInterface(default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sy/miniconda3/envs/ktransformers2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 49, in init
self.model = custom_modelsconfig.architectures[0]
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'Qwen3MoeForCausalLM'
复现步骤
sudo apt-get update
sudo apt-get install build-essential cmake ninja-build patchelf
conda create --name ktransformers python=3.11
conda activate ktransformers # you may need to run ‘conda init’ and reopen shell first
conda install -c conda-forge libstdcxx-ng # Anaconda provides a package called
libstdcxx-ngthat includes a newer version oflibstdc++, which can be installed viaconda-forge.strings ~/anaconda3/envs/ktransformers/lib/libstdc++.so.6 | grep GLIBCXX
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip3 install packaging ninja cpufeature numpy
sudo apt-get install libnuma-dev
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
Make sure your system has dual sockets and double size RAM than the model's size (e.g. 1T RAM for 512G model)
sudo apt install libnuma-dev
export USE_NUMA=1
bash install.sh # or #make dev_install
ktransformers --model_path Qwen/Qwen3-30B-A3B --gguf_path ktransformers-0.3.1/Qwen3-30B-A3B --port 10002 --web True
环境信息
WSL2 Ubuntu-22.04
Cuda 12.4
torch
Version: 2.6.0+cu124
Name: torchvision
Version: 0.21.0+cu124
Name: flash_attn
Version: 2.7.4.post1
GPU:RTX3080 10GB
@Pitaya618 commented on GitHub (May 18, 2025):
我也遇到了同样的问题。
I also encountered the same problem.
WSL2 Ubuntu-22.04
Version: torch-2.7.0+cu129
GPU：RTX5090 32GB
@kuku-jiusgan commented on GitHub (May 19, 2025):
启动命令的问题吧，--backend_type balance_serve 这个就没问题，但是用单发的命令就报错
@qiuqiu10 commented on GitHub (May 19, 2025):
qwen3目前用balance-serve后端吧，ktransformers后端还没支持
@Pitaya618 commented on GitHub (May 19, 2025):
我尝试使用--backend_type balance_serve后出现了关于sched_ext模块缺失的问题，目前根据另一个issue中的步骤尝试修改了部分代码后出现了新的问题：
目前不知道如何解决。
@xiaoxin112211 commented on GitHub (May 24, 2025):
envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py
from ktransformers.models.modeling_qwen3_moe import Qwen3MoeForCausalLM
custom_models = {
"DeepseekV2ForCausalLM": DeepseekV2ForCausalLM,
"DeepseekV3ForCausalLM": DeepseekV3ForCausalLM,
"Qwen3MoeForCausalLM": Qwen3MoeForCausalLM, #add model
"Qwen2MoeForCausalLM": Qwen2MoeForCausalLM,
"LlamaForCausalLM": LlamaForCausalLM,
"MixtralForCausalLM": MixtralForCausalLM,
}