kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1264] [Bug] 5090D CUDA12.8 torch-nightly - AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? #857

New issue

Open

opened 2026-10-29 14:45:26 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 14:45:26 +00:00

Owner

Originally created by @forhonourlx on GitHub (May 6, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1264

Originally assigned to: @Alisehen on GitHub.

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

5090D试了一天，unsloth/Qwen3-235B-A22B-128K-GGUF/Q4_K_M 成功加载了，一旦运行开唠就报错了，请大佬们棒棒忙，提前感谢。

Chat: 你是什么模型？
Traceback (most recent call last):
  File "/root/ktransformers/ktransformers/util/utils.py", line 206, in prefill_and_generate
    model._get_logits_warper(generation_config,device=inputs.device)
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1944, in __getattr__
    raise AttributeError(
AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ktransformers/ktransformers/local_chat.py", line 185, in <module>
    fire.Fire(local_chat)
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/util/utils.py", line 210, in prefill_and_generate
    model._get_logits_warper(generation_config)
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1944, in __getattr__
    raise AttributeError(
AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?

复现步骤

GGUF files: Q4_K_M https://www.modelscope.cn/models/unsloth/Qwen3-235B-A22B-128K-GGUF/files

安装命令步骤供参考：

wsl --install Ubuntu-22.04

wsl

sudo -i

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-wsl-ubuntu-12-8-local_12.8.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-12-8-local_12.8.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

apt-get install nvidia-utils-570

#导入环境变量
export CUDA_HOME=/usr/local/cuda
export CUDA_PATH=/usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
export PATH=/usr/local/cuda/bin:$PATH
#刷新
. ~/.bashrc

#安装 gcc g++ cmake ninja
sudo apt-get update
sudo apt-get install gcc g++ cmake ninja-build

#安装conda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
#回车 确定 回车 yes

bash

conda create --name kt python=3.12
conda activate kt

#安装python包
pip install packaging ninja cpufeature numpy -i https://pypi.tuna.tsinghua.edu.cn/simple

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128

#更新libstdc++
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install --only-upgrade libstdc++6


#编译安装flash-attention  !!!好久好久
pip install flash-attn --no-build-isolation -i https://pypi.tuna.tsinghua.edu.cn/simple

#源代码安装ktransformers
git clone https://github.com/kvcache-ai/ktransformers.git

cd ktransformers

git submodule update --init --recursive


#安装web
cd ktransformers
sudo apt-get update -y && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/nodesource.gpg
sudo chmod 644 /usr/share/keyrings/nodesource.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_23.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
sudo apt-get update -y
sudo apt-get install nodejs -y

cd ktransformers/website
npm install @vue/cli
npm run build
cd ../../


#编译ktransformers
make dev_install
#运行安装脚本
bash install.sh


#更新gcc
sudo apt-get update
sudo apt-get install gcc-11 g++-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 80 --slave /usr/bin/g++ g++ /usr/bin/g++-11
ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /root/miniconda3/envs/kt/bin/../lib/libstdc++.so.6

#更换为hf镜像源
export HF_ENDPOINT=https://hf-mirror.com



#激活kt环境
conda activate kt

#进入ktransformers目录
cd /ktransformers

#windows 获取WSL文件夹权限，用windows文件资源管理器就可以直接拖放文件，把下载的配置文件和权重文件拷贝到目标文件夹
sudo chmod -R 777 /root

python /root/ktransformers/ktransformers/local_chat.py --model_path /root/LLM/Qwen3-235B-A22B-128K-GGUF --gguf_path /root/LLM/Qwen3-235B-A22B-128K-GGUF/Q4_K_M --optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 30 --max_new_tokens 131072
#加载完成Chat:出来了

环境信息

WSL - Ubuntu-22.04
GPU: 5090D
CPU: AMD 9950X3D
RAM: 192G
CUDA: 12.8.0
torch-nightly: 2.8.0.dev20250505+cu128

Originally created by @forhonourlx on GitHub (May 6, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1264 Originally assigned to: @Alisehen on GitHub. ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 5090D试了一天，`unsloth/Qwen3-235B-A22B-128K-GGUF/Q4_K_M` 成功加载了，一旦运行开唠就报错了，请大佬们棒棒忙，提前感谢。 ``` Chat: 你是什么模型？ Traceback (most recent call last): File "/root/ktransformers/ktransformers/util/utils.py", line 206, in prefill_and_generate model._get_logits_warper(generation_config,device=inputs.device) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1944, in __getattr__ raise AttributeError( AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/ktransformers/ktransformers/local_chat.py", line 185, in <module> fire.Fire(local_chat) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/local_chat.py", line 179, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/util/utils.py", line 210, in prefill_and_generate model._get_logits_warper(generation_config) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1944, in __getattr__ raise AttributeError( AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? ``` ### 复现步骤 GGUF files: Q4_K_M [https://www.modelscope.cn/models/unsloth/Qwen3-235B-A22B-128K-GGUF/files](https://www.modelscope.cn/models/unsloth/Qwen3-235B-A22B-128K-GGUF/files) 安装命令步骤供参考： ``` wsl --install Ubuntu-22.04 wsl sudo -i wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-wsl-ubuntu-12-8-local_12.8.0-1_amd64.deb sudo dpkg -i cuda-repo-wsl-ubuntu-12-8-local_12.8.0-1_amd64.deb sudo cp /var/cuda-repo-wsl-ubuntu-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update sudo apt-get -y install cuda-toolkit-12-8 apt-get install nvidia-utils-570 #导入环境变量 export CUDA_HOME=/usr/local/cuda export CUDA_PATH=/usr/local/cuda export LD_LIBRARY_PATH=/usr/local/cuda/lib64 export PATH=/usr/local/cuda/bin:$PATH #刷新 . ~/.bashrc #安装 gcc g++ cmake ninja sudo apt-get update sudo apt-get install gcc g++ cmake ninja-build #安装conda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh chmod +x Miniconda3-latest-Linux-x86_64.sh ./Miniconda3-latest-Linux-x86_64.sh #回车确定回车 yes bash conda create --name kt python=3.12 conda activate kt #安装python包 pip install packaging ninja cpufeature numpy -i https://pypi.tuna.tsinghua.edu.cn/simple pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128 #更新libstdc++ sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 #编译安装flash-attention !!!好久好久 pip install flash-attn --no-build-isolation -i https://pypi.tuna.tsinghua.edu.cn/simple #源代码安装ktransformers git clone https://github.com/kvcache-ai/ktransformers.git cd ktransformers git submodule update --init --recursive #安装web cd ktransformers sudo apt-get update -y && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/nodesource.gpg sudo chmod 644 /usr/share/keyrings/nodesource.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_23.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list sudo apt-get update -y sudo apt-get install nodejs -y cd ktransformers/website npm install @vue/cli npm run build cd ../../ #编译ktransformers make dev_install #运行安装脚本 bash install.sh #更新gcc sudo apt-get update sudo apt-get install gcc-11 g++-11 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 80 --slave /usr/bin/g++ g++ /usr/bin/g++-11 ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /root/miniconda3/envs/kt/bin/../lib/libstdc++.so.6 #更换为hf镜像源 export HF_ENDPOINT=https://hf-mirror.com #激活kt环境 conda activate kt #进入ktransformers目录 cd /ktransformers #windows 获取WSL文件夹权限，用windows文件资源管理器就可以直接拖放文件，把下载的配置文件和权重文件拷贝到目标文件夹 sudo chmod -R 777 /root python /root/ktransformers/ktransformers/local_chat.py --model_path /root/LLM/Qwen3-235B-A22B-128K-GGUF --gguf_path /root/LLM/Qwen3-235B-A22B-128K-GGUF/Q4_K_M --optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 30 --max_new_tokens 131072 #加载完成Chat:出来了 ``` ### 环境信息 WSL - Ubuntu-22.04 GPU: 5090D CPU: AMD 9950X3D RAM: 192G CUDA: 12.8.0 torch-nightly: 2.8.0.dev20250505+cu128

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@forhonourlx commented on GitHub (May 9, 2025):

@Alisehen 恳请大佬帮忙看看，是哪方面的原因呢？
--optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml 这个指定的对吗？

@forhonourlx commented on GitHub (May 9, 2025): @Alisehen 恳请大佬帮忙看看，是哪方面的原因呢？ `--optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml` 这个指定的对吗？

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@Atream commented on GitHub (May 9, 2025):

Qwen3目前只支持--backend_type balance_serve，local chat暂时不支持。

@Atream commented on GitHub (May 9, 2025): Qwen3目前只支持--backend_type balance_serve，local chat暂时不支持。

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@forhonourlx commented on GitHub (May 9, 2025):

Qwen3目前只支持--backend_type balance_serve，local chat暂时不支持。

@Alisehen 原来是这样啊。。

A.若运行：

(kt) root@DESKTOP-N1A6O5M:~# python /root/ktransformers/ktransformers/server/main.py --model_path /root/LLM/Qwen3-235B-A22B-128K-GGUF --gguf_path /root/LLM/Qwen3-235B-A22B-128K-GGUF/Q4_K_M --optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 30 --max_new_tokens 131072 --backend_type balance_serve

报错：

no balance_serve
Traceback (most recent call last):
  File "/root/ktransformers/ktransformers/server/main.py", line 10, in <module>
    from ktransformers.server.args import ArgumentParser
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/args.py", line 3, in <module>
    from ktransformers.util.utils import get_free_ports
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/util/utils.py", line 17, in <module>
    from ktransformers.models.custom_cache import StaticCache
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/custom_cache.py", line 278, in <module>
    class KGQACache(nn.Module):
  File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/custom_cache.py", line 296, in KGQACache
    def load(self, inference_context: sched_ext.InferenceContext):
                                      ^^^^^^^^^
NameError: name 'sched_ext' is not defined

如果用其他issue提出的替换引入的办法是否可行呢?

try:
    from ktransformers.server.balance_serve.settings import sched_ext
except Exception as e:
    # Create dummy class to stand in for missing module
    class DummyModule:
        class InferenceContext:
            pass
    sched_ext = DummyModule()

============================================================

B.安装balance_serve失败，报错如下，该如何修复呢？

  -- Found CUDA: /usr/local/cuda (found version "12.8")
  -- Looking for a CUDA compiler
  -- Looking for a CUDA compiler - NOTFOUND
  -- enabling CUDA
  -- The CUDA compiler identification is unknown
  CMake Error at /root/miniconda3/envs/kt/lib/python3.12/site-packages/cmake/data/share/cmake-4.0/Modules/CMakeDetermineCUDACompiler.cmake:266 (message):
    Failed to detect a default CUDA architecture.



    Compiler output:

  Call Stack (most recent call first):
    CMakeLists.txt:317 (enable_language)


  -- Configuring incomplete, errors occurred!
  CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1']
  CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.3+cu128torch28fancy']
  build_temp: /root/ktransformers/csrc/ktransformers_ext/build
  Traceback (most recent call last):
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
      main()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
      return _build_backend().build_wheel(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 438, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 426, in _build
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
      self.run_setup()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 320, in run_setup
      exec(code, locals())
    File "<string>", line 647, in <module>
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 186, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
      dist.run_commands()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
      self.run_command(cmd)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
      cmd_obj.run()
    File "<string>", line 251, in run
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
      self.run_command("build")
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
      cmd_obj.run()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
      cmd_obj.run()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 99, in run
      _build_ext.run(self)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
      self.build_extensions()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1050, in build_extensions
      build_ext.build_extensions(self)
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
      self._build_extensions_serial()
    File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
      self.build_extension(ext)
    File "<string>", line 577, in build_extension
    File "<string>", line 358, in run_command_with_live_tail
    File "/root/miniconda3/envs/kt/lib/python3.12/subprocess.py", line 573, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cmake', '/root/ktransformers/csrc/ktransformers_ext', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.3+cu128torch28fancy']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for ktransformers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /root/miniconda3/envs/kt/bin/python3.12 /root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfg2jjx1f
  cwd: /root/ktransformers
  Building wheel for ktransformers (pyproject.toml) ... error
  ERROR: Failed building wheel for ktransformers
Failed to build ktransformers
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ktransformers)

@forhonourlx commented on GitHub (May 9, 2025): > Qwen3目前只支持--backend_type balance_serve，local chat暂时不支持。 @Alisehen 原来是这样啊。。 A.若运行： ``` (kt) root@DESKTOP-N1A6O5M:~# python /root/ktransformers/ktransformers/server/main.py --model_path /root/LLM/Qwen3-235B-A22B-128K-GGUF --gguf_path /root/LLM/Qwen3-235B-A22B-128K-GGUF/Q4_K_M --optimize_config_path /root/ktransformers/ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 30 --max_new_tokens 131072 --backend_type balance_serve ``` 报错： ``` no balance_serve Traceback (most recent call last): File "/root/ktransformers/ktransformers/server/main.py", line 10, in <module> from ktransformers.server.args import ArgumentParser File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/args.py", line 3, in <module> from ktransformers.util.utils import get_free_ports File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/util/utils.py", line 17, in <module> from ktransformers.models.custom_cache import StaticCache File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/custom_cache.py", line 278, in <module> class KGQACache(nn.Module): File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/custom_cache.py", line 296, in KGQACache def load(self, inference_context: sched_ext.InferenceContext): ^^^^^^^^^ NameError: name 'sched_ext' is not defined ``` 如果用其他issue提出的替换引入的办法是否可行呢? ``` try: from ktransformers.server.balance_serve.settings import sched_ext except Exception as e: # Create dummy class to stand in for missing module class DummyModule: class InferenceContext: pass sched_ext = DummyModule() ``` ============================================================ B.安装balance_serve失败，报错如下，该如何修复呢？ ``` -- Found CUDA: /usr/local/cuda (found version "12.8") -- Looking for a CUDA compiler -- Looking for a CUDA compiler - NOTFOUND -- enabling CUDA -- The CUDA compiler identification is unknown CMake Error at /root/miniconda3/envs/kt/lib/python3.12/site-packages/cmake/data/share/cmake-4.0/Modules/CMakeDetermineCUDACompiler.cmake:266 (message): Failed to detect a default CUDA architecture. Compiler output: Call Stack (most recent call first): CMakeLists.txt:317 (enable_language) -- Configuring incomplete, errors occurred! CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1'] CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.3+cu128torch28fancy'] build_temp: /root/ktransformers/csrc/ktransformers_ext/build Traceback (most recent call last): File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> main() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main json_out["return_val"] = hook(**hook_input["kwargs"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel return _build_backend().build_wheel( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 438, in build_wheel return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 426, in _build return self._build_with_temp_dir( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir self.run_setup() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/build_meta.py", line 320, in run_setup exec(code, locals()) File "<string>", line 647, in <module> File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup return distutils.core.setup(**attrs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 186, in setup return run_commands(dist) ^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 202, in run_commands dist.run_commands() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands self.run_command(cmd) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command super().run_command(command) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command cmd_obj.run() File "<string>", line 251, in run File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 370, in run self.run_command("build") File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command self.distribution.run_command(command) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command super().run_command(command) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command cmd_obj.run() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command self.distribution.run_command(command) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/dist.py", line 1104, in run_command super().run_command(command) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command cmd_obj.run() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 99, in run _build_ext.run(self) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run self.build_extensions() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1050, in build_extensions build_ext.build_extensions(self) File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions self._build_extensions_serial() File "/root/miniconda3/envs/kt/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial self.build_extension(ext) File "<string>", line 577, in build_extension File "<string>", line 358, in run_command_with_live_tail File "/root/miniconda3/envs/kt/lib/python3.12/subprocess.py", line 573, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['cmake', '/root/ktransformers/csrc/ktransformers_ext', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-312/', '-DPYTHON_EXECUTABLE=/root/miniconda3/envs/kt/bin/python3.12', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-D_GLIBCXX_USE_CXX11_ABI=1', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.3+cu128torch28fancy']' returned non-zero exit status 1. error: subprocess-exited-with-error × Building wheel for ktransformers (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /root/miniconda3/envs/kt/bin/python3.12 /root/miniconda3/envs/kt/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfg2jjx1f cwd: /root/ktransformers Building wheel for ktransformers (pyproject.toml) ... error ERROR: Failed building wheel for ktransformers Failed to build ktransformers ERROR: Failed to build installable wheels for some pyproject.toml based projects (ktransformers) ```

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@forhonourlx commented on GitHub (May 9, 2025):

@Atream 请教大佬

@forhonourlx commented on GitHub (May 9, 2025): @Atream 请教大佬

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@Atream commented on GitHub (May 11, 2025):

@forhonourlx 编译时加上USE_BALANCE_SERVE=1环境变量

@Atream commented on GitHub (May 11, 2025): @forhonourlx 编译时加上USE_BALANCE_SERVE=1环境变量

bsakaguchi commented

2026-10-29 14:45:27 +00:00

Author

Owner

@forhonourlx commented on GitHub (May 11, 2025):

@forhonourlx 编译时加上USE_BALANCE_SERVE=1环境变量

加上USE_BALANCE_SERVE=1环境变量，cmake 4.0.0编译出错：

  -- Found CUDA: /usr/local/cuda (found version "12.8")
  -- Looking for a CUDA compiler
  -- Looking for a CUDA compiler - NOTFOUND
  -- enabling CUDA
  -- The CUDA compiler identification is unknown
  CMake Error at /root/miniconda3/envs/kt/lib/python3.12/site-packages/cmake/data/share/cmake-4.0/Modules/CMakeDetermineCUDACompiler.cmake:266 (message):
    Failed to detect a default CUDA architecture.



    Compiler output:

  Call Stack (most recent call first):
    CMakeLists.txt:317 (enable_language)
......

@forhonourlx commented on GitHub (May 11, 2025): > [@forhonourlx](https://github.com/forhonourlx) 编译时加上USE_BALANCE_SERVE=1环境变量加上USE_BALANCE_SERVE=1环境变量，cmake 4.0.0编译出错： ``` -- Found CUDA: /usr/local/cuda (found version "12.8") -- Looking for a CUDA compiler -- Looking for a CUDA compiler - NOTFOUND -- enabling CUDA -- The CUDA compiler identification is unknown CMake Error at /root/miniconda3/envs/kt/lib/python3.12/site-packages/cmake/data/share/cmake-4.0/Modules/CMakeDetermineCUDACompiler.cmake:266 (message): Failed to detect a default CUDA architecture. Compiler output: Call Stack (most recent call first): CMakeLists.txt:317 (enable_language) ...... ```