mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1075] [Bug] #759
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#759
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @chenhongxx on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1075
检查清单
问题描述
有没有安装成功的大神，放出安装环境的具体版本阿？搞了三天还是有各种错误。Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as ktransformers.operators.layernorm . RMSNorm
Injecting model.layers.60.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
Injecting model.norm as ktransformers.operators.layernorm . RMSNorm
Injecting cache as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 244, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 162, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 130, in optimize_and_load_gguf
load_weights(module.lm_head, gguf_loader, "lm_head.")
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 839, in load
self.generate_linear.load(w=w)
File "/home/huadi/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 455, in load
weight = w.view(self.orin_out_features, self.orin_in_features).T
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[129280, 7168]' is invalid for input of size 209715200
复现步骤
python ktransformers/server/main.py
--port 10002
--model_path /home/huadi/deepseek-r1-info
--gguf_path /home/huadi/deepseek-r1-671b.gguf
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 32768
--chunk_size 256
--max_batch_size 4
--backend_type balance_serve
--force_think
环境信息
(ktransformers) huadi@huadi:~/ktransformers$ conda list
packages in environment at /home/huadi/anaconda3/envs/ktransformers:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 1.6.0 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.9.0 pypi_0 pypi
binutils 2.40 h4852527_7 conda-forge
binutils_impl_linux-64 2.40 h5293946_0
binutils_linux-64 2.40 hb3c18ed_9 conda-forge
blessed 1.20.0 pypi_0 pypi
blobfile 3.0.0 pypi_0 pypi
build 1.2.2.post1 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
c-ares 1.34.4 hb9d3cd8_0 conda-forge
c-compiler 1.9.0 h2b85faf_0 conda-forge
ca-certificates 2025.2.25 h06a4308_0
certifi 2025.1.31 pypi_0 pypi
charset-normalizer 3.4.1 pypi_0 pypi
click 8.1.8 pypi_0 pypi
cmake 4.0.0 h74e3db0_0 conda-forge
colorlog 6.9.0 pypi_0 pypi
compilers 1.9.0 ha770c72_0 conda-forge
cpufeature 0.2.1 pypi_0 pypi
cxx-compiler 1.9.0 h1a2810e_0 conda-forge
distro 1.9.0 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
fastapi 0.115.12 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fire 0.7.0 pypi_0 pypi
flash-attention 1.0.0 pypi_0 pypi
flash-attn 2.5.8 pypi_0 pypi
flashinfer-python 0.2.3 pypi_0 pypi
fortran-compiler 1.9.0 h36df796_0 conda-forge
fsspec 2024.6.1 pypi_0 pypi
gcc 13.3.0 h9576a4e_2 conda-forge
gcc_impl_linux-64 13.3.0 h1e990d8_2 conda-forge
gcc_linux-64 13.3.0 hc28eda2_8 conda-forge
gfortran 13.3.0 h9576a4e_2 conda-forge
gfortran_impl_linux-64 13.3.0 h84c1745_2 conda-forge
gfortran_linux-64 13.3.0 hb919d3a_8 conda-forge
greenlet 3.1.1 pypi_0 pypi
gxx 13.3.0 h9576a4e_2 conda-forge
gxx_impl_linux-64 13.3.0 hae580e1_2 conda-forge
gxx_linux-64 13.3.0 h6834431_8 conda-forge
h11 0.14.0 pypi_0 pypi
httpcore 1.0.7 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.30.1 pypi_0 pypi
idna 3.10 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
jiter 0.9.0 pypi_0 pypi
jsonpatch 1.33 pypi_0 pypi
jsonpointer 3.0.0 pypi_0 pypi
kernel-headers_linux-64 3.10.0 he073ed8_18 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
ktransformers 0.2.4.post1+cu126torch26fancy pypi_0 pypi
langchain 0.3.23 pypi_0 pypi
langchain-core 0.3.51 pypi_0 pypi
langchain-text-splitters 0.3.8 pypi_0 pypi
langsmith 0.3.24 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libcurl 8.13.0 h332b0f4_0 conda-forge
libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
libev 4.33 hd590300_2 conda-forge
libexpat 2.7.0 h5888daf_0 conda-forge
libffi 3.4.4 h6a678d5_1
libgcc 14.2.0 h767d61c_2 conda-forge
libgcc-devel_linux-64 13.3.0 hc03c837_102 conda-forge
libgcc-ng 14.2.0 h69a702a_2 conda-forge
libgfortran5 14.2.0 hf1ad2bd_2 conda-forge
libgomp 14.2.0 h767d61c_2 conda-forge
liblzma 5.8.1 hb9d3cd8_0 conda-forge
libnghttp2 1.64.0 h161d5f1_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsanitizer 13.3.0 he8ea267_2 conda-forge
libsqlite 3.49.1 hee588c1_2 conda-forge
libssh2 1.11.1 hf672d98_0 conda-forge
libstdcxx 14.2.0 h8f9b012_2 conda-forge
libstdcxx-devel_linux-64 13.3.0 hc03c837_102 conda-forge
libstdcxx-ng 14.2.0 h4852527_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.50.0 hb9d3cd8_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
lxml 5.3.2 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
modelscope 1.24.1 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.5 h2d0b736_3 conda-forge
networkx 3.3 pypi_0 pypi
ninja 1.11.1.4 pypi_0 pypi
numpy 2.2.4 pypi_0 pypi
nvidia-cublas-cu12 12.6.4.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.6.80 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.6.77 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.6.77 pypi_0 pypi
nvidia-cudnn-cu12 9.5.1.17 pypi_0 pypi
nvidia-cufft-cu12 11.3.0.4 pypi_0 pypi
nvidia-curand-cu12 10.3.7.77 pypi_0 pypi
nvidia-cusolver-cu12 11.7.1.2 pypi_0 pypi
nvidia-cusparse-cu12 12.5.4.2 pypi_0 pypi
nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.6.85 pypi_0 pypi
nvidia-nvtx-cu12 12.6.77 pypi_0 pypi
openai 1.70.0 pypi_0 pypi
openssl 3.4.1 h7b32b05_0 conda-forge
orjson 3.10.16 pypi_0 pypi
packaging 24.2 pypi_0 pypi
pillow 11.0.0 pypi_0 pypi
pip 25.0 py311h06a4308_0
protobuf 6.30.2 pypi_0 pypi
psutil 7.0.0 pypi_0 pypi
pycryptodomex 3.22.0 pypi_0 pypi
pydantic 2.11.2 pypi_0 pypi
pydantic-core 2.33.1 pypi_0 pypi
pyproject-hooks 1.2.0 pypi_0 pypi
python 3.11.11 h9e4cc4f_2_cpython conda-forge
pyyaml 6.0.2 pypi_0 pypi
pyzmq 26.4.0 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
requests-toolbelt 1.0.0 pypi_0 pypi
rhash 1.4.5 hb9d3cd8_0 conda-forge
safetensors 0.5.3 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 75.8.0 py311h06a4308_0
six 1.17.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlalchemy 2.0.40 pypi_0 pypi
sqlite 3.49.1 h9eae976_2 conda-forge
starlette 0.46.1 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
sysroot_linux-64 2.17 h0157908_18 conda-forge
tenacity 9.1.2 pypi_0 pypi
termcolor 3.0.1 pypi_0 pypi
tiktoken 0.9.0 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tokenizers 0.21.1 pypi_0 pypi
torch 2.6.0+cu126 pypi_0 pypi
torchaudio 2.6.0+cu126 pypi_0 pypi
torchvision 0.21.0+cu126 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
transformers 4.51.0 pypi_0 pypi
triton 3.2.0 pypi_0 pypi
typing-extensions 4.13.1 pypi_0 pypi
typing-inspection 0.4.0 pypi_0 pypi
tzdata 2025a h04d1e81_0
urllib3 2.3.0 pypi_0 pypi
uvicorn 0.34.0 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
wheel 0.45.1 py311h06a4308_0
xz 5.6.4 h5eee18b_1
zlib 1.3.1 hb9d3cd8_2 conda-forge
zmq 0.0.0 pypi_0 pypi
zstandard 0.23.0 pypi_0 pypi
zstd 1.5.7 hb8e6e7a_2 conda-forge
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0
nvidia-smi
Tue Apr 8 07:56:42 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05 Driver Version: 560.35.05 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 Off | Off |
| 0% 40C P8 14W / 450W | 15MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 7555 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+
@qiyuxinlin commented on GitHub (Apr 8, 2025):
您好，麻烦贴一下启动命令
@magikRUKKOLA commented on GitHub (Apr 8, 2025):
That seems to be incorrect. In the turorial they specified explicidly:
@chenhongxx commented on GitHub (Apr 8, 2025):
python ktransformers/server/main.py
--port 10002
--model_path /home/huadi/DeepSeek-R1/
--gguf_path /home/huadi/deepseek-r1-671b.gguf
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 32768
--chunk_size 256
--max_batch_size 4
--backend_type balance_serve
--force_think
修改了还是不行