main.py脚本时报错 #634

New issue

Open

opened 2026-10-29 14:42:13 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:42:13 +00:00

Owner

Originally created by @save-FGG on GitHub (Mar 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/878

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

想启用api端口，

“python -m ktransformers.local_chat /mnt/DeepSeek-R1-config --gguf_path /mnt/deepseek --cpu_infer 18 --max_new_tokens 10240 --force_think truee --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --port 10002”时会直接进入chat：，

2.而“

”时会报错/

flashinfer not found, use triton for linux
usage: kvcache.ai [-h] [--host HOST] [--port PORT] [--api_key API_KEY] [--ssl_keyfile SSL_KEYFILE] [--ssl_certfile SSL_CERTFILE] [--web WEB]
[--model_name MODEL_NAME] [--model_dir MODEL_DIR] [--model_path MODEL_PATH] [--device DEVICE] [--gguf_path GGUF_PATH]
[--optimize_config_path OPTIMIZE_CONFIG_PATH] [--cpu_infer CPU_INFER] [--type TYPE] [--chunk_prefill_size CHUNK_PREFILL_SIZE]
[--paged PAGED] [--total_context TOTAL_CONTEXT] [--max_batch_size MAX_BATCH_SIZE] [--max_new_tokens MAX_NEW_TOKENS]
[--json_mode JSON_MODE] [--healing HEALING] [--ban_strings BAN_STRINGS] [--gpu_split GPU_SPLIT] [--length LENGTH]
[--rope_scale ROPE_SCALE] [--rope_alpha ROPE_ALPHA] [--no_flash_attn NO_FLASH_ATTN] [--low_mem LOW_MEM]
[--experts_per_token EXPERTS_PER_TOKEN] [--load_q4 LOAD_Q4] [--fast_safetensors FAST_SAFETENSORS]
[--draft_model_dir DRAFT_MODEL_DIR] [--no_draft_scale NO_DRAFT_SCALE] [--modes MODES] [--mode MODE] [--username USERNAME]
[--botname BOTNAME] [--system_prompt SYSTEM_PROMPT] [--temperature TEMPERATURE] [--smoothing_factor SMOOTHING_FACTOR]
[--dynamic_temperature DYNAMIC_TEMPERATURE] [--top_k TOP_K] [--top_p TOP_P] [--top_a TOP_A] [--skew SKEW] [--typical TYPICAL]
[--repetition_penalty REPETITION_PENALTY] [--frequency_penalty FREQUENCY_PENALTY] [--presence_penalty PRESENCE_PENALTY]
[--max_response_tokens MAX_RESPONSE_TOKENS] [--response_chunk RESPONSE_CHUNK] [--no_code_formatting NO_CODE_FORMATTING]
[--cache_8bit CACHE_8BIT] [--cache_q4 CACHE_Q4] [--ngram_decoding NGRAM_DECODING] [--print_timings PRINT_TIMINGS]
[--amnesia AMNESIA] [--batch_size BATCH_SIZE] [--cache_lens CACHE_LENS] [--log_dir LOG_DIR] [--log_file LOG_FILE]
[--log_level LOG_LEVEL] [--backup_count BACKUP_COUNT] [--db_type DB_TYPE] [--db_host DB_HOST] [--db_port DB_PORT]
[--db_name DB_NAME] [--db_pool_size DB_POOL_SIZE] [--db_database DB_DATABASE] [--user_secret_key USER_SECRET_KEY]
[--user_algorithm USER_ALGORITHM] [--force_think | --no-force_think] [--use_cuda_graph | --no-use_cuda_graph]
[--web_cross_domain WEB_CROSS_DOMAIN] [--file_upload_dir FILE_UPLOAD_DIR] [--assistant_store_dir ASSISTANT_STORE_DIR]
[--prompt_file PROMPT_FILE]
kvcache.ai: error: unrecognized arguments: truee
(ktransformers) root@xunwei:~/ktransformers#

复现步骤

python ./ktransformers/server/main.py --model_path /mnt/DeepSeek-R1-config --gguf_path /mnt/deepseek/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL --cpu_infer 18 --max_new_tokens 2048 --port 10002 --cache_lens 12288 --force_think truee --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml

环境信息

gpu:v100 32G
cpu:Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz *2 10核
内存：512G
硬盘2.2T

(ktransformers) root@xunwei:~/ktransformers/ktransformers/server# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
(ktransformers) root@xunwei:~/ktransformers/ktransformers/server#

(ktransformers) root@xunwei:~/ktransformers/ktransformers/server# pip list
Package Version

accelerate 1.4.0
annotated-types 0.7.0
anyio 4.8.0
async-timeout 4.0.3
blessed 1.20.0
blobfile 3.0.0
build 1.2.2.post1
certifi 2025.1.31
charset-normalizer 3.4.1
click 8.1.8
colorlog 6.9.0
cpufeature 0.2.1
distro 1.9.0
einops 0.8.1
exceptiongroup 1.2.2
fastapi 0.115.11
filelock 3.13.1
fire 0.7.0
flash-attn 2.6.3
fsspec 2024.6.1
greenlet 3.1.1
h11 0.14.0
httpcore 1.0.7
httpx 0.28.1
huggingface-hub 0.29.3
idna 3.10
Jinja2 3.1.4
jiter 0.9.0
jsonpatch 1.33
jsonpointer 3.0.0
ktransformers 0.2.3.post1+cu126torch26fancy
langchain 0.3.20
langchain-core 0.3.43
langchain-text-splitters 0.3.6
langsmith 0.3.13
lxml 5.3.1
MarkupSafe 2.1.5
modelscope 1.23.2
mpmath 1.3.0
networkx 3.3
ninja 1.11.1.3
numpy 2.1.2
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvtx-cu12 12.6.77
openai 1.66.3
orjson 3.10.15
packaging 24.2
pillow 11.0.0
pip 25.0
protobuf 6.30.0
psutil 7.0.0
pycryptodomex 3.21.0
pydantic 2.10.6
pydantic_core 2.27.2
pyproject_hooks 1.2.0
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
safetensors 0.5.3
sentencepiece 0.2.0
setuptools 75.8.0
six 1.17.0
sniffio 1.3.1
SQLAlchemy 2.0.39
starlette 0.46.1
sympy 1.13.1
tenacity 9.0.0
termcolor 2.5.0
tiktoken 0.9.0
tokenizers 0.19.1
tomli 2.2.1
torch 2.6.0+cu126
torchaudio 2.6.0+cu126
torchvision 0.21.0+cu126
tqdm 4.67.1
transformers 4.43.2
triton 3.2.0
typing_extensions 4.12.2
urllib3 2.3.0
uvicorn 0.34.0
wcwidth 0.2.13
wheel 0.45.1
zstandard 0.23.0
(ktransformers) root@xunwei:~/ktransformers/ktransformers/server#

Originally created by @save-FGG on GitHub (Mar 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/878 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [ ] 2. 该问题在最新版本中尚未修复 - [ ] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [ ] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [ ] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述想启用api端口， 1. “python -m ktransformers.local_chat /mnt/DeepSeek-R1-config --gguf_path /mnt/deepseek --cpu_infer 18 --max_new_tokens 10240 --force_think truee --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --port 10002”时会直接进入chat：， 2.而“ ![Image](https://github.com/user-attachments/assets/6ecb9054-c1d1-4f64-bfb4-62624806b8f0) ”时会报错/ flashinfer not found, use triton for linux usage: kvcache.ai [-h] [--host HOST] [--port PORT] [--api_key API_KEY] [--ssl_keyfile SSL_KEYFILE] [--ssl_certfile SSL_CERTFILE] [--web WEB] [--model_name MODEL_NAME] [--model_dir MODEL_DIR] [--model_path MODEL_PATH] [--device DEVICE] [--gguf_path GGUF_PATH] [--optimize_config_path OPTIMIZE_CONFIG_PATH] [--cpu_infer CPU_INFER] [--type TYPE] [--chunk_prefill_size CHUNK_PREFILL_SIZE] [--paged PAGED] [--total_context TOTAL_CONTEXT] [--max_batch_size MAX_BATCH_SIZE] [--max_new_tokens MAX_NEW_TOKENS] [--json_mode JSON_MODE] [--healing HEALING] [--ban_strings BAN_STRINGS] [--gpu_split GPU_SPLIT] [--length LENGTH] [--rope_scale ROPE_SCALE] [--rope_alpha ROPE_ALPHA] [--no_flash_attn NO_FLASH_ATTN] [--low_mem LOW_MEM] [--experts_per_token EXPERTS_PER_TOKEN] [--load_q4 LOAD_Q4] [--fast_safetensors FAST_SAFETENSORS] [--draft_model_dir DRAFT_MODEL_DIR] [--no_draft_scale NO_DRAFT_SCALE] [--modes MODES] [--mode MODE] [--username USERNAME] [--botname BOTNAME] [--system_prompt SYSTEM_PROMPT] [--temperature TEMPERATURE] [--smoothing_factor SMOOTHING_FACTOR] [--dynamic_temperature DYNAMIC_TEMPERATURE] [--top_k TOP_K] [--top_p TOP_P] [--top_a TOP_A] [--skew SKEW] [--typical TYPICAL] [--repetition_penalty REPETITION_PENALTY] [--frequency_penalty FREQUENCY_PENALTY] [--presence_penalty PRESENCE_PENALTY] [--max_response_tokens MAX_RESPONSE_TOKENS] [--response_chunk RESPONSE_CHUNK] [--no_code_formatting NO_CODE_FORMATTING] [--cache_8bit CACHE_8BIT] [--cache_q4 CACHE_Q4] [--ngram_decoding NGRAM_DECODING] [--print_timings PRINT_TIMINGS] [--amnesia AMNESIA] [--batch_size BATCH_SIZE] [--cache_lens CACHE_LENS] [--log_dir LOG_DIR] [--log_file LOG_FILE] [--log_level LOG_LEVEL] [--backup_count BACKUP_COUNT] [--db_type DB_TYPE] [--db_host DB_HOST] [--db_port DB_PORT] [--db_name DB_NAME] [--db_pool_size DB_POOL_SIZE] [--db_database DB_DATABASE] [--user_secret_key USER_SECRET_KEY] [--user_algorithm USER_ALGORITHM] [--force_think | --no-force_think] [--use_cuda_graph | --no-use_cuda_graph] [--web_cross_domain WEB_CROSS_DOMAIN] [--file_upload_dir FILE_UPLOAD_DIR] [--assistant_store_dir ASSISTANT_STORE_DIR] [--prompt_file PROMPT_FILE] kvcache.ai: error: unrecognized arguments: truee (ktransformers) root@xunwei:~/ktransformers# ### 复现步骤 python ./ktransformers/server/main.py --model_path /mnt/DeepSeek-R1-config --gguf_path /mnt/deepseek/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL --cpu_infer 18 --max_new_tokens 2048 --port 10002 --cache_lens 12288 --force_think truee --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml ### 环境信息 gpu:v100 32G cpu:Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz *2 10核内存：512G 硬盘2.2T (ktransformers) root@xunwei:~/ktransformers/ktransformers/server# nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Fri_Jun_14_16:34:21_PDT_2024 Cuda compilation tools, release 12.6, V12.6.20 Build cuda_12.6.r12.6/compiler.34431801_0 (ktransformers) root@xunwei:~/ktransformers/ktransformers/server# nvidia-smi Thu Mar 13 08:16:48 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 Tesla V100S-PCIE-32GB Off | 00000000:86:00.0 Off | Off | | N/A 70C P0 51W / 250W | 1MiB / 32768MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ (ktransformers) root@xunwei:~/ktransformers/ktransformers/server# (ktransformers) root@xunwei:~/ktransformers/ktransformers/server# pip list Package Version ------------------------ ----------------------------- accelerate 1.4.0 annotated-types 0.7.0 anyio 4.8.0 async-timeout 4.0.3 blessed 1.20.0 blobfile 3.0.0 build 1.2.2.post1 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 colorlog 6.9.0 cpufeature 0.2.1 distro 1.9.0 einops 0.8.1 exceptiongroup 1.2.2 fastapi 0.115.11 filelock 3.13.1 fire 0.7.0 flash-attn 2.6.3 fsspec 2024.6.1 greenlet 3.1.1 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.29.3 idna 3.10 Jinja2 3.1.4 jiter 0.9.0 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.2.3.post1+cu126torch26fancy langchain 0.3.20 langchain-core 0.3.43 langchain-text-splitters 0.3.6 langsmith 0.3.13 lxml 5.3.1 MarkupSafe 2.1.5 modelscope 1.23.2 mpmath 1.3.0 networkx 3.3 ninja 1.11.1.3 numpy 2.1.2 nvidia-cublas-cu12 12.6.4.1 nvidia-cuda-cupti-cu12 12.6.80 nvidia-cuda-nvrtc-cu12 12.6.77 nvidia-cuda-runtime-cu12 12.6.77 nvidia-cudnn-cu12 9.5.1.17 nvidia-cufft-cu12 11.3.0.4 nvidia-curand-cu12 10.3.7.77 nvidia-cusolver-cu12 11.7.1.2 nvidia-cusparse-cu12 12.5.4.2 nvidia-cusparselt-cu12 0.6.3 nvidia-nccl-cu12 2.21.5 nvidia-nvjitlink-cu12 12.6.85 nvidia-nvtx-cu12 12.6.77 openai 1.66.3 orjson 3.10.15 packaging 24.2 pillow 11.0.0 pip 25.0 protobuf 6.30.0 psutil 7.0.0 pycryptodomex 3.21.0 pydantic 2.10.6 pydantic_core 2.27.2 pyproject_hooks 1.2.0 PyYAML 6.0.2 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 75.8.0 six 1.17.0 sniffio 1.3.1 SQLAlchemy 2.0.39 starlette 0.46.1 sympy 1.13.1 tenacity 9.0.0 termcolor 2.5.0 tiktoken 0.9.0 tokenizers 0.19.1 tomli 2.2.1 torch 2.6.0+cu126 torchaudio 2.6.0+cu126 torchvision 0.21.0+cu126 tqdm 4.67.1 transformers 4.43.2 triton 3.2.0 typing_extensions 4.12.2 urllib3 2.3.0 uvicorn 0.34.0 wcwidth 0.2.13 wheel 0.45.1 zstandard 0.23.0 (ktransformers) root@xunwei:~/ktransformers/ktransformers/server#

bsakaguchi commented

2026-10-29 14:42:13 +00:00

Author

Owner

@ubergarm commented on GitHub (Mar 14, 2025):

--force_think truee ---> --force_think true

@ubergarm commented on GitHub (Mar 14, 2025): `--force_think truee` ---> `--force_think true`

bsakaguchi referenced this issue

2026-10-29 14:52:47 +00:00

[PR #781] [MERGED] update documentation to fix error in numa enablement #1480