kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #788] [Bug] AttributeError: 'Qwen2MoeConfig' object has no attribute 'kv_lora_rank' #571

New issue
Closed
opened 2026-10-29 14:41:21 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-29 14:41:21 +00:00
Owner
Copy link

Originally created by @zivenyang on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/788

Originally assigned to: @Azure-Tang on GitHub.

检查清单

  • 1. 我已经搜索过相关问题，但未能获得预期的帮助
  • 2. 该问题在最新版本中尚未修复
  • 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
  • 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
  • 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

Qwen2-57B-A14B-Instruct-q4_k_m 报错 AttributeError: 'Qwen2MoeConfig' object has no attribute 'kv_lora_rank'

2025-03-04 08:14:08,280 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 22])
2025-03-04 08:14:08,281 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[136]: input_ids: torch.Size([1, 22])
2025-03-04 08:14:08,281 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[161]: same prefix len: 0
2025-03-04 08:14:08,284 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[170]: input_ids: torch.Size([1, 22])
2025-03-04 08:14:08,285 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[171]: generate_ids: torch.Size([1, 0])
2025-03-04 08:14:08,293 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[186]: cache position: 0 to 22
ÎÒÊÇ2025-03-04 08:14:09,257 INFO /opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py[337]: max_new_tokens: 38192
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    await func()
  File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive
    await self.message_event.wait()
  File "/root/anaconda3/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 78cbd44a4c10

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap
    |     await func()
    |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    |     async for event in async_events:
    |   File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    |     async for event in async_events:
    |   File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    |     async for event in async_events:
    |   File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
    |     async for event in async_events:
    |   File "/opt/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 34, in inner
    |     async for token in interface.inference(input_message,id,create.temperature,create.top_p):
    |   File "/opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 232, in inference
    |     async for v in super().inference(local_messages, thread_id, temperature, top_p):
    |   File "/opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 403, in inference
    |     for t in self.generate():
    |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    |     response = gen.send(None)
    |                ^^^^^^^^^^^^^^
    |   File "/opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 344, in generate
    |     num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank,
    |                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    |     return super().__getattribute__(key)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | AttributeError: 'Qwen2MoeConfig' object has no attribute 'kv_lora_rank'
    +------------------------------------

复现步骤

gguf模型地址：https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct-GGUF

export USE_NUMA=1
export HF_ENDPOINT=https://hf-mirror.com

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py  \
   --gguf_path /mnt/ai/Qwen2-57B-A14B-Q4_K_M  \
   --model_path Qwen/Qwen2-57B-A14B-Instruct   \
   --model_name Qwen2-57B-A14B-Instruct-q4_k_m  \
   --cpu_infer 34  \
   --max_new_tokens 38192  \
   --cache_lens 132768  \
   --total_context 132768   \
   --cache_q4 true  \
   --temperature 0.6  \
   --top_p 0.95   \
   --use_cuda_graph \
   --host 0.0.0.0 \
   --port 8080 \
   --optimize_config_path ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml

环境信息

操作系统：ubuntu24.04
CPU：Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz * 2
内存：768G
GPU: NVIDIA RTX A5000

相同环境deepseek r1 q4km运行正常

Package                  Version
------------------------ -------------------------
accelerate               1.4.0
aiohappyeyeballs         2.4.6
aiohttp                  3.11.13
aiosignal                1.3.2
annotated-types          0.7.0
anyio                    4.8.0
attrs                    25.1.0
blessed                  1.20.0
blobfile                 3.0.0
build                    1.2.2.post1
certifi                  2025.1.31
charset-normalizer       3.4.1
click                    8.1.8
colorlog                 6.9.0
cpufeature               0.2.1
einops                   0.8.1
fastapi                  0.115.8
filelock                 3.13.1
fire                     0.7.0
flash_attn               2.7.4.post1
flashinfer-python        0.2.2.post1
frozenlist               1.5.0
fsspec                   2024.6.1
greenlet                 3.1.1
h11                      0.14.0
httpcore                 1.0.7
httpx                    0.28.1
huggingface-hub          0.29.1
idna                     3.10
Jinja2                   3.1.4
jsonpatch                1.33
jsonpointer              3.0.0
ktransformers            0.2.2rc2+cu128torch26avx2
langchain                0.3.19
langchain-core           0.3.40
langchain-text-splitters 0.3.6
langsmith                0.3.11
lxml                     5.3.1
MarkupSafe               2.1.5
mpmath                   1.3.0
multidict                6.1.0
networkx                 3.3
ninja                    1.11.1.3
numpy                    1.26.4
nvidia-cublas-cu12       12.6.4.1
nvidia-cuda-cupti-cu12   12.6.80
nvidia-cuda-nvrtc-cu12   12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12        9.5.1.17
nvidia-cufft-cu12        11.3.0.4
nvidia-curand-cu12       10.3.7.77
nvidia-cusolver-cu12     11.7.1.2
nvidia-cusparse-cu12     12.5.4.2
nvidia-cusparselt-cu12   0.6.3
nvidia-nccl-cu12         2.21.5
nvidia-nvjitlink-cu12    12.6.85
nvidia-nvtx-cu12         12.6.77
orjson                   3.10.15
packaging                24.2
pillow                   11.0.0
pip                      25.0
propcache                0.3.0
protobuf                 5.29.3
psutil                   7.0.0
pycryptodomex            3.21.0
pydantic                 2.10.6
pydantic_core            2.27.2
pyproject_hooks          1.2.0
PyYAML                   6.0.2
regex                    2024.11.6
requests                 2.32.3
requests-toolbelt        1.0.0
safetensors              0.5.3
sentencepiece            0.2.0
setuptools               75.8.0
six                      1.17.0
sniffio                  1.3.1
SQLAlchemy               2.0.38
starlette                0.45.3
sympy                    1.13.1
tenacity                 9.0.0
termcolor                2.5.0
tiktoken                 0.9.0
tokenizers               0.19.1
torch                    2.6.0+cu126
torchaudio               2.6.0+cu126
torchvision              0.21.0+cu126
tqdm                     4.67.1
transformers             4.43.2
triton                   3.2.0
typing_extensions        4.12.2
urllib3                  2.3.0
uvicorn                  0.34.0
wcwidth                  0.2.13
wheel                    0.45.1
yarl                     1.18.3
zstandard                0.23.0
Originally created by @zivenyang on GitHub (Mar 4, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/788 Originally assigned to: @Azure-Tang on GitHub. ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 Qwen2-57B-A14B-Instruct-q4_k_m 报错 AttributeError: 'Qwen2MoeConfig' object has no attribute 'kv_lora_rank' ```bash 2025-03-04 08:14:08,280 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 22]) 2025-03-04 08:14:08,281 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[136]: input_ids: torch.Size([1, 22]) 2025-03-04 08:14:08,281 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[161]: same prefix len: 0 2025-03-04 08:14:08,284 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[170]: input_ids: torch.Size([1, 22]) 2025-03-04 08:14:08,285 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[171]: generate_ids: torch.Size([1, 0]) 2025-03-04 08:14:08,293 DEBUG /opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[186]: cache position: 0 to 22 ÎÒÊÇ2025-03-04 08:14:09,257 INFO /opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py[337]: max_new_tokens: 38192 ERROR: Exception in ASGI application Traceback (most recent call last): File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 268, in __call__ await wrap(partial(self.listen_for_disconnect, receive)) File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap await func() File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 233, in listen_for_disconnect message = await receive() ^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 531, in receive await self.message_event.wait() File "/root/anaconda3/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope 78cbd44a4c10 During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ | await self.middleware_stack(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 261, in __call__ | async with anyio.create_task_group() as task_group: | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 264, in wrap | await func() | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 245, in stream_response | async for chunk in self.body_iterator: | File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link | async for event in async_events: | File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply | async for event in async_events: | File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done | async for event in async_events: | File "/opt/ktransformers/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk | async for event in async_events: | File "/opt/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 34, in inner | async for token in interface.inference(input_message,id,create.temperature,create.top_p): | File "/opt/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 232, in inference | async for v in super().inference(local_messages, thread_id, temperature, top_p): | File "/opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 403, in inference | for t in self.generate(): | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context | response = gen.send(None) | ^^^^^^^^^^^^^^ | File "/opt/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 344, in generate | num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank, | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__ | return super().__getattribute__(key) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | AttributeError: 'Qwen2MoeConfig' object has no attribute 'kv_lora_rank' +------------------------------------ ``` ### 复现步骤 gguf模型地址：https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct-GGUF ```bash export USE_NUMA=1 export HF_ENDPOINT=https://hf-mirror.com PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py \ --gguf_path /mnt/ai/Qwen2-57B-A14B-Q4_K_M \ --model_path Qwen/Qwen2-57B-A14B-Instruct \ --model_name Qwen2-57B-A14B-Instruct-q4_k_m \ --cpu_infer 34 \ --max_new_tokens 38192 \ --cache_lens 132768 \ --total_context 132768 \ --cache_q4 true \ --temperature 0.6 \ --top_p 0.95 \ --use_cuda_graph \ --host 0.0.0.0 \ --port 8080 \ --optimize_config_path ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml ``` ### 环境信息 操作系统：ubuntu24.04 CPU：Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz * 2 内存：768G GPU: NVIDIA RTX A5000 相同环境deepseek r1 q4km运行正常 ```bash Package Version ------------------------ ------------------------- accelerate 1.4.0 aiohappyeyeballs 2.4.6 aiohttp 3.11.13 aiosignal 1.3.2 annotated-types 0.7.0 anyio 4.8.0 attrs 25.1.0 blessed 1.20.0 blobfile 3.0.0 build 1.2.2.post1 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 colorlog 6.9.0 cpufeature 0.2.1 einops 0.8.1 fastapi 0.115.8 filelock 3.13.1 fire 0.7.0 flash_attn 2.7.4.post1 flashinfer-python 0.2.2.post1 frozenlist 1.5.0 fsspec 2024.6.1 greenlet 3.1.1 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.29.1 idna 3.10 Jinja2 3.1.4 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.2.2rc2+cu128torch26avx2 langchain 0.3.19 langchain-core 0.3.40 langchain-text-splitters 0.3.6 langsmith 0.3.11 lxml 5.3.1 MarkupSafe 2.1.5 mpmath 1.3.0 multidict 6.1.0 networkx 3.3 ninja 1.11.1.3 numpy 1.26.4 nvidia-cublas-cu12 12.6.4.1 nvidia-cuda-cupti-cu12 12.6.80 nvidia-cuda-nvrtc-cu12 12.6.77 nvidia-cuda-runtime-cu12 12.6.77 nvidia-cudnn-cu12 9.5.1.17 nvidia-cufft-cu12 11.3.0.4 nvidia-curand-cu12 10.3.7.77 nvidia-cusolver-cu12 11.7.1.2 nvidia-cusparse-cu12 12.5.4.2 nvidia-cusparselt-cu12 0.6.3 nvidia-nccl-cu12 2.21.5 nvidia-nvjitlink-cu12 12.6.85 nvidia-nvtx-cu12 12.6.77 orjson 3.10.15 packaging 24.2 pillow 11.0.0 pip 25.0 propcache 0.3.0 protobuf 5.29.3 psutil 7.0.0 pycryptodomex 3.21.0 pydantic 2.10.6 pydantic_core 2.27.2 pyproject_hooks 1.2.0 PyYAML 6.0.2 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 75.8.0 six 1.17.0 sniffio 1.3.1 SQLAlchemy 2.0.38 starlette 0.45.3 sympy 1.13.1 tenacity 9.0.0 termcolor 2.5.0 tiktoken 0.9.0 tokenizers 0.19.1 torch 2.6.0+cu126 torchaudio 2.6.0+cu126 torchvision 0.21.0+cu126 tqdm 4.67.1 transformers 4.43.2 triton 3.2.0 typing_extensions 4.12.2 urllib3 2.3.0 uvicorn 0.34.0 wcwidth 0.2.13 wheel 0.45.1 yarl 1.18.3 zstandard 0.23.0 ```
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Mar 4, 2025):

qwen不应该会有kv_lora_rank这个属性，您是不是改过yaml？可以尝试一下不指定optimize config运行？

<!-- gh-comment-id:2696971973 --> @Azure-Tang commented on GitHub (Mar 4, 2025): qwen不应该会有kv_lora_rank这个属性，您是不是改过yaml？可以尝试一下不指定optimize config运行？
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@zivenyang commented on GitHub (Mar 4, 2025):

qwen不应该会有kv_lora_rank这个属性，您是不是改过yaml？可以尝试一下不指定optimize config运行？

没有改过代码，一开始没指定optimize config就报了这个错误，后来才加的还是报这个错误

<!-- gh-comment-id:2697105848 --> @zivenyang commented on GitHub (Mar 4, 2025): > qwen不应该会有kv_lora_rank这个属性，您是不是改过yaml？可以尝试一下不指定optimize config运行？ 没有改过代码，一开始没指定optimize config就报了这个错误，后来才加的还是报这个错误
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Mar 4, 2025):

试一下local chat，打个print看一下实际跑的是哪个yaml？

<!-- gh-comment-id:2697164125 --> @Azure-Tang commented on GitHub (Mar 4, 2025): 试一下local chat，打个print看一下实际跑的是哪个yaml？
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@zivenyang commented on GitHub (Mar 4, 2025):

试一下local chat，打个print看一下实际跑的是哪个yaml？

我把flashinfer-python卸载了，现在可以了。

<!-- gh-comment-id:2697171313 --> @zivenyang commented on GitHub (Mar 4, 2025): > 试一下local chat，打个print看一下实际跑的是哪个yaml？ 我把flashinfer-python卸载了，现在可以了。
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@zivenyang commented on GitHub (Mar 4, 2025):

试一下local chat，打个print看一下实际跑的是哪个yaml？

我把flashinfer-python卸载了，现在可以了。

又重新装了flashinfer-python，问题复现了，确认是flashinfer的问题

<!-- gh-comment-id:2697224165 --> @zivenyang commented on GitHub (Mar 4, 2025): > > 试一下local chat，打个print看一下实际跑的是哪个yaml？ > > 我把flashinfer-python卸载了，现在可以了。 又重新装了flashinfer-python，问题复现了，确认是flashinfer的问题
bsakaguchi commented 2026-10-29 14:41:22 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Mar 5, 2025):

fixed by #753

<!-- gh-comment-id:2700536998 --> @Atream commented on GitHub (Mar 5, 2025): fixed by #753
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#571
No description provided.