[GH-ISSUE #961] [Bug] (flashinfer) Mixtral inference with flashinfer encounters missing 'kv_lora_rank' attribute when using flashinfer #682

New issue

Open

opened 2026-10-29 14:42:49 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented 2026-10-29 14:42:49 +00:00

Owner

Copy link

Originally created by @pl752 on GitHub (Mar 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/961

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
• 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Loading mixtral 8x7b with default Mixtral.yaml config completes, but any message results in following exception, when using flashinfer

Traceback (most recent call last):
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/cors.py", line 144, in simple_response
    await self.app(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    await response(scope, receive, send)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
    with collapse_excgroups():
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
    await func()
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
    async for chunk in self.body_iterator:
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    async for event in async_events:
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    async for event in async_events:
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    async for event in async_events:
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 101, in filter_api_event
    async for event in async_events:
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/runs.py", line 28, in inner
    async for event in ctx.work():
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/base.py", line 146, in work
    async for res in self.interface.inference(local_messages,self.thread.id):
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference
    async for v in super().inference(local_messages, thread_id, temperature, top_p):
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 411, in inference
    for t, finish_reason in self.generate():
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 345, in generate
    num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank,
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank'

Reproduction

ktransformers --model_path mistralai/Mixtral-8x7B-Instruct-v0.1 --gguf_path /home/pl752/mxtrl7b/ --optimize_config_path ktransformers/optimize/optimize_rules/Mixtral.yaml --web True

Loading mixtral 8x7b with default Mixtral.yaml config completes, but any message results in following exception

Environment

Ubuntu 24.10, python 3.11, torch 2.8 nightly cuda 12.8, ktransformers latest main, flashinfer from pipy

pip list

Package                  Version
------------------------ ------------------------
accelerate               1.5.2
annotated-types          0.7.0
anyio                    4.9.0
blessed                  1.20.0
blobfile                 3.0.0
build                    1.2.2.post1
certifi                  2025.1.31
cffi                     1.17.1
charset-normalizer       3.4.1
click                    8.1.8
colorlog                 6.9.0
cpufeature               0.2.1
distro                   1.9.0
einops                   0.8.1
fastapi                  0.115.11
filelock                 3.16.1
fire                     0.7.0
flash_attn               2.7.4.post1
flashinfer-python        0.2.3
fsspec                   2024.10.0
greenlet                 3.1.1
h11                      0.14.0
httpcore                 1.0.7
httpx                    0.28.1
huggingface-hub          0.29.3
idna                     3.10
Jinja2                   3.1.4
jiter                    0.9.0
jsonpatch                1.33
jsonpointer              3.0.0
ktransformers            0.2.3.post2+torch28avx2
langchain                0.3.21
langchain-core           0.3.47
langchain-text-splitters 0.3.7
langsmith                0.3.18
lxml                     5.3.1
MarkupSafe               2.1.5
mpmath                   1.3.0
networkx                 3.4.2
ninja                    1.11.1.3
numpy                    2.1.2
nvidia-cublas-cu12       12.8.3.14
nvidia-cuda-cupti-cu12   12.8.57
nvidia-cuda-nvrtc-cu12   12.8.61
nvidia-cuda-runtime-cu12 12.8.57
nvidia-cudnn-cu12        9.8.0.87
nvidia-cufft-cu12        11.3.3.41
nvidia-cufile-cu12       1.13.0.11
nvidia-curand-cu12       10.3.9.55
nvidia-cusolver-cu12     11.7.2.55
nvidia-cusparse-cu12     12.5.7.53
nvidia-cusparselt-cu12   0.6.3
nvidia-nccl-cu12         2.26.2
nvidia-nvjitlink-cu12    12.8.61
nvidia-nvtx-cu12         12.8.55
openai                   1.68.0
orjson                   3.10.15
packaging                24.2
pillow                   11.0.0
pip                      25.0.1
protobuf                 6.30.1
psutil                   7.0.0
pycparser                2.22
pycryptodomex            3.22.0
pydantic                 2.10.6
pydantic_core            2.27.2
pyproject_hooks          1.2.0
pytorch-triton           3.3.0+git96316ce5
PyYAML                   6.0.2
regex                    2024.11.6
requests                 2.32.3
requests-toolbelt        1.0.0
safetensors              0.5.3
sentencepiece            0.2.0
setuptools               75.8.2
six                      1.17.0
sniffio                  1.3.1
sounddevice              0.5.1
SQLAlchemy               2.0.39
starlette                0.46.1
sympy                    1.13.3
tenacity                 9.0.0
termcolor                2.5.0
tiktoken                 0.9.0
tokenizers               0.19.1
torch                    2.8.0.dev20250322+cu128
torchaudio               2.6.0.dev20250322+cu128
torchvision              0.22.0.dev20250322+cu128
tqdm                     4.67.1
transformers             4.43.2
triton                   3.2.0
typing_extensions        4.12.2
urllib3                  2.3.0
uvicorn                  0.34.0
wcwidth                  0.2.13
wheel                    0.45.1
zstandard                0.23.0

Originally created by @pl752 on GitHub (Mar 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/961 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug Loading mixtral 8x7b with default Mixtral.yaml config completes, but any message results in following exception, when using flashinfer ``` Traceback (most recent call last): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__ await self.simple_response(scope, receive, send, request_headers=headers) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/cors.py", line 144, in simple_response await self.app(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/routing.py", line 74, in app await response(scope, receive, send) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__ with collapse_excgroups(): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/contextlib.py", line 158, in __exit__ self.gen.throw(typ, value, traceback) File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap await func() File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response async for chunk in self.body_iterator: File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link async for event in async_events: File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply async for event in async_events: File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done async for event in async_events: File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 101, in filter_api_event async for event in async_events: File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/runs.py", line 28, in inner async for event in ctx.work(): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/base.py", line 146, in work async for res in self.interface.inference(local_messages,self.thread.id): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 411, in inference for t, finish_reason in self.generate(): File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 345, in generate num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pl752/miniforge3/envs/ktr/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank' ``` ### Reproduction `ktransformers --model_path mistralai/Mixtral-8x7B-Instruct-v0.1 --gguf_path /home/pl752/mxtrl7b/ --optimize_config_path ktransformers/optimize/optimize_rules/Mixtral.yaml --web True` Loading mixtral 8x7b with default Mixtral.yaml config completes, but any message results in following exception ### Environment Ubuntu 24.10, python 3.11, torch 2.8 nightly cuda 12.8, ktransformers latest main, flashinfer from pipy pip list ``` Package Version ------------------------ ------------------------ accelerate 1.5.2 annotated-types 0.7.0 anyio 4.9.0 blessed 1.20.0 blobfile 3.0.0 build 1.2.2.post1 certifi 2025.1.31 cffi 1.17.1 charset-normalizer 3.4.1 click 8.1.8 colorlog 6.9.0 cpufeature 0.2.1 distro 1.9.0 einops 0.8.1 fastapi 0.115.11 filelock 3.16.1 fire 0.7.0 flash_attn 2.7.4.post1 flashinfer-python 0.2.3 fsspec 2024.10.0 greenlet 3.1.1 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.29.3 idna 3.10 Jinja2 3.1.4 jiter 0.9.0 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.2.3.post2+torch28avx2 langchain 0.3.21 langchain-core 0.3.47 langchain-text-splitters 0.3.7 langsmith 0.3.18 lxml 5.3.1 MarkupSafe 2.1.5 mpmath 1.3.0 networkx 3.4.2 ninja 1.11.1.3 numpy 2.1.2 nvidia-cublas-cu12 12.8.3.14 nvidia-cuda-cupti-cu12 12.8.57 nvidia-cuda-nvrtc-cu12 12.8.61 nvidia-cuda-runtime-cu12 12.8.57 nvidia-cudnn-cu12 9.8.0.87 nvidia-cufft-cu12 11.3.3.41 nvidia-cufile-cu12 1.13.0.11 nvidia-curand-cu12 10.3.9.55 nvidia-cusolver-cu12 11.7.2.55 nvidia-cusparse-cu12 12.5.7.53 nvidia-cusparselt-cu12 0.6.3 nvidia-nccl-cu12 2.26.2 nvidia-nvjitlink-cu12 12.8.61 nvidia-nvtx-cu12 12.8.55 openai 1.68.0 orjson 3.10.15 packaging 24.2 pillow 11.0.0 pip 25.0.1 protobuf 6.30.1 psutil 7.0.0 pycparser 2.22 pycryptodomex 3.22.0 pydantic 2.10.6 pydantic_core 2.27.2 pyproject_hooks 1.2.0 pytorch-triton 3.3.0+git96316ce5 PyYAML 6.0.2 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 75.8.2 six 1.17.0 sniffio 1.3.1 sounddevice 0.5.1 SQLAlchemy 2.0.39 starlette 0.46.1 sympy 1.13.3 tenacity 9.0.0 termcolor 2.5.0 tiktoken 0.9.0 tokenizers 0.19.1 torch 2.8.0.dev20250322+cu128 torchaudio 2.6.0.dev20250322+cu128 torchvision 0.22.0.dev20250322+cu128 tqdm 4.67.1 transformers 4.43.2 triton 3.2.0 typing_extensions 4.12.2 urllib3 2.3.0 uvicorn 0.34.0 wcwidth 0.2.13 wheel 0.45.1 zstandard 0.23.0 ```

bsakaguchi commented 2026-10-29 14:42:50 +00:00

Author

Owner

Copy link

@pl752 commented on GitHub (Mar 24, 2025):

Please, consider checking does the model support MLA

<!-- gh-comment-id:2749101253 --> @pl752 commented on GitHub (Mar 24, 2025): Please, consider checking does the model support MLA

bsakaguchi commented 2026-10-29 14:42:50 +00:00

Author

Owner

Copy link

@Sub0X commented on GitHub (Apr 10, 2025):

Getting the same issue here as well:

(ktransformers) sub01@Sub01-Strix-G16:~/Server/ktransformers$ python ktransformers/server/main.py --model_path mistralai/Mixtral-8x7B-v0.1 --gguf_path /home/sub01/Server/models/Mixtral-8x7B-v0.1 --optimize_config_path ktransformers/optimize/optimize_rules/Mixtral.yaml --port 10002 --max_new_tokens 4096
Warning: sched_ext module not found, balance_serve functionality will be limited
2025-04-09 22:20:01,431 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
Injecting model as default
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as default
Injecting model.layers.0.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.0.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.0.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as default
Injecting model.layers.1.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.1.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.1.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as default
Injecting model.layers.2.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.2.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.2.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as default
Injecting model.layers.3.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.3.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.3.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as default
Injecting model.layers.4.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.4.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.4.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as default
Injecting model.layers.5.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.5.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.5.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as default
Injecting model.layers.6.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.6.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.6.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as default
Injecting model.layers.7.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.7.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.7.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as default
Injecting model.layers.8.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.8.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.8.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as default
Injecting model.layers.9.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.9.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.9.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as default
Injecting model.layers.10.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.10.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.10.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as default
Injecting model.layers.11.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.11.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.11.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as default
Injecting model.layers.12.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.12.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.12.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as default
Injecting model.layers.13.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.13.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.13.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as default
Injecting model.layers.14.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.14.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.14.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as default
Injecting model.layers.15.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.15.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.15.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as default
Injecting model.layers.16.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.16.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.16.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as default
Injecting model.layers.17.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.17.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.17.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as default
Injecting model.layers.18.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.18.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.18.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as default
Injecting model.layers.19.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.19.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.19.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as default
Injecting model.layers.20.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.20.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.20.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as default
Injecting model.layers.21.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.21.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.21.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as default
Injecting model.layers.22.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.22.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.22.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as default
Injecting model.layers.23.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.23.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.23.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as default
Injecting model.layers.24.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.24.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.24.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as default
Injecting model.layers.25.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.25.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.25.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as default
Injecting model.layers.26.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.26.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.26.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as default
Injecting model.layers.27.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.27.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.27.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as default
Injecting model.layers.28.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.28.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.28.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as default
Injecting model.layers.29.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.29.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.29.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as default
Injecting model.layers.30.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.30.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.30.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as default
Injecting model.layers.31.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.31.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock
Injecting model.layers.31.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_norm.weight to cuda:0
loading blk.3.ffn_norm.weight to cuda:0
loading blk.4.attn_norm.weight to cuda:0
loading blk.4.ffn_norm.weight to cuda:0
loading blk.5.attn_norm.weight to cuda:0
loading blk.5.ffn_norm.weight to cuda:0
loading blk.6.attn_norm.weight to cuda:0
loading blk.6.ffn_norm.weight to cuda:0
loading blk.7.attn_norm.weight to cuda:0
loading blk.7.ffn_norm.weight to cuda:0
loading blk.8.attn_norm.weight to cuda:0
loading blk.8.ffn_norm.weight to cuda:0
loading blk.9.attn_norm.weight to cuda:0
loading blk.9.ffn_norm.weight to cuda:0
loading blk.10.attn_norm.weight to cuda:0
loading blk.10.ffn_norm.weight to cuda:0
loading blk.11.attn_norm.weight to cuda:0
loading blk.11.ffn_norm.weight to cuda:0
loading blk.12.attn_norm.weight to cuda:0
loading blk.12.ffn_norm.weight to cuda:0
loading blk.13.attn_norm.weight to cuda:0
loading blk.13.ffn_norm.weight to cuda:0
loading blk.14.attn_norm.weight to cuda:0
loading blk.14.ffn_norm.weight to cuda:0
loading blk.15.attn_norm.weight to cuda:0
loading blk.15.ffn_norm.weight to cuda:0
loading blk.16.attn_norm.weight to cuda:0
loading blk.16.ffn_norm.weight to cuda:0
loading blk.17.attn_norm.weight to cuda:0
loading blk.17.ffn_norm.weight to cuda:0
loading blk.18.attn_norm.weight to cuda:0
loading blk.18.ffn_norm.weight to cuda:0
loading blk.19.attn_norm.weight to cuda:0
loading blk.19.ffn_norm.weight to cuda:0
loading blk.20.attn_norm.weight to cuda:0
loading blk.20.ffn_norm.weight to cuda:0
loading blk.21.attn_norm.weight to cuda:0
loading blk.21.ffn_norm.weight to cuda:0
loading blk.22.attn_norm.weight to cuda:0
loading blk.22.ffn_norm.weight to cuda:0
loading blk.23.attn_norm.weight to cuda:0
loading blk.23.ffn_norm.weight to cuda:0
loading blk.24.attn_norm.weight to cuda:0
loading blk.24.ffn_norm.weight to cuda:0
loading blk.25.attn_norm.weight to cuda:0
loading blk.25.ffn_norm.weight to cuda:0
loading blk.26.attn_norm.weight to cuda:0
loading blk.26.ffn_norm.weight to cuda:0
loading blk.27.attn_norm.weight to cuda:0
loading blk.27.ffn_norm.weight to cuda:0
loading blk.28.attn_norm.weight to cuda:0
loading blk.28.ffn_norm.weight to cuda:0
loading blk.29.attn_norm.weight to cuda:0
loading blk.29.ffn_norm.weight to cuda:0
loading blk.30.attn_norm.weight to cuda:0
loading blk.30.ffn_norm.weight to cuda:0
loading blk.31.attn_norm.weight to cuda:0
loading blk.31.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
2025-04-09 22:20:23,932 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/context_manager.py[23]: Creating Context Manager
2025-04-09 22:20:23,933 INFO /home/sub01/Server/ktransformers/ktransformers/server/main.py[25]: Creating SQL tables
2025-04-09 22:20:23,935 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
INFO:     Started server process [10420]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
INFO:     127.0.0.1:33614 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:33628 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.
2025-04-09 22:20:40,398 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 37])
2025-04-09 22:20:40,398 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 37])
2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 0
2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 37])
2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 0])
2025-04-09 22:20:40,401 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 0 to 37

2025-04-09 22:20:42,335 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[335]: args.max_new_tokens: 4096, cache_lens: 8192, seq_length: 38
2025-04-09 22:20:42,335 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[340]: max_new_tokens: 4095
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
    |     await route.handle(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    |     await response(scope, receive, send)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
    |     with collapse_excgroups():
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    |     raise exc
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
    |     await func()
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    |     async for event in async_events:
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    |     async for event in async_events:
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    |     async for event in async_events:
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
    |     async for event in async_events:
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 49, in inner
    |     async for res in interface.inference(input_message,id, create.temperature, create.top_p):
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference
    |     async for v in super().inference(local_messages, thread_id, temperature, top_p):
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 413, in inference
    |     for t, finish_reason in self.generate():
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    |     response = gen.send(None)
    |                ^^^^^^^^^^^^^^
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 347, in generate
    |     num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank,
    |                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    |     return super().__getattribute__(key)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank'
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    await response(scope, receive, send)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__
    with collapse_excgroups():
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap
    await func()
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response
    async for chunk in self.body_iterator:
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    async for event in async_events:
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    async for event in async_events:
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    async for event in async_events:
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
    async for event in async_events:
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 49, in inner
    async for res in interface.inference(input_message,id, create.temperature, create.top_p):
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference
    async for v in super().inference(local_messages, thread_id, temperature, top_p):
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 413, in inference
    for t, finish_reason in self.generate():
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 347, in generate
    num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank,
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank'

<!-- gh-comment-id:2791385624 --> @Sub0X commented on GitHub (Apr 10, 2025): Getting the same issue here as well: ``` (ktransformers) sub01@Sub01-Strix-G16:~/Server/ktransformers$ python ktransformers/server/main.py --model_path mistralai/Mixtral-8x7B-v0.1 --gguf_path /home/sub01/Server/models/Mixtral-8x7B-v0.1 --optimize_config_path ktransformers/optimize/optimize_rules/Mixtral.yaml --port 10002 --max_new_tokens 4096 Warning: sched_ext module not found, balance_serve functionality will be limited 2025-04-09 22:20:01,431 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend found flashinfer Injecting model as default Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as default Injecting model.layers.0.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.0.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.0.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.0.input_layernorm as default Injecting model.layers.0.post_attention_layernorm as default Injecting model.layers.1 as default Injecting model.layers.1.self_attn as default Injecting model.layers.1.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.1.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.1.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.1.input_layernorm as default Injecting model.layers.1.post_attention_layernorm as default Injecting model.layers.2 as default Injecting model.layers.2.self_attn as default Injecting model.layers.2.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.2.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.2.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.2.input_layernorm as default Injecting model.layers.2.post_attention_layernorm as default Injecting model.layers.3 as default Injecting model.layers.3.self_attn as default Injecting model.layers.3.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.3.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.3.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.3.input_layernorm as default Injecting model.layers.3.post_attention_layernorm as default Injecting model.layers.4 as default Injecting model.layers.4.self_attn as default Injecting model.layers.4.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.4.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.4.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.4.input_layernorm as default Injecting model.layers.4.post_attention_layernorm as default Injecting model.layers.5 as default Injecting model.layers.5.self_attn as default Injecting model.layers.5.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.5.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.5.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.5.input_layernorm as default Injecting model.layers.5.post_attention_layernorm as default Injecting model.layers.6 as default Injecting model.layers.6.self_attn as default Injecting model.layers.6.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.6.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.6.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.6.input_layernorm as default Injecting model.layers.6.post_attention_layernorm as default Injecting model.layers.7 as default Injecting model.layers.7.self_attn as default Injecting model.layers.7.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.7.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.7.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.7.input_layernorm as default Injecting model.layers.7.post_attention_layernorm as default Injecting model.layers.8 as default Injecting model.layers.8.self_attn as default Injecting model.layers.8.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.8.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.8.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.8.input_layernorm as default Injecting model.layers.8.post_attention_layernorm as default Injecting model.layers.9 as default Injecting model.layers.9.self_attn as default Injecting model.layers.9.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.9.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.9.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.9.input_layernorm as default Injecting model.layers.9.post_attention_layernorm as default Injecting model.layers.10 as default Injecting model.layers.10.self_attn as default Injecting model.layers.10.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.10.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.10.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.10.input_layernorm as default Injecting model.layers.10.post_attention_layernorm as default Injecting model.layers.11 as default Injecting model.layers.11.self_attn as default Injecting model.layers.11.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.11.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.11.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.11.input_layernorm as default Injecting model.layers.11.post_attention_layernorm as default Injecting model.layers.12 as default Injecting model.layers.12.self_attn as default Injecting model.layers.12.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.12.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.12.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.12.input_layernorm as default Injecting model.layers.12.post_attention_layernorm as default Injecting model.layers.13 as default Injecting model.layers.13.self_attn as default Injecting model.layers.13.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.13.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.13.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.13.input_layernorm as default Injecting model.layers.13.post_attention_layernorm as default Injecting model.layers.14 as default Injecting model.layers.14.self_attn as default Injecting model.layers.14.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.14.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.14.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.14.input_layernorm as default Injecting model.layers.14.post_attention_layernorm as default Injecting model.layers.15 as default Injecting model.layers.15.self_attn as default Injecting model.layers.15.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.15.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.15.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.15.input_layernorm as default Injecting model.layers.15.post_attention_layernorm as default Injecting model.layers.16 as default Injecting model.layers.16.self_attn as default Injecting model.layers.16.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.16.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.16.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.16.input_layernorm as default Injecting model.layers.16.post_attention_layernorm as default Injecting model.layers.17 as default Injecting model.layers.17.self_attn as default Injecting model.layers.17.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.17.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.17.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.17.input_layernorm as default Injecting model.layers.17.post_attention_layernorm as default Injecting model.layers.18 as default Injecting model.layers.18.self_attn as default Injecting model.layers.18.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.18.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.18.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.18.input_layernorm as default Injecting model.layers.18.post_attention_layernorm as default Injecting model.layers.19 as default Injecting model.layers.19.self_attn as default Injecting model.layers.19.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.19.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.19.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.19.input_layernorm as default Injecting model.layers.19.post_attention_layernorm as default Injecting model.layers.20 as default Injecting model.layers.20.self_attn as default Injecting model.layers.20.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.20.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.20.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.20.input_layernorm as default Injecting model.layers.20.post_attention_layernorm as default Injecting model.layers.21 as default Injecting model.layers.21.self_attn as default Injecting model.layers.21.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.21.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.21.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.21.input_layernorm as default Injecting model.layers.21.post_attention_layernorm as default Injecting model.layers.22 as default Injecting model.layers.22.self_attn as default Injecting model.layers.22.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.22.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.22.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.22.input_layernorm as default Injecting model.layers.22.post_attention_layernorm as default Injecting model.layers.23 as default Injecting model.layers.23.self_attn as default Injecting model.layers.23.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.23.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.23.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.23.input_layernorm as default Injecting model.layers.23.post_attention_layernorm as default Injecting model.layers.24 as default Injecting model.layers.24.self_attn as default Injecting model.layers.24.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.24.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.24.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.24.input_layernorm as default Injecting model.layers.24.post_attention_layernorm as default Injecting model.layers.25 as default Injecting model.layers.25.self_attn as default Injecting model.layers.25.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.25.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.25.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.25.input_layernorm as default Injecting model.layers.25.post_attention_layernorm as default Injecting model.layers.26 as default Injecting model.layers.26.self_attn as default Injecting model.layers.26.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.26.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.26.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.26.input_layernorm as default Injecting model.layers.26.post_attention_layernorm as default Injecting model.layers.27 as default Injecting model.layers.27.self_attn as default Injecting model.layers.27.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.27.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.27.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.27.input_layernorm as default Injecting model.layers.27.post_attention_layernorm as default Injecting model.layers.28 as default Injecting model.layers.28.self_attn as default Injecting model.layers.28.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.28.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.28.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.28.input_layernorm as default Injecting model.layers.28.post_attention_layernorm as default Injecting model.layers.29 as default Injecting model.layers.29.self_attn as default Injecting model.layers.29.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.29.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.29.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.29.input_layernorm as default Injecting model.layers.29.post_attention_layernorm as default Injecting model.layers.30 as default Injecting model.layers.30.self_attn as default Injecting model.layers.30.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.30.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.30.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.30.input_layernorm as default Injecting model.layers.30.post_attention_layernorm as default Injecting model.layers.31 as default Injecting model.layers.31.self_attn as default Injecting model.layers.31.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.31.block_sparse_moe as ktransformers.operators.experts . KMistralSparseMoEBlock Injecting model.layers.31.block_sparse_moe.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.block_sparse_moe.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.31.input_layernorm as default Injecting model.layers.31.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as ktransformers.operators.linear . KTransformersLinear loading token_embd.weight to cpu loading token_embd.weight with CPU loading blk.0.attn_norm.weight to cuda:0 loading blk.0.ffn_norm.weight to cuda:0 loading blk.1.attn_norm.weight to cuda:0 loading blk.1.ffn_norm.weight to cuda:0 loading blk.2.attn_norm.weight to cuda:0 loading blk.2.ffn_norm.weight to cuda:0 loading blk.3.attn_norm.weight to cuda:0 loading blk.3.ffn_norm.weight to cuda:0 loading blk.4.attn_norm.weight to cuda:0 loading blk.4.ffn_norm.weight to cuda:0 loading blk.5.attn_norm.weight to cuda:0 loading blk.5.ffn_norm.weight to cuda:0 loading blk.6.attn_norm.weight to cuda:0 loading blk.6.ffn_norm.weight to cuda:0 loading blk.7.attn_norm.weight to cuda:0 loading blk.7.ffn_norm.weight to cuda:0 loading blk.8.attn_norm.weight to cuda:0 loading blk.8.ffn_norm.weight to cuda:0 loading blk.9.attn_norm.weight to cuda:0 loading blk.9.ffn_norm.weight to cuda:0 loading blk.10.attn_norm.weight to cuda:0 loading blk.10.ffn_norm.weight to cuda:0 loading blk.11.attn_norm.weight to cuda:0 loading blk.11.ffn_norm.weight to cuda:0 loading blk.12.attn_norm.weight to cuda:0 loading blk.12.ffn_norm.weight to cuda:0 loading blk.13.attn_norm.weight to cuda:0 loading blk.13.ffn_norm.weight to cuda:0 loading blk.14.attn_norm.weight to cuda:0 loading blk.14.ffn_norm.weight to cuda:0 loading blk.15.attn_norm.weight to cuda:0 loading blk.15.ffn_norm.weight to cuda:0 loading blk.16.attn_norm.weight to cuda:0 loading blk.16.ffn_norm.weight to cuda:0 loading blk.17.attn_norm.weight to cuda:0 loading blk.17.ffn_norm.weight to cuda:0 loading blk.18.attn_norm.weight to cuda:0 loading blk.18.ffn_norm.weight to cuda:0 loading blk.19.attn_norm.weight to cuda:0 loading blk.19.ffn_norm.weight to cuda:0 loading blk.20.attn_norm.weight to cuda:0 loading blk.20.ffn_norm.weight to cuda:0 loading blk.21.attn_norm.weight to cuda:0 loading blk.21.ffn_norm.weight to cuda:0 loading blk.22.attn_norm.weight to cuda:0 loading blk.22.ffn_norm.weight to cuda:0 loading blk.23.attn_norm.weight to cuda:0 loading blk.23.ffn_norm.weight to cuda:0 loading blk.24.attn_norm.weight to cuda:0 loading blk.24.ffn_norm.weight to cuda:0 loading blk.25.attn_norm.weight to cuda:0 loading blk.25.ffn_norm.weight to cuda:0 loading blk.26.attn_norm.weight to cuda:0 loading blk.26.ffn_norm.weight to cuda:0 loading blk.27.attn_norm.weight to cuda:0 loading blk.27.ffn_norm.weight to cuda:0 loading blk.28.attn_norm.weight to cuda:0 loading blk.28.ffn_norm.weight to cuda:0 loading blk.29.attn_norm.weight to cuda:0 loading blk.29.ffn_norm.weight to cuda:0 loading blk.30.attn_norm.weight to cuda:0 loading blk.30.ffn_norm.weight to cuda:0 loading blk.31.attn_norm.weight to cuda:0 loading blk.31.ffn_norm.weight to cuda:0 loading output_norm.weight to cuda:0 2025-04-09 22:20:23,932 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/context_manager.py[23]: Creating Context Manager 2025-04-09 22:20:23,933 INFO /home/sub01/Server/ktransformers/ktransformers/server/main.py[25]: Creating SQL tables 2025-04-09 22:20:23,935 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant INFO: Started server process [10420] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit) INFO: 127.0.0.1:33614 - "GET /v1/models HTTP/1.1" 200 OK INFO: 127.0.0.1:33628 - "GET /v1/models HTTP/1.1" 200 OK INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues. 2025-04-09 22:20:40,398 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 37]) 2025-04-09 22:20:40,398 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 37]) 2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 0 2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 37]) 2025-04-09 22:20:40,399 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 0]) 2025-04-09 22:20:40,401 DEBUG /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 0 to 37 2025-04-09 22:20:42,335 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[335]: args.max_new_tokens: 4096, cache_lens: 8192, seq_length: 38 2025-04-09 22:20:42,335 INFO /home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[340]: max_new_tokens: 4095 ERROR: Exception in ASGI application + Exception Group Traceback (most recent call last): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups | yield | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 263, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app | await route.handle(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | raise exc | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app | await app(scope, receive, sender) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__ | with collapse_excgroups(): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__ | self.gen.throw(typ, value, traceback) | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups | raise exc | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap | await func() | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response | async for chunk in self.body_iterator: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link | async for event in async_events: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply | async for event in async_events: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done | async for event in async_events: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk | async for event in async_events: | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 49, in inner | async for res in interface.inference(input_message,id, create.temperature, create.top_p): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference | async for v in super().inference(local_messages, thread_id, temperature, top_p): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 413, in inference | for t, finish_reason in self.generate(): | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context | response = gen.send(None) | ^^^^^^^^^^^^^^ | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 347, in generate | num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank, | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__ | return super().__getattribute__(key) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank' +------------------------------------ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app await response(scope, receive, send) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__ with collapse_excgroups(): File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/contextlib.py", line 158, in __exit__ self.gen.throw(typ, value, traceback) File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap await func() File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response async for chunk in self.body_iterator: File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link async for event in async_events: File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply async for event in async_events: File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done async for event in async_events: File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk async for event in async_events: File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 49, in inner async for res in interface.inference(input_message,id, create.temperature, create.top_p): File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 233, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p): File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 413, in inference for t, finish_reason in self.generate(): File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 347, in generate num_heads=self.model.config.num_attention_heads, head_dim_ckv=self.model.config.kv_lora_rank, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sub01/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'MixtralConfig' object has no attribute 'kv_lora_rank' ```

bsakaguchi commented 2026-10-29 14:42:50 +00:00

Author

Owner

Copy link

@addisjeams commented on GitHub (May 4, 2025):

我也遇到了同样的错误,非常的令人沮丧

<!-- gh-comment-id:2849085231 --> @addisjeams commented on GitHub (May 4, 2025): 我也遇到了同样的错误,非常的令人沮丧

bsakaguchi commented 2026-10-29 14:42:50 +00:00

Author

Owner

Copy link

@Qarqor5555555 commented on GitHub (Jun 15, 2025):

The model is supported and I faced a similar problem with a model that is also supported.internllllm

<!-- gh-comment-id:2973834566 --> @Qarqor5555555 commented on GitHub (Jun 15, 2025): The model is supported and I faced a similar problem with a model that is also supported.internllllm

bsakaguchi commented 2026-10-29 14:42:50 +00:00

Author

Owner

Copy link

@Qarqor5555555 commented on GitHub (Jun 15, 2025):

Internlm

<!-- gh-comment-id:2973835222 --> @Qarqor5555555 commented on GitHub (Jun 15, 2025): Internlm

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#682

No description provided.