kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #304] 反复出现GLIBCXX_3.4.30 not found 和 CUDA error: no kernel image is available for execution on the device #193

New issue
Closed
opened 2026-10-29 14:35:39 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-29 14:35:39 +00:00
Owner
Copy link

Originally created by @xiongsp on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/304

CPU: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
GPU: 2x Tesla V100-PCIE-32GB
命令: ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/

使用Docker版本时候，出现GLIBCXX_3.4.30 not found，通过使用Release中的avx512版本解决，但是随后出现CUDA Error。

  1. 使用参数--optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml后并没有解决。
  2. 按照README和 #44 修改配置并重新编译（bash install.sh）也无解决。（修改了DeepSeek-V3-Chat和DeepSeek-V3-Chat-multi-gpu配置）

myrule/DeepSeek-V3-Chat-multi-gpu.yaml的内容如下:

- match:
    name: "^model.embed_tokens"
  replace:
    class: "default"
    kwargs:
        generate_device: "cpu"
        prefill_device: "cpu"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\."
    class: ktransformers.models.modeling_deepseek_v3.DeepseekV3RotaryEmbedding
  replace:
    class: ktransformers.operators.RoPE.YarnRotaryEmbeddingV3
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\."
    class: ktransformers.models.modeling_deepseek_v3.DeepseekV3RotaryEmbedding
  replace:
    class: ktransformers.operators.RoPE.YarnRotaryEmbeddingV3
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.(?!self_attn\\.kv_b_proj).*$"  # regular expression 
    class: torch.nn.Linear  # only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # optimized Kernel on quantized data types
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
      generate_op: "KLinearTorch"
      prefill_op: "KLinearTorch"

- match:
    name: "^model\\.layers\\.([3456][0-9])\\.(?!self_attn\\.kv_b_proj).*$"  # regular expression 
    class: torch.nn.Linear  # only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # optimized Kernel on quantized data types
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"
      generate_op: "KLinearTorch"
      prefill_op: "KLinearTorch"
  
- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp$"
    class: ktransformers.models.modeling_deepseek_v3.DeepseekV3MoE
  replace:
    class: ktransformers.operators.experts.KDeepseekV3MoE     # mlp module with custom forward function
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\.mlp$"
    class: ktransformers.models.modeling_deepseek_v3.DeepseekV3MoE
  replace:
    class: ktransformers.operators.experts.KDeepseekV3MoE     # mlp module with custom forward function
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.gate$"
    class: ktransformers.models.modeling_deepseek_v3.MoEGate
  replace:
    class: ktransformers.operators.gate.KMoEGate
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.gate$"
    class: ktransformers.models.modeling_deepseek_v3.MoEGate
  replace:
    class: ktransformers.operators.gate.KMoEGate     # mlp module with custom forward function
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:0"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda:1"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:1"
  recursive: False # don't recursively inject submodules of this module

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.self_attn$"
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"
- match:
    name: "^model\\.layers\\.([3456][0-9])\\.self_attn$"
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"
- match:
    name: "^model$"
  replace:
    class: "ktransformers.operators.models.KDeepseekV2Model"
    kwargs:
      per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill
      transfer_map: 
        30: "cuda:1"

- match:
    name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\."
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:0"
      prefill_device: "cuda:0"

- match:
    name: "(^model\\.layers\\.([3456][0-9])\\.)|(model.norm)|(lm_head)"
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:1"
      prefill_device: "cuda:1"

已经参考issue #117、#44

请问如何解决，非常感谢

Originally created by @xiongsp on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/304 CPU: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz GPU: 2x Tesla V100-PCIE-32GB 命令: `ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/` 使用Docker版本时候，出现`GLIBCXX_3.4.30 not found`，通过使用Release中的`avx512`版本解决，但是随后出现CUDA Error。 1. 使用参数`--optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml`后并没有解决。 2. 按照README和 #44 修改配置并重新编译（`bash install.sh`）也无解决。（修改了`DeepSeek-V3-Chat`和`DeepSeek-V3-Chat-multi-gpu`配置） `myrule/DeepSeek-V3-Chat-multi-gpu.yaml`的内容如下: ``` - match: name: "^model.embed_tokens" replace: class: "default" kwargs: generate_device: "cpu" prefill_device: "cpu" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\." class: ktransformers.models.modeling_deepseek_v3.DeepseekV3RotaryEmbedding replace: class: ktransformers.operators.RoPE.YarnRotaryEmbeddingV3 kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\." class: ktransformers.models.modeling_deepseek_v3.DeepseekV3RotaryEmbedding replace: class: ktransformers.operators.RoPE.YarnRotaryEmbeddingV3 kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.(?!self_attn\\.kv_b_proj).*$" # regular expression class: torch.nn.Linear # only match modules matching name and class simultaneously replace: class: ktransformers.operators.linear.KTransformersLinear # optimized Kernel on quantized data types kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" generate_op: "KLinearTorch" prefill_op: "KLinearTorch" - match: name: "^model\\.layers\\.([3456][0-9])\\.(?!self_attn\\.kv_b_proj).*$" # regular expression class: torch.nn.Linear # only match modules matching name and class simultaneously replace: class: ktransformers.operators.linear.KTransformersLinear # optimized Kernel on quantized data types kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" generate_op: "KLinearTorch" prefill_op: "KLinearTorch" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp$" class: ktransformers.models.modeling_deepseek_v3.DeepseekV3MoE replace: class: ktransformers.operators.experts.KDeepseekV3MoE # mlp module with custom forward function kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\.mlp$" class: ktransformers.models.modeling_deepseek_v3.DeepseekV3MoE replace: class: ktransformers.operators.experts.KDeepseekV3MoE # mlp module with custom forward function kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.gate$" class: ktransformers.models.modeling_deepseek_v3.MoEGate replace: class: ktransformers.operators.gate.KMoEGate kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.gate$" class: ktransformers.models.modeling_deepseek_v3.MoEGate replace: class: ktransformers.operators.gate.KMoEGate # mlp module with custom forward function kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:0" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.([3456][0-9])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda:1" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:1" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\.self_attn$" replace: class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "^model\\.layers\\.([3456][0-9])\\.self_attn$" replace: class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" - match: name: "^model$" replace: class: "ktransformers.operators.models.KDeepseekV2Model" kwargs: per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill transfer_map: 30: "cuda:1" - match: name: "^model\\.layers\\.(0|[1-9]|[12][0-9])\\." replace: class: "default" kwargs: generate_device: "cuda:0" prefill_device: "cuda:0" - match: name: "(^model\\.layers\\.([3456][0-9])\\.)|(model.norm)|(lm_head)" replace: class: "default" kwargs: generate_device: "cuda:1" prefill_device: "cuda:1" ``` 已经参考issue #117、#44 请问如何解决，非常感谢
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@moneyisallyouneed commented on GitHub (Feb 15, 2025):

GLIBCXX not found这个问题我是通过这样解决的，不知道你能不能用：
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6

<!-- gh-comment-id:2660944269 --> @moneyisallyouneed commented on GitHub (Feb 15, 2025): GLIBCXX not found这个问题我是通过这样解决的，不知道你能不能用： export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@xiongsp commented on GitHub (Feb 15, 2025):

@moneyisallyouneed 感谢你。GLIBCxxx的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。

root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output

2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.q_a_layernorm as default
Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_layernorm as default
Injecting model.layers.9.self_attn.kv_b_proj as default
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.9.mlp.shared_experts as default
Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.act_fn as default
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.q_a_layernorm as default
Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_layernorm as default
Injecting model.layers.10.self_attn.kv_b_proj as default
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.10.mlp.shared_experts as default
Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.act_fn as default
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.q_a_layernorm as default
Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_layernorm as default
Injecting model.layers.11.self_attn.kv_b_proj as default
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.11.mlp.shared_experts as default
Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.act_fn as default
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.q_a_layernorm as default
Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_layernorm as default
Injecting model.layers.12.self_attn.kv_b_proj as default
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.12.mlp.shared_experts as default
Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.act_fn as default
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.q_a_layernorm as default
Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_layernorm as default
Injecting model.layers.13.self_attn.kv_b_proj as default
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.13.mlp.shared_experts as default
Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.act_fn as default
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.q_a_layernorm as default
Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_layernorm as default
Injecting model.layers.14.self_attn.kv_b_proj as default
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.14.mlp.shared_experts as default
Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.act_fn as default
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.q_a_layernorm as default
Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_layernorm as default
Injecting model.layers.15.self_attn.kv_b_proj as default
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.15.mlp.shared_experts as default
Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.act_fn as default
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.q_a_layernorm as default
Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_layernorm as default
Injecting model.layers.16.self_attn.kv_b_proj as default
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.16.mlp.shared_experts as default
Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.act_fn as default
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.q_a_layernorm as default
Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_layernorm as default
Injecting model.layers.17.self_attn.kv_b_proj as default
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.17.mlp.shared_experts as default
Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.act_fn as default
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.q_a_layernorm as default
Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_layernorm as default
Injecting model.layers.18.self_attn.kv_b_proj as default
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.18.mlp.shared_experts as default
Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.act_fn as default
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.q_a_layernorm as default
Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_layernorm as default
Injecting model.layers.19.self_attn.kv_b_proj as default
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.19.mlp.shared_experts as default
Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.act_fn as default
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.q_a_layernorm as default
Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_layernorm as default
Injecting model.layers.20.self_attn.kv_b_proj as default
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.20.mlp.shared_experts as default
Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.act_fn as default
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.q_a_layernorm as default
Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_layernorm as default
Injecting model.layers.21.self_attn.kv_b_proj as default
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.21.mlp.shared_experts as default
Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.act_fn as default
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.q_a_layernorm as default
Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_layernorm as default
Injecting model.layers.22.self_attn.kv_b_proj as default
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.22.mlp.shared_experts as default
Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.act_fn as default
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.q_a_layernorm as default
Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_layernorm as default
Injecting model.layers.23.self_attn.kv_b_proj as default
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.23.mlp.shared_experts as default
Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.act_fn as default
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.q_a_layernorm as default
Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_layernorm as default
Injecting model.layers.24.self_attn.kv_b_proj as default
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.24.mlp.shared_experts as default
Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.act_fn as default
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.q_a_layernorm as default
Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_layernorm as default
Injecting model.layers.25.self_attn.kv_b_proj as default
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.25.mlp.shared_experts as default
Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.act_fn as default
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.q_a_layernorm as default
Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_layernorm as default
Injecting model.layers.26.self_attn.kv_b_proj as default
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.26.mlp.shared_experts as default
Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.act_fn as default
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.q_a_layernorm as default
Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_layernorm as default
Injecting model.layers.27.self_attn.kv_b_proj as default
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.27.mlp.shared_experts as default
Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.act_fn as default
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.q_a_layernorm as default
Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_layernorm as default
Injecting model.layers.28.self_attn.kv_b_proj as default
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.28.mlp.shared_experts as default
Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.act_fn as default
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.q_a_layernorm as default
Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_layernorm as default
Injecting model.layers.29.self_attn.kv_b_proj as default
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.29.mlp.shared_experts as default
Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.act_fn as default
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.q_a_layernorm as default
Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_layernorm as default
Injecting model.layers.30.self_attn.kv_b_proj as default
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.30.mlp.shared_experts as default
Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.act_fn as default
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.q_a_layernorm as default
Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_layernorm as default
Injecting model.layers.31.self_attn.kv_b_proj as default
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.31.mlp.shared_experts as default
Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.act_fn as default
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.layers.32 as default
Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.q_a_layernorm as default
Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_layernorm as default
Injecting model.layers.32.self_attn.kv_b_proj as default
Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.32.mlp.shared_experts as default
Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.act_fn as default
Injecting model.layers.32.input_layernorm as default
Injecting model.layers.32.post_attention_layernorm as default
Injecting model.layers.33 as default
Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.q_a_layernorm as default
Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_layernorm as default
Injecting model.layers.33.self_attn.kv_b_proj as default
Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.33.mlp.shared_experts as default
Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.act_fn as default
Injecting model.layers.33.input_layernorm as default
Injecting model.layers.33.post_attention_layernorm as default
Injecting model.layers.34 as default
Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.q_a_layernorm as default
Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_layernorm as default
Injecting model.layers.34.self_attn.kv_b_proj as default
Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.34.mlp.shared_experts as default
Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.act_fn as default
Injecting model.layers.34.input_layernorm as default
Injecting model.layers.34.post_attention_layernorm as default
Injecting model.layers.35 as default
Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.q_a_layernorm as default
Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_layernorm as default
Injecting model.layers.35.self_attn.kv_b_proj as default
Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.35.mlp.shared_experts as default
Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.act_fn as default
Injecting model.layers.35.input_layernorm as default
Injecting model.layers.35.post_attention_layernorm as default
Injecting model.layers.36 as default
Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.q_a_layernorm as default
Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_layernorm as default
Injecting model.layers.36.self_attn.kv_b_proj as default
Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.36.mlp.shared_experts as default
Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.act_fn as default
Injecting model.layers.36.input_layernorm as default
Injecting model.layers.36.post_attention_layernorm as default
Injecting model.layers.37 as default
Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.q_a_layernorm as default
Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_layernorm as default
Injecting model.layers.37.self_attn.kv_b_proj as default
Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.37.mlp.shared_experts as default
Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.act_fn as default
Injecting model.layers.37.input_layernorm as default
Injecting model.layers.37.post_attention_layernorm as default
Injecting model.layers.38 as default
Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.q_a_layernorm as default
Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_layernorm as default
Injecting model.layers.38.self_attn.kv_b_proj as default
Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.38.mlp.shared_experts as default
Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.act_fn as default
Injecting model.layers.38.input_layernorm as default
Injecting model.layers.38.post_attention_layernorm as default
Injecting model.layers.39 as default
Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.q_a_layernorm as default
Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_layernorm as default
Injecting model.layers.39.self_attn.kv_b_proj as default
Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.39.mlp.shared_experts as default
Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.act_fn as default
Injecting model.layers.39.input_layernorm as default
Injecting model.layers.39.post_attention_layernorm as default
Injecting model.layers.40 as default
Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.q_a_layernorm as default
Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_layernorm as default
Injecting model.layers.40.self_attn.kv_b_proj as default
Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.40.mlp.shared_experts as default
Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.act_fn as default
Injecting model.layers.40.input_layernorm as default
Injecting model.layers.40.post_attention_layernorm as default
Injecting model.layers.41 as default
Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.q_a_layernorm as default
Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_layernorm as default
Injecting model.layers.41.self_attn.kv_b_proj as default
Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.41.mlp.shared_experts as default
Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.act_fn as default
Injecting model.layers.41.input_layernorm as default
Injecting model.layers.41.post_attention_layernorm as default
Injecting model.layers.42 as default
Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.q_a_layernorm as default
Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_layernorm as default
Injecting model.layers.42.self_attn.kv_b_proj as default
Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.42.mlp.shared_experts as default
Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.act_fn as default
Injecting model.layers.42.input_layernorm as default
Injecting model.layers.42.post_attention_layernorm as default
Injecting model.layers.43 as default
Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.q_a_layernorm as default
Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_layernorm as default
Injecting model.layers.43.self_attn.kv_b_proj as default
Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.43.mlp.shared_experts as default
Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.act_fn as default
Injecting model.layers.43.input_layernorm as default
Injecting model.layers.43.post_attention_layernorm as default
Injecting model.layers.44 as default
Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.q_a_layernorm as default
Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_layernorm as default
Injecting model.layers.44.self_attn.kv_b_proj as default
Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.44.mlp.shared_experts as default
Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.act_fn as default
Injecting model.layers.44.input_layernorm as default
Injecting model.layers.44.post_attention_layernorm as default
Injecting model.layers.45 as default
Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.q_a_layernorm as default
Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_layernorm as default
Injecting model.layers.45.self_attn.kv_b_proj as default
Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.45.mlp.shared_experts as default
Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.act_fn as default
Injecting model.layers.45.input_layernorm as default
Injecting model.layers.45.post_attention_layernorm as default
Injecting model.layers.46 as default
Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.q_a_layernorm as default
Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_layernorm as default
Injecting model.layers.46.self_attn.kv_b_proj as default
Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.46.mlp.shared_experts as default
Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.act_fn as default
Injecting model.layers.46.input_layernorm as default
Injecting model.layers.46.post_attention_layernorm as default
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_a_layernorm as default
Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_layernorm as default
Injecting model.layers.47.self_attn.kv_b_proj as default
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.47.mlp.shared_experts as default
Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.act_fn as default
Injecting model.layers.47.input_layernorm as default
Injecting model.layers.47.post_attention_layernorm as default
Injecting model.layers.48 as default
Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.q_a_layernorm as default
Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_layernorm as default
Injecting model.layers.48.self_attn.kv_b_proj as default
Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.48.mlp.shared_experts as default
Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.act_fn as default
Injecting model.layers.48.input_layernorm as default
Injecting model.layers.48.post_attention_layernorm as default
Injecting model.layers.49 as default
Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.q_a_layernorm as default
Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_layernorm as default
Injecting model.layers.49.self_attn.kv_b_proj as default
Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.49.mlp.shared_experts as default
Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.act_fn as default
Injecting model.layers.49.input_layernorm as default
Injecting model.layers.49.post_attention_layernorm as default
Injecting model.layers.50 as default
Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.q_a_layernorm as default
Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_layernorm as default
Injecting model.layers.50.self_attn.kv_b_proj as default
Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.50.mlp.shared_experts as default
Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.act_fn as default
Injecting model.layers.50.input_layernorm as default
Injecting model.layers.50.post_attention_layernorm as default
Injecting model.layers.51 as default
Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.q_a_layernorm as default
Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_layernorm as default
Injecting model.layers.51.self_attn.kv_b_proj as default
Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.51.mlp.shared_experts as default
Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.act_fn as default
Injecting model.layers.51.input_layernorm as default
Injecting model.layers.51.post_attention_layernorm as default
Injecting model.layers.52 as default
Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.q_a_layernorm as default
Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_layernorm as default
Injecting model.layers.52.self_attn.kv_b_proj as default
Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.52.mlp.shared_experts as default
Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.act_fn as default
Injecting model.layers.52.input_layernorm as default
Injecting model.layers.52.post_attention_layernorm as default
Injecting model.layers.53 as default
Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.q_a_layernorm as default
Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_layernorm as default
Injecting model.layers.53.self_attn.kv_b_proj as default
Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.53.mlp.shared_experts as default
Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.act_fn as default
Injecting model.layers.53.input_layernorm as default
Injecting model.layers.53.post_attention_layernorm as default
Injecting model.layers.54 as default
Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.q_a_layernorm as default
Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_layernorm as default
Injecting model.layers.54.self_attn.kv_b_proj as default
Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.54.mlp.shared_experts as default
Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.act_fn as default
Injecting model.layers.54.input_layernorm as default
Injecting model.layers.54.post_attention_layernorm as default
Injecting model.layers.55 as default
Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.q_a_layernorm as default
Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_layernorm as default
Injecting model.layers.55.self_attn.kv_b_proj as default
Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.55.mlp.shared_experts as default
Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.act_fn as default
Injecting model.layers.55.input_layernorm as default
Injecting model.layers.55.post_attention_layernorm as default
Injecting model.layers.56 as default
Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.q_a_layernorm as default
Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_layernorm as default
Injecting model.layers.56.self_attn.kv_b_proj as default
Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.56.mlp.shared_experts as default
Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.act_fn as default
Injecting model.layers.56.input_layernorm as default
Injecting model.layers.56.post_attention_layernorm as default
Injecting model.layers.57 as default
Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.q_a_layernorm as default
Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_layernorm as default
Injecting model.layers.57.self_attn.kv_b_proj as default
Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.57.mlp.shared_experts as default
Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.act_fn as default
Injecting model.layers.57.input_layernorm as default
Injecting model.layers.57.post_attention_layernorm as default
Injecting model.layers.58 as default
Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.q_a_layernorm as default
Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_layernorm as default
Injecting model.layers.58.self_attn.kv_b_proj as default
Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.58.mlp.shared_experts as default
Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.act_fn as default
Injecting model.layers.58.input_layernorm as default
Injecting model.layers.58.post_attention_layernorm as default
Injecting model.layers.59 as default
Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.q_a_layernorm as default
Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_layernorm as default
Injecting model.layers.59.self_attn.kv_b_proj as default
Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.59.mlp.shared_experts as default
Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.act_fn as default
Injecting model.layers.59.input_layernorm as default
Injecting model.layers.59.post_attention_layernorm as default
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
/opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.)
  data = torch.from_numpy(data)
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict
    weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
<!-- gh-comment-id:2660957075 --> @xiongsp commented on GitHub (Feb 15, 2025): @moneyisallyouneed 感谢你。`GLIBCxxx`的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。 ``` root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output 2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables 2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.q_a_layernorm as default Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.kv_a_layernorm as default Injecting model.layers.0.self_attn.kv_b_proj as default Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.0.mlp as default Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.act_fn as default Injecting model.layers.0.input_layernorm as default Injecting model.layers.0.post_attention_layernorm as default Injecting model.layers.1 as default Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.q_a_layernorm as default Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.kv_a_layernorm as default Injecting model.layers.1.self_attn.kv_b_proj as default Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.1.mlp as default Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.act_fn as default Injecting model.layers.1.input_layernorm as default Injecting model.layers.1.post_attention_layernorm as default Injecting model.layers.2 as default Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.q_a_layernorm as default Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.kv_a_layernorm as default Injecting model.layers.2.self_attn.kv_b_proj as default Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.2.mlp as default Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.act_fn as default Injecting model.layers.2.input_layernorm as default Injecting model.layers.2.post_attention_layernorm as default Injecting model.layers.3 as default Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.q_a_layernorm as default Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.kv_a_layernorm as default Injecting model.layers.3.self_attn.kv_b_proj as default Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.3.mlp.shared_experts as default Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.act_fn as default Injecting model.layers.3.input_layernorm as default Injecting model.layers.3.post_attention_layernorm as default Injecting model.layers.4 as default Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.q_a_layernorm as default Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.kv_a_layernorm as default Injecting model.layers.4.self_attn.kv_b_proj as default Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.4.mlp.shared_experts as default Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.act_fn as default Injecting model.layers.4.input_layernorm as default Injecting model.layers.4.post_attention_layernorm as default Injecting model.layers.5 as default Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.q_a_layernorm as default Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.kv_a_layernorm as default Injecting model.layers.5.self_attn.kv_b_proj as default Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.5.mlp.shared_experts as default Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.5.mlp.shared_experts.act_fn as default Injecting model.layers.5.input_layernorm as default Injecting model.layers.5.post_attention_layernorm as default Injecting model.layers.6 as default Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.q_a_layernorm as default Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.kv_a_layernorm as default Injecting model.layers.6.self_attn.kv_b_proj as default Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.6.mlp.shared_experts as default Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.6.mlp.shared_experts.act_fn as default Injecting model.layers.6.input_layernorm as default Injecting model.layers.6.post_attention_layernorm as default Injecting model.layers.7 as default Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.q_a_layernorm as default Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.kv_a_layernorm as default Injecting model.layers.7.self_attn.kv_b_proj as default Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.7.mlp.shared_experts as default Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.7.mlp.shared_experts.act_fn as default Injecting model.layers.7.input_layernorm as default Injecting model.layers.7.post_attention_layernorm as default Injecting model.layers.8 as default Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.q_a_layernorm as default Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.kv_a_layernorm as default Injecting model.layers.8.self_attn.kv_b_proj as default Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.8.mlp.shared_experts as default Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.8.mlp.shared_experts.act_fn as default Injecting model.layers.8.input_layernorm as default Injecting model.layers.8.post_attention_layernorm as default Injecting model.layers.9 as default Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.q_a_layernorm as default Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.kv_a_layernorm as default Injecting model.layers.9.self_attn.kv_b_proj as default Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.9.mlp.shared_experts as default Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.9.mlp.shared_experts.act_fn as default Injecting model.layers.9.input_layernorm as default Injecting model.layers.9.post_attention_layernorm as default Injecting model.layers.10 as default Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.q_a_layernorm as default Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.kv_a_layernorm as default Injecting model.layers.10.self_attn.kv_b_proj as default Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.10.mlp.shared_experts as default Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.10.mlp.shared_experts.act_fn as default Injecting model.layers.10.input_layernorm as default Injecting model.layers.10.post_attention_layernorm as default Injecting model.layers.11 as default Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.q_a_layernorm as default Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.kv_a_layernorm as default Injecting model.layers.11.self_attn.kv_b_proj as default Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.11.mlp.shared_experts as default Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.11.mlp.shared_experts.act_fn as default Injecting model.layers.11.input_layernorm as default Injecting model.layers.11.post_attention_layernorm as default Injecting model.layers.12 as default Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.q_a_layernorm as default Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.kv_a_layernorm as default Injecting model.layers.12.self_attn.kv_b_proj as default Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.12.mlp.shared_experts as default Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.12.mlp.shared_experts.act_fn as default Injecting model.layers.12.input_layernorm as default Injecting model.layers.12.post_attention_layernorm as default Injecting model.layers.13 as default Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.q_a_layernorm as default Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.kv_a_layernorm as default Injecting model.layers.13.self_attn.kv_b_proj as default Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.13.mlp.shared_experts as default Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.13.mlp.shared_experts.act_fn as default Injecting model.layers.13.input_layernorm as default Injecting model.layers.13.post_attention_layernorm as default Injecting model.layers.14 as default Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.q_a_layernorm as default Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.kv_a_layernorm as default Injecting model.layers.14.self_attn.kv_b_proj as default Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.14.mlp.shared_experts as default Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.14.mlp.shared_experts.act_fn as default Injecting model.layers.14.input_layernorm as default Injecting model.layers.14.post_attention_layernorm as default Injecting model.layers.15 as default Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.q_a_layernorm as default Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.kv_a_layernorm as default Injecting model.layers.15.self_attn.kv_b_proj as default Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.15.mlp.shared_experts as default Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.15.mlp.shared_experts.act_fn as default Injecting model.layers.15.input_layernorm as default Injecting model.layers.15.post_attention_layernorm as default Injecting model.layers.16 as default Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.q_a_layernorm as default Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.kv_a_layernorm as default Injecting model.layers.16.self_attn.kv_b_proj as default Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.16.mlp.shared_experts as default Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.16.mlp.shared_experts.act_fn as default Injecting model.layers.16.input_layernorm as default Injecting model.layers.16.post_attention_layernorm as default Injecting model.layers.17 as default Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.q_a_layernorm as default Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.kv_a_layernorm as default Injecting model.layers.17.self_attn.kv_b_proj as default Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.17.mlp.shared_experts as default Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.17.mlp.shared_experts.act_fn as default Injecting model.layers.17.input_layernorm as default Injecting model.layers.17.post_attention_layernorm as default Injecting model.layers.18 as default Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.q_a_layernorm as default Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.kv_a_layernorm as default Injecting model.layers.18.self_attn.kv_b_proj as default Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.18.mlp.shared_experts as default Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.18.mlp.shared_experts.act_fn as default Injecting model.layers.18.input_layernorm as default Injecting model.layers.18.post_attention_layernorm as default Injecting model.layers.19 as default Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.q_a_layernorm as default Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.kv_a_layernorm as default Injecting model.layers.19.self_attn.kv_b_proj as default Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.19.mlp.shared_experts as default Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.19.mlp.shared_experts.act_fn as default Injecting model.layers.19.input_layernorm as default Injecting model.layers.19.post_attention_layernorm as default Injecting model.layers.20 as default Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.q_a_layernorm as default Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.kv_a_layernorm as default Injecting model.layers.20.self_attn.kv_b_proj as default Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.20.mlp.shared_experts as default Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.20.mlp.shared_experts.act_fn as default Injecting model.layers.20.input_layernorm as default Injecting model.layers.20.post_attention_layernorm as default Injecting model.layers.21 as default Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.q_a_layernorm as default Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.kv_a_layernorm as default Injecting model.layers.21.self_attn.kv_b_proj as default Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.21.mlp.shared_experts as default Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.21.mlp.shared_experts.act_fn as default Injecting model.layers.21.input_layernorm as default Injecting model.layers.21.post_attention_layernorm as default Injecting model.layers.22 as default Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.q_a_layernorm as default Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.kv_a_layernorm as default Injecting model.layers.22.self_attn.kv_b_proj as default Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.22.mlp.shared_experts as default Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.22.mlp.shared_experts.act_fn as default Injecting model.layers.22.input_layernorm as default Injecting model.layers.22.post_attention_layernorm as default Injecting model.layers.23 as default Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.q_a_layernorm as default Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.kv_a_layernorm as default Injecting model.layers.23.self_attn.kv_b_proj as default Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.23.mlp.shared_experts as default Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.23.mlp.shared_experts.act_fn as default Injecting model.layers.23.input_layernorm as default Injecting model.layers.23.post_attention_layernorm as default Injecting model.layers.24 as default Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.q_a_layernorm as default Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.kv_a_layernorm as default Injecting model.layers.24.self_attn.kv_b_proj as default Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.24.mlp.shared_experts as default Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.24.mlp.shared_experts.act_fn as default Injecting model.layers.24.input_layernorm as default Injecting model.layers.24.post_attention_layernorm as default Injecting model.layers.25 as default Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.q_a_layernorm as default Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.kv_a_layernorm as default Injecting model.layers.25.self_attn.kv_b_proj as default Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.25.mlp.shared_experts as default Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.25.mlp.shared_experts.act_fn as default Injecting model.layers.25.input_layernorm as default Injecting model.layers.25.post_attention_layernorm as default Injecting model.layers.26 as default Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.q_a_layernorm as default Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.kv_a_layernorm as default Injecting model.layers.26.self_attn.kv_b_proj as default Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.26.mlp.shared_experts as default Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.26.mlp.shared_experts.act_fn as default Injecting model.layers.26.input_layernorm as default Injecting model.layers.26.post_attention_layernorm as default Injecting model.layers.27 as default Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.q_a_layernorm as default Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.kv_a_layernorm as default Injecting model.layers.27.self_attn.kv_b_proj as default Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.27.mlp.shared_experts as default Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.27.mlp.shared_experts.act_fn as default Injecting model.layers.27.input_layernorm as default Injecting model.layers.27.post_attention_layernorm as default Injecting model.layers.28 as default Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.q_a_layernorm as default Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.kv_a_layernorm as default Injecting model.layers.28.self_attn.kv_b_proj as default Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.28.mlp.shared_experts as default Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.28.mlp.shared_experts.act_fn as default Injecting model.layers.28.input_layernorm as default Injecting model.layers.28.post_attention_layernorm as default Injecting model.layers.29 as default Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.q_a_layernorm as default Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.kv_a_layernorm as default Injecting model.layers.29.self_attn.kv_b_proj as default Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.29.mlp.shared_experts as default Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.29.mlp.shared_experts.act_fn as default Injecting model.layers.29.input_layernorm as default Injecting model.layers.29.post_attention_layernorm as default Injecting model.layers.30 as default Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.q_a_layernorm as default Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.kv_a_layernorm as default Injecting model.layers.30.self_attn.kv_b_proj as default Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.30.mlp.shared_experts as default Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.30.mlp.shared_experts.act_fn as default Injecting model.layers.30.input_layernorm as default Injecting model.layers.30.post_attention_layernorm as default Injecting model.layers.31 as default Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.q_a_layernorm as default Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.kv_a_layernorm as default Injecting model.layers.31.self_attn.kv_b_proj as default Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.31.mlp.shared_experts as default Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.31.mlp.shared_experts.act_fn as default Injecting model.layers.31.input_layernorm as default Injecting model.layers.31.post_attention_layernorm as default Injecting model.layers.32 as default Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.self_attn.q_a_layernorm as default Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.self_attn.kv_a_layernorm as default Injecting model.layers.32.self_attn.kv_b_proj as default Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.32.mlp.shared_experts as default Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.32.mlp.shared_experts.act_fn as default Injecting model.layers.32.input_layernorm as default Injecting model.layers.32.post_attention_layernorm as default Injecting model.layers.33 as default Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.self_attn.q_a_layernorm as default Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.self_attn.kv_a_layernorm as default Injecting model.layers.33.self_attn.kv_b_proj as default Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.33.mlp.shared_experts as default Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.33.mlp.shared_experts.act_fn as default Injecting model.layers.33.input_layernorm as default Injecting model.layers.33.post_attention_layernorm as default Injecting model.layers.34 as default Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.self_attn.q_a_layernorm as default Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.self_attn.kv_a_layernorm as default Injecting model.layers.34.self_attn.kv_b_proj as default Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.34.mlp.shared_experts as default Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.34.mlp.shared_experts.act_fn as default Injecting model.layers.34.input_layernorm as default Injecting model.layers.34.post_attention_layernorm as default Injecting model.layers.35 as default Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.self_attn.q_a_layernorm as default Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.self_attn.kv_a_layernorm as default Injecting model.layers.35.self_attn.kv_b_proj as default Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.35.mlp.shared_experts as default Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.35.mlp.shared_experts.act_fn as default Injecting model.layers.35.input_layernorm as default Injecting model.layers.35.post_attention_layernorm as default Injecting model.layers.36 as default Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.self_attn.q_a_layernorm as default Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.self_attn.kv_a_layernorm as default Injecting model.layers.36.self_attn.kv_b_proj as default Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.36.mlp.shared_experts as default Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.36.mlp.shared_experts.act_fn as default Injecting model.layers.36.input_layernorm as default Injecting model.layers.36.post_attention_layernorm as default Injecting model.layers.37 as default Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.self_attn.q_a_layernorm as default Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.self_attn.kv_a_layernorm as default Injecting model.layers.37.self_attn.kv_b_proj as default Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.37.mlp.shared_experts as default Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.37.mlp.shared_experts.act_fn as default Injecting model.layers.37.input_layernorm as default Injecting model.layers.37.post_attention_layernorm as default Injecting model.layers.38 as default Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.self_attn.q_a_layernorm as default Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.self_attn.kv_a_layernorm as default Injecting model.layers.38.self_attn.kv_b_proj as default Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.38.mlp.shared_experts as default Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.38.mlp.shared_experts.act_fn as default Injecting model.layers.38.input_layernorm as default Injecting model.layers.38.post_attention_layernorm as default Injecting model.layers.39 as default Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.self_attn.q_a_layernorm as default Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.self_attn.kv_a_layernorm as default Injecting model.layers.39.self_attn.kv_b_proj as default Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.39.mlp.shared_experts as default Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.39.mlp.shared_experts.act_fn as default Injecting model.layers.39.input_layernorm as default Injecting model.layers.39.post_attention_layernorm as default Injecting model.layers.40 as default Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.self_attn.q_a_layernorm as default Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.self_attn.kv_a_layernorm as default Injecting model.layers.40.self_attn.kv_b_proj as default Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.40.mlp.shared_experts as default Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.40.mlp.shared_experts.act_fn as default Injecting model.layers.40.input_layernorm as default Injecting model.layers.40.post_attention_layernorm as default Injecting model.layers.41 as default Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.self_attn.q_a_layernorm as default Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.self_attn.kv_a_layernorm as default Injecting model.layers.41.self_attn.kv_b_proj as default Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.41.mlp.shared_experts as default Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.41.mlp.shared_experts.act_fn as default Injecting model.layers.41.input_layernorm as default Injecting model.layers.41.post_attention_layernorm as default Injecting model.layers.42 as default Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.self_attn.q_a_layernorm as default Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.self_attn.kv_a_layernorm as default Injecting model.layers.42.self_attn.kv_b_proj as default Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.42.mlp.shared_experts as default Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.42.mlp.shared_experts.act_fn as default Injecting model.layers.42.input_layernorm as default Injecting model.layers.42.post_attention_layernorm as default Injecting model.layers.43 as default Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.self_attn.q_a_layernorm as default Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.self_attn.kv_a_layernorm as default Injecting model.layers.43.self_attn.kv_b_proj as default Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.43.mlp.shared_experts as default Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.43.mlp.shared_experts.act_fn as default Injecting model.layers.43.input_layernorm as default Injecting model.layers.43.post_attention_layernorm as default Injecting model.layers.44 as default Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.self_attn.q_a_layernorm as default Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.self_attn.kv_a_layernorm as default Injecting model.layers.44.self_attn.kv_b_proj as default Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.44.mlp.shared_experts as default Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.44.mlp.shared_experts.act_fn as default Injecting model.layers.44.input_layernorm as default Injecting model.layers.44.post_attention_layernorm as default Injecting model.layers.45 as default Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.self_attn.q_a_layernorm as default Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.self_attn.kv_a_layernorm as default Injecting model.layers.45.self_attn.kv_b_proj as default Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.45.mlp.shared_experts as default Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.45.mlp.shared_experts.act_fn as default Injecting model.layers.45.input_layernorm as default Injecting model.layers.45.post_attention_layernorm as default Injecting model.layers.46 as default Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.self_attn.q_a_layernorm as default Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.self_attn.kv_a_layernorm as default Injecting model.layers.46.self_attn.kv_b_proj as default Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.46.mlp.shared_experts as default Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.46.mlp.shared_experts.act_fn as default Injecting model.layers.46.input_layernorm as default Injecting model.layers.46.post_attention_layernorm as default Injecting model.layers.47 as default Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.q_a_layernorm as default Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.kv_a_layernorm as default Injecting model.layers.47.self_attn.kv_b_proj as default Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.47.mlp.shared_experts as default Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.mlp.shared_experts.act_fn as default Injecting model.layers.47.input_layernorm as default Injecting model.layers.47.post_attention_layernorm as default Injecting model.layers.48 as default Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.self_attn.q_a_layernorm as default Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.self_attn.kv_a_layernorm as default Injecting model.layers.48.self_attn.kv_b_proj as default Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.48.mlp.shared_experts as default Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.48.mlp.shared_experts.act_fn as default Injecting model.layers.48.input_layernorm as default Injecting model.layers.48.post_attention_layernorm as default Injecting model.layers.49 as default Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.self_attn.q_a_layernorm as default Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.self_attn.kv_a_layernorm as default Injecting model.layers.49.self_attn.kv_b_proj as default Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.49.mlp.shared_experts as default Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.49.mlp.shared_experts.act_fn as default Injecting model.layers.49.input_layernorm as default Injecting model.layers.49.post_attention_layernorm as default Injecting model.layers.50 as default Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.self_attn.q_a_layernorm as default Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.self_attn.kv_a_layernorm as default Injecting model.layers.50.self_attn.kv_b_proj as default Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.50.mlp.shared_experts as default Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.50.mlp.shared_experts.act_fn as default Injecting model.layers.50.input_layernorm as default Injecting model.layers.50.post_attention_layernorm as default Injecting model.layers.51 as default Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.self_attn.q_a_layernorm as default Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.self_attn.kv_a_layernorm as default Injecting model.layers.51.self_attn.kv_b_proj as default Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.51.mlp.shared_experts as default Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.51.mlp.shared_experts.act_fn as default Injecting model.layers.51.input_layernorm as default Injecting model.layers.51.post_attention_layernorm as default Injecting model.layers.52 as default Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.self_attn.q_a_layernorm as default Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.self_attn.kv_a_layernorm as default Injecting model.layers.52.self_attn.kv_b_proj as default Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.52.mlp.shared_experts as default Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.52.mlp.shared_experts.act_fn as default Injecting model.layers.52.input_layernorm as default Injecting model.layers.52.post_attention_layernorm as default Injecting model.layers.53 as default Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.self_attn.q_a_layernorm as default Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.self_attn.kv_a_layernorm as default Injecting model.layers.53.self_attn.kv_b_proj as default Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.53.mlp.shared_experts as default Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.53.mlp.shared_experts.act_fn as default Injecting model.layers.53.input_layernorm as default Injecting model.layers.53.post_attention_layernorm as default Injecting model.layers.54 as default Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.self_attn.q_a_layernorm as default Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.self_attn.kv_a_layernorm as default Injecting model.layers.54.self_attn.kv_b_proj as default Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.54.mlp.shared_experts as default Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.54.mlp.shared_experts.act_fn as default Injecting model.layers.54.input_layernorm as default Injecting model.layers.54.post_attention_layernorm as default Injecting model.layers.55 as default Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.self_attn.q_a_layernorm as default Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.self_attn.kv_a_layernorm as default Injecting model.layers.55.self_attn.kv_b_proj as default Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.55.mlp.shared_experts as default Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.55.mlp.shared_experts.act_fn as default Injecting model.layers.55.input_layernorm as default Injecting model.layers.55.post_attention_layernorm as default Injecting model.layers.56 as default Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.self_attn.q_a_layernorm as default Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.self_attn.kv_a_layernorm as default Injecting model.layers.56.self_attn.kv_b_proj as default Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.56.mlp.shared_experts as default Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.56.mlp.shared_experts.act_fn as default Injecting model.layers.56.input_layernorm as default Injecting model.layers.56.post_attention_layernorm as default Injecting model.layers.57 as default Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.self_attn.q_a_layernorm as default Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.self_attn.kv_a_layernorm as default Injecting model.layers.57.self_attn.kv_b_proj as default Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.57.mlp.shared_experts as default Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.57.mlp.shared_experts.act_fn as default Injecting model.layers.57.input_layernorm as default Injecting model.layers.57.post_attention_layernorm as default Injecting model.layers.58 as default Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.self_attn.q_a_layernorm as default Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.self_attn.kv_a_layernorm as default Injecting model.layers.58.self_attn.kv_b_proj as default Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.58.mlp.shared_experts as default Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.58.mlp.shared_experts.act_fn as default Injecting model.layers.58.input_layernorm as default Injecting model.layers.58.post_attention_layernorm as default Injecting model.layers.59 as default Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.self_attn.q_a_layernorm as default Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.self_attn.kv_a_layernorm as default Injecting model.layers.59.self_attn.kv_b_proj as default Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.59.mlp.shared_experts as default Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.59.mlp.shared_experts.act_fn as default Injecting model.layers.59.input_layernorm as default Injecting model.layers.59.post_attention_layernorm as default Injecting model.layers.60 as default Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear /opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.) data = torch.from_numpy(data) Injecting model.layers.60.self_attn.q_a_layernorm as default Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_layernorm as default Injecting model.layers.60.self_attn.kv_b_proj as default Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.60.mlp.shared_experts as default Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.act_fn as default Injecting model.layers.60.input_layernorm as default Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as default loading token_embd.weight to cpu loading blk.0.attn_q_a_norm.weight to cuda:0 loading blk.0.attn_kv_a_norm.weight to cuda:0 loading blk.0.attn_kv_b.weight to cuda:0 Traceback (most recent call last): File "/opt/conda/bin/ktransformers", line 8, in <module> sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main create_interface(config=cfg, default_args=cfg) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface GlobalInterface.interface = BackendInterface(default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf load_weights(module, gguf_loader) File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights module.load() File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights module.load() File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights load_cur_state_dict(module, gguf_loader, prefix) File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype) RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ```
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@moneyisallyouneed commented on GitHub (Feb 15, 2025):

@moneyisallyouneed 感谢你。GLIBCxxx的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。

root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output

�[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m
�[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.q_a_layernorm as default
Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_layernorm as default
Injecting model.layers.9.self_attn.kv_b_proj as default
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.9.mlp.shared_experts as default
Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.act_fn as default
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.q_a_layernorm as default
Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_layernorm as default
Injecting model.layers.10.self_attn.kv_b_proj as default
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.10.mlp.shared_experts as default
Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.act_fn as default
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.q_a_layernorm as default
Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_layernorm as default
Injecting model.layers.11.self_attn.kv_b_proj as default
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.11.mlp.shared_experts as default
Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.act_fn as default
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.q_a_layernorm as default
Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_layernorm as default
Injecting model.layers.12.self_attn.kv_b_proj as default
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.12.mlp.shared_experts as default
Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.act_fn as default
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.q_a_layernorm as default
Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_layernorm as default
Injecting model.layers.13.self_attn.kv_b_proj as default
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.13.mlp.shared_experts as default
Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.act_fn as default
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.q_a_layernorm as default
Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_layernorm as default
Injecting model.layers.14.self_attn.kv_b_proj as default
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.14.mlp.shared_experts as default
Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.act_fn as default
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.q_a_layernorm as default
Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_layernorm as default
Injecting model.layers.15.self_attn.kv_b_proj as default
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.15.mlp.shared_experts as default
Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.act_fn as default
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.q_a_layernorm as default
Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_layernorm as default
Injecting model.layers.16.self_attn.kv_b_proj as default
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.16.mlp.shared_experts as default
Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.act_fn as default
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.q_a_layernorm as default
Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_layernorm as default
Injecting model.layers.17.self_attn.kv_b_proj as default
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.17.mlp.shared_experts as default
Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.act_fn as default
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.q_a_layernorm as default
Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_layernorm as default
Injecting model.layers.18.self_attn.kv_b_proj as default
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.18.mlp.shared_experts as default
Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.act_fn as default
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.q_a_layernorm as default
Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_layernorm as default
Injecting model.layers.19.self_attn.kv_b_proj as default
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.19.mlp.shared_experts as default
Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.act_fn as default
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.q_a_layernorm as default
Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_layernorm as default
Injecting model.layers.20.self_attn.kv_b_proj as default
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.20.mlp.shared_experts as default
Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.act_fn as default
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.q_a_layernorm as default
Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_layernorm as default
Injecting model.layers.21.self_attn.kv_b_proj as default
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.21.mlp.shared_experts as default
Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.act_fn as default
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.q_a_layernorm as default
Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_layernorm as default
Injecting model.layers.22.self_attn.kv_b_proj as default
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.22.mlp.shared_experts as default
Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.act_fn as default
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.q_a_layernorm as default
Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_layernorm as default
Injecting model.layers.23.self_attn.kv_b_proj as default
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.23.mlp.shared_experts as default
Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.act_fn as default
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.q_a_layernorm as default
Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_layernorm as default
Injecting model.layers.24.self_attn.kv_b_proj as default
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.24.mlp.shared_experts as default
Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.act_fn as default
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.q_a_layernorm as default
Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_layernorm as default
Injecting model.layers.25.self_attn.kv_b_proj as default
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.25.mlp.shared_experts as default
Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.act_fn as default
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.q_a_layernorm as default
Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_layernorm as default
Injecting model.layers.26.self_attn.kv_b_proj as default
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.26.mlp.shared_experts as default
Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.act_fn as default
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.q_a_layernorm as default
Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_layernorm as default
Injecting model.layers.27.self_attn.kv_b_proj as default
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.27.mlp.shared_experts as default
Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.act_fn as default
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.q_a_layernorm as default
Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_layernorm as default
Injecting model.layers.28.self_attn.kv_b_proj as default
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.28.mlp.shared_experts as default
Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.act_fn as default
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.q_a_layernorm as default
Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_layernorm as default
Injecting model.layers.29.self_attn.kv_b_proj as default
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.29.mlp.shared_experts as default
Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.act_fn as default
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.q_a_layernorm as default
Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_layernorm as default
Injecting model.layers.30.self_attn.kv_b_proj as default
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.30.mlp.shared_experts as default
Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.act_fn as default
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.q_a_layernorm as default
Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_layernorm as default
Injecting model.layers.31.self_attn.kv_b_proj as default
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.31.mlp.shared_experts as default
Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.act_fn as default
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.layers.32 as default
Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.q_a_layernorm as default
Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_layernorm as default
Injecting model.layers.32.self_attn.kv_b_proj as default
Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.32.mlp.shared_experts as default
Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.act_fn as default
Injecting model.layers.32.input_layernorm as default
Injecting model.layers.32.post_attention_layernorm as default
Injecting model.layers.33 as default
Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.q_a_layernorm as default
Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_layernorm as default
Injecting model.layers.33.self_attn.kv_b_proj as default
Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.33.mlp.shared_experts as default
Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.act_fn as default
Injecting model.layers.33.input_layernorm as default
Injecting model.layers.33.post_attention_layernorm as default
Injecting model.layers.34 as default
Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.q_a_layernorm as default
Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_layernorm as default
Injecting model.layers.34.self_attn.kv_b_proj as default
Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.34.mlp.shared_experts as default
Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.act_fn as default
Injecting model.layers.34.input_layernorm as default
Injecting model.layers.34.post_attention_layernorm as default
Injecting model.layers.35 as default
Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.q_a_layernorm as default
Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_layernorm as default
Injecting model.layers.35.self_attn.kv_b_proj as default
Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.35.mlp.shared_experts as default
Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.act_fn as default
Injecting model.layers.35.input_layernorm as default
Injecting model.layers.35.post_attention_layernorm as default
Injecting model.layers.36 as default
Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.q_a_layernorm as default
Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_layernorm as default
Injecting model.layers.36.self_attn.kv_b_proj as default
Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.36.mlp.shared_experts as default
Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.act_fn as default
Injecting model.layers.36.input_layernorm as default
Injecting model.layers.36.post_attention_layernorm as default
Injecting model.layers.37 as default
Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.q_a_layernorm as default
Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_layernorm as default
Injecting model.layers.37.self_attn.kv_b_proj as default
Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.37.mlp.shared_experts as default
Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.act_fn as default
Injecting model.layers.37.input_layernorm as default
Injecting model.layers.37.post_attention_layernorm as default
Injecting model.layers.38 as default
Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.q_a_layernorm as default
Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_layernorm as default
Injecting model.layers.38.self_attn.kv_b_proj as default
Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.38.mlp.shared_experts as default
Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.act_fn as default
Injecting model.layers.38.input_layernorm as default
Injecting model.layers.38.post_attention_layernorm as default
Injecting model.layers.39 as default
Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.q_a_layernorm as default
Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_layernorm as default
Injecting model.layers.39.self_attn.kv_b_proj as default
Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.39.mlp.shared_experts as default
Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.act_fn as default
Injecting model.layers.39.input_layernorm as default
Injecting model.layers.39.post_attention_layernorm as default
Injecting model.layers.40 as default
Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.q_a_layernorm as default
Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_layernorm as default
Injecting model.layers.40.self_attn.kv_b_proj as default
Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.40.mlp.shared_experts as default
Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.act_fn as default
Injecting model.layers.40.input_layernorm as default
Injecting model.layers.40.post_attention_layernorm as default
Injecting model.layers.41 as default
Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.q_a_layernorm as default
Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_layernorm as default
Injecting model.layers.41.self_attn.kv_b_proj as default
Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.41.mlp.shared_experts as default
Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.act_fn as default
Injecting model.layers.41.input_layernorm as default
Injecting model.layers.41.post_attention_layernorm as default
Injecting model.layers.42 as default
Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.q_a_layernorm as default
Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_layernorm as default
Injecting model.layers.42.self_attn.kv_b_proj as default
Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.42.mlp.shared_experts as default
Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.act_fn as default
Injecting model.layers.42.input_layernorm as default
Injecting model.layers.42.post_attention_layernorm as default
Injecting model.layers.43 as default
Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.q_a_layernorm as default
Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_layernorm as default
Injecting model.layers.43.self_attn.kv_b_proj as default
Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.43.mlp.shared_experts as default
Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.act_fn as default
Injecting model.layers.43.input_layernorm as default
Injecting model.layers.43.post_attention_layernorm as default
Injecting model.layers.44 as default
Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.q_a_layernorm as default
Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_layernorm as default
Injecting model.layers.44.self_attn.kv_b_proj as default
Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.44.mlp.shared_experts as default
Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.act_fn as default
Injecting model.layers.44.input_layernorm as default
Injecting model.layers.44.post_attention_layernorm as default
Injecting model.layers.45 as default
Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.q_a_layernorm as default
Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_layernorm as default
Injecting model.layers.45.self_attn.kv_b_proj as default
Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.45.mlp.shared_experts as default
Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.act_fn as default
Injecting model.layers.45.input_layernorm as default
Injecting model.layers.45.post_attention_layernorm as default
Injecting model.layers.46 as default
Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.q_a_layernorm as default
Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_layernorm as default
Injecting model.layers.46.self_attn.kv_b_proj as default
Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.46.mlp.shared_experts as default
Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.act_fn as default
Injecting model.layers.46.input_layernorm as default
Injecting model.layers.46.post_attention_layernorm as default
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_a_layernorm as default
Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_layernorm as default
Injecting model.layers.47.self_attn.kv_b_proj as default
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.47.mlp.shared_experts as default
Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.act_fn as default
Injecting model.layers.47.input_layernorm as default
Injecting model.layers.47.post_attention_layernorm as default
Injecting model.layers.48 as default
Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.q_a_layernorm as default
Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_layernorm as default
Injecting model.layers.48.self_attn.kv_b_proj as default
Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.48.mlp.shared_experts as default
Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.act_fn as default
Injecting model.layers.48.input_layernorm as default
Injecting model.layers.48.post_attention_layernorm as default
Injecting model.layers.49 as default
Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.q_a_layernorm as default
Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_layernorm as default
Injecting model.layers.49.self_attn.kv_b_proj as default
Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.49.mlp.shared_experts as default
Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.act_fn as default
Injecting model.layers.49.input_layernorm as default
Injecting model.layers.49.post_attention_layernorm as default
Injecting model.layers.50 as default
Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.q_a_layernorm as default
Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_layernorm as default
Injecting model.layers.50.self_attn.kv_b_proj as default
Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.50.mlp.shared_experts as default
Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.act_fn as default
Injecting model.layers.50.input_layernorm as default
Injecting model.layers.50.post_attention_layernorm as default
Injecting model.layers.51 as default
Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.q_a_layernorm as default
Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_layernorm as default
Injecting model.layers.51.self_attn.kv_b_proj as default
Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.51.mlp.shared_experts as default
Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.act_fn as default
Injecting model.layers.51.input_layernorm as default
Injecting model.layers.51.post_attention_layernorm as default
Injecting model.layers.52 as default
Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.q_a_layernorm as default
Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_layernorm as default
Injecting model.layers.52.self_attn.kv_b_proj as default
Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.52.mlp.shared_experts as default
Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.act_fn as default
Injecting model.layers.52.input_layernorm as default
Injecting model.layers.52.post_attention_layernorm as default
Injecting model.layers.53 as default
Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.q_a_layernorm as default
Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_layernorm as default
Injecting model.layers.53.self_attn.kv_b_proj as default
Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.53.mlp.shared_experts as default
Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.act_fn as default
Injecting model.layers.53.input_layernorm as default
Injecting model.layers.53.post_attention_layernorm as default
Injecting model.layers.54 as default
Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.q_a_layernorm as default
Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_layernorm as default
Injecting model.layers.54.self_attn.kv_b_proj as default
Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.54.mlp.shared_experts as default
Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.act_fn as default
Injecting model.layers.54.input_layernorm as default
Injecting model.layers.54.post_attention_layernorm as default
Injecting model.layers.55 as default
Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.q_a_layernorm as default
Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_layernorm as default
Injecting model.layers.55.self_attn.kv_b_proj as default
Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.55.mlp.shared_experts as default
Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.act_fn as default
Injecting model.layers.55.input_layernorm as default
Injecting model.layers.55.post_attention_layernorm as default
Injecting model.layers.56 as default
Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.q_a_layernorm as default
Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_layernorm as default
Injecting model.layers.56.self_attn.kv_b_proj as default
Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.56.mlp.shared_experts as default
Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.act_fn as default
Injecting model.layers.56.input_layernorm as default
Injecting model.layers.56.post_attention_layernorm as default
Injecting model.layers.57 as default
Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.q_a_layernorm as default
Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_layernorm as default
Injecting model.layers.57.self_attn.kv_b_proj as default
Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.57.mlp.shared_experts as default
Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.act_fn as default
Injecting model.layers.57.input_layernorm as default
Injecting model.layers.57.post_attention_layernorm as default
Injecting model.layers.58 as default
Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.q_a_layernorm as default
Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_layernorm as default
Injecting model.layers.58.self_attn.kv_b_proj as default
Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.58.mlp.shared_experts as default
Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.act_fn as default
Injecting model.layers.58.input_layernorm as default
Injecting model.layers.58.post_attention_layernorm as default
Injecting model.layers.59 as default
Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.q_a_layernorm as default
Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_layernorm as default
Injecting model.layers.59.self_attn.kv_b_proj as default
Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.59.mlp.shared_experts as default
Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.act_fn as default
Injecting model.layers.59.input_layernorm as default
Injecting model.layers.59.post_attention_layernorm as default
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
/opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.)
  data = torch.from_numpy(data)
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict
    weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

建议你不要使用whl，直接在main上build试试，然后使用export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6。再检查下检查cuda和torch版本啥的

<!-- gh-comment-id:2660961217 --> @moneyisallyouneed commented on GitHub (Feb 15, 2025): > [@moneyisallyouneed](https://github.com/moneyisallyouneed) 感谢你。`GLIBCxxx`的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。 > > ``` > root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output > > �[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m > �[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m > Injecting model as ktransformers.operators.models . KDeepseekV2Model > Injecting model.embed_tokens as default > Injecting model.layers as default > Injecting model.layers.0 as default > Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.q_a_layernorm as default > Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_layernorm as default > Injecting model.layers.0.self_attn.kv_b_proj as default > Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.0.mlp as default > Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.act_fn as default > Injecting model.layers.0.input_layernorm as default > Injecting model.layers.0.post_attention_layernorm as default > Injecting model.layers.1 as default > Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.q_a_layernorm as default > Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_layernorm as default > Injecting model.layers.1.self_attn.kv_b_proj as default > Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.1.mlp as default > Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.act_fn as default > Injecting model.layers.1.input_layernorm as default > Injecting model.layers.1.post_attention_layernorm as default > Injecting model.layers.2 as default > Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.q_a_layernorm as default > Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_layernorm as default > Injecting model.layers.2.self_attn.kv_b_proj as default > Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.2.mlp as default > Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.act_fn as default > Injecting model.layers.2.input_layernorm as default > Injecting model.layers.2.post_attention_layernorm as default > Injecting model.layers.3 as default > Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.q_a_layernorm as default > Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_layernorm as default > Injecting model.layers.3.self_attn.kv_b_proj as default > Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.3.mlp.shared_experts as default > Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.act_fn as default > Injecting model.layers.3.input_layernorm as default > Injecting model.layers.3.post_attention_layernorm as default > Injecting model.layers.4 as default > Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.q_a_layernorm as default > Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_layernorm as default > Injecting model.layers.4.self_attn.kv_b_proj as default > Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.4.mlp.shared_experts as default > Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.act_fn as default > Injecting model.layers.4.input_layernorm as default > Injecting model.layers.4.post_attention_layernorm as default > Injecting model.layers.5 as default > Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.q_a_layernorm as default > Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_layernorm as default > Injecting model.layers.5.self_attn.kv_b_proj as default > Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.5.mlp.shared_experts as default > Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.act_fn as default > Injecting model.layers.5.input_layernorm as default > Injecting model.layers.5.post_attention_layernorm as default > Injecting model.layers.6 as default > Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.q_a_layernorm as default > Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_layernorm as default > Injecting model.layers.6.self_attn.kv_b_proj as default > Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.6.mlp.shared_experts as default > Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.act_fn as default > Injecting model.layers.6.input_layernorm as default > Injecting model.layers.6.post_attention_layernorm as default > Injecting model.layers.7 as default > Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.q_a_layernorm as default > Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_layernorm as default > Injecting model.layers.7.self_attn.kv_b_proj as default > Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.7.mlp.shared_experts as default > Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.act_fn as default > Injecting model.layers.7.input_layernorm as default > Injecting model.layers.7.post_attention_layernorm as default > Injecting model.layers.8 as default > Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.q_a_layernorm as default > Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_layernorm as default > Injecting model.layers.8.self_attn.kv_b_proj as default > Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.8.mlp.shared_experts as default > Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.act_fn as default > Injecting model.layers.8.input_layernorm as default > Injecting model.layers.8.post_attention_layernorm as default > Injecting model.layers.9 as default > Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.q_a_layernorm as default > Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_layernorm as default > Injecting model.layers.9.self_attn.kv_b_proj as default > Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.9.mlp.shared_experts as default > Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.act_fn as default > Injecting model.layers.9.input_layernorm as default > Injecting model.layers.9.post_attention_layernorm as default > Injecting model.layers.10 as default > Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.q_a_layernorm as default > Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_layernorm as default > Injecting model.layers.10.self_attn.kv_b_proj as default > Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.10.mlp.shared_experts as default > Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.act_fn as default > Injecting model.layers.10.input_layernorm as default > Injecting model.layers.10.post_attention_layernorm as default > Injecting model.layers.11 as default > Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.q_a_layernorm as default > Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_layernorm as default > Injecting model.layers.11.self_attn.kv_b_proj as default > Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.11.mlp.shared_experts as default > Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.act_fn as default > Injecting model.layers.11.input_layernorm as default > Injecting model.layers.11.post_attention_layernorm as default > Injecting model.layers.12 as default > Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.q_a_layernorm as default > Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_layernorm as default > Injecting model.layers.12.self_attn.kv_b_proj as default > Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.12.mlp.shared_experts as default > Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.act_fn as default > Injecting model.layers.12.input_layernorm as default > Injecting model.layers.12.post_attention_layernorm as default > Injecting model.layers.13 as default > Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.q_a_layernorm as default > Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_layernorm as default > Injecting model.layers.13.self_attn.kv_b_proj as default > Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.13.mlp.shared_experts as default > Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.act_fn as default > Injecting model.layers.13.input_layernorm as default > Injecting model.layers.13.post_attention_layernorm as default > Injecting model.layers.14 as default > Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.q_a_layernorm as default > Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_layernorm as default > Injecting model.layers.14.self_attn.kv_b_proj as default > Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.14.mlp.shared_experts as default > Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.act_fn as default > Injecting model.layers.14.input_layernorm as default > Injecting model.layers.14.post_attention_layernorm as default > Injecting model.layers.15 as default > Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.q_a_layernorm as default > Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_layernorm as default > Injecting model.layers.15.self_attn.kv_b_proj as default > Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.15.mlp.shared_experts as default > Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.act_fn as default > Injecting model.layers.15.input_layernorm as default > Injecting model.layers.15.post_attention_layernorm as default > Injecting model.layers.16 as default > Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.q_a_layernorm as default > Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_layernorm as default > Injecting model.layers.16.self_attn.kv_b_proj as default > Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.16.mlp.shared_experts as default > Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.act_fn as default > Injecting model.layers.16.input_layernorm as default > Injecting model.layers.16.post_attention_layernorm as default > Injecting model.layers.17 as default > Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.q_a_layernorm as default > Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_layernorm as default > Injecting model.layers.17.self_attn.kv_b_proj as default > Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.17.mlp.shared_experts as default > Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.act_fn as default > Injecting model.layers.17.input_layernorm as default > Injecting model.layers.17.post_attention_layernorm as default > Injecting model.layers.18 as default > Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.q_a_layernorm as default > Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_layernorm as default > Injecting model.layers.18.self_attn.kv_b_proj as default > Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.18.mlp.shared_experts as default > Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.act_fn as default > Injecting model.layers.18.input_layernorm as default > Injecting model.layers.18.post_attention_layernorm as default > Injecting model.layers.19 as default > Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.q_a_layernorm as default > Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_layernorm as default > Injecting model.layers.19.self_attn.kv_b_proj as default > Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.19.mlp.shared_experts as default > Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.act_fn as default > Injecting model.layers.19.input_layernorm as default > Injecting model.layers.19.post_attention_layernorm as default > Injecting model.layers.20 as default > Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.q_a_layernorm as default > Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_layernorm as default > Injecting model.layers.20.self_attn.kv_b_proj as default > Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.20.mlp.shared_experts as default > Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.act_fn as default > Injecting model.layers.20.input_layernorm as default > Injecting model.layers.20.post_attention_layernorm as default > Injecting model.layers.21 as default > Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.q_a_layernorm as default > Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_layernorm as default > Injecting model.layers.21.self_attn.kv_b_proj as default > Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.21.mlp.shared_experts as default > Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.act_fn as default > Injecting model.layers.21.input_layernorm as default > Injecting model.layers.21.post_attention_layernorm as default > Injecting model.layers.22 as default > Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.q_a_layernorm as default > Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_layernorm as default > Injecting model.layers.22.self_attn.kv_b_proj as default > Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.22.mlp.shared_experts as default > Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.act_fn as default > Injecting model.layers.22.input_layernorm as default > Injecting model.layers.22.post_attention_layernorm as default > Injecting model.layers.23 as default > Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.q_a_layernorm as default > Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_layernorm as default > Injecting model.layers.23.self_attn.kv_b_proj as default > Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.23.mlp.shared_experts as default > Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.act_fn as default > Injecting model.layers.23.input_layernorm as default > Injecting model.layers.23.post_attention_layernorm as default > Injecting model.layers.24 as default > Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.q_a_layernorm as default > Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_layernorm as default > Injecting model.layers.24.self_attn.kv_b_proj as default > Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.24.mlp.shared_experts as default > Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.act_fn as default > Injecting model.layers.24.input_layernorm as default > Injecting model.layers.24.post_attention_layernorm as default > Injecting model.layers.25 as default > Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.q_a_layernorm as default > Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_layernorm as default > Injecting model.layers.25.self_attn.kv_b_proj as default > Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.25.mlp.shared_experts as default > Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.act_fn as default > Injecting model.layers.25.input_layernorm as default > Injecting model.layers.25.post_attention_layernorm as default > Injecting model.layers.26 as default > Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.q_a_layernorm as default > Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_layernorm as default > Injecting model.layers.26.self_attn.kv_b_proj as default > Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.26.mlp.shared_experts as default > Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.act_fn as default > Injecting model.layers.26.input_layernorm as default > Injecting model.layers.26.post_attention_layernorm as default > Injecting model.layers.27 as default > Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.q_a_layernorm as default > Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_layernorm as default > Injecting model.layers.27.self_attn.kv_b_proj as default > Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.27.mlp.shared_experts as default > Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.act_fn as default > Injecting model.layers.27.input_layernorm as default > Injecting model.layers.27.post_attention_layernorm as default > Injecting model.layers.28 as default > Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.q_a_layernorm as default > Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_layernorm as default > Injecting model.layers.28.self_attn.kv_b_proj as default > Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.28.mlp.shared_experts as default > Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.act_fn as default > Injecting model.layers.28.input_layernorm as default > Injecting model.layers.28.post_attention_layernorm as default > Injecting model.layers.29 as default > Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.q_a_layernorm as default > Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_layernorm as default > Injecting model.layers.29.self_attn.kv_b_proj as default > Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.29.mlp.shared_experts as default > Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.act_fn as default > Injecting model.layers.29.input_layernorm as default > Injecting model.layers.29.post_attention_layernorm as default > Injecting model.layers.30 as default > Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.q_a_layernorm as default > Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_layernorm as default > Injecting model.layers.30.self_attn.kv_b_proj as default > Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.30.mlp.shared_experts as default > Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.act_fn as default > Injecting model.layers.30.input_layernorm as default > Injecting model.layers.30.post_attention_layernorm as default > Injecting model.layers.31 as default > Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.q_a_layernorm as default > Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_layernorm as default > Injecting model.layers.31.self_attn.kv_b_proj as default > Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.31.mlp.shared_experts as default > Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.act_fn as default > Injecting model.layers.31.input_layernorm as default > Injecting model.layers.31.post_attention_layernorm as default > Injecting model.layers.32 as default > Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.q_a_layernorm as default > Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_layernorm as default > Injecting model.layers.32.self_attn.kv_b_proj as default > Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.32.mlp.shared_experts as default > Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.act_fn as default > Injecting model.layers.32.input_layernorm as default > Injecting model.layers.32.post_attention_layernorm as default > Injecting model.layers.33 as default > Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.q_a_layernorm as default > Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_layernorm as default > Injecting model.layers.33.self_attn.kv_b_proj as default > Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.33.mlp.shared_experts as default > Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.act_fn as default > Injecting model.layers.33.input_layernorm as default > Injecting model.layers.33.post_attention_layernorm as default > Injecting model.layers.34 as default > Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.q_a_layernorm as default > Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_layernorm as default > Injecting model.layers.34.self_attn.kv_b_proj as default > Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.34.mlp.shared_experts as default > Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.act_fn as default > Injecting model.layers.34.input_layernorm as default > Injecting model.layers.34.post_attention_layernorm as default > Injecting model.layers.35 as default > Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.q_a_layernorm as default > Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_layernorm as default > Injecting model.layers.35.self_attn.kv_b_proj as default > Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.35.mlp.shared_experts as default > Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.act_fn as default > Injecting model.layers.35.input_layernorm as default > Injecting model.layers.35.post_attention_layernorm as default > Injecting model.layers.36 as default > Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.q_a_layernorm as default > Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_layernorm as default > Injecting model.layers.36.self_attn.kv_b_proj as default > Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.36.mlp.shared_experts as default > Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.act_fn as default > Injecting model.layers.36.input_layernorm as default > Injecting model.layers.36.post_attention_layernorm as default > Injecting model.layers.37 as default > Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.q_a_layernorm as default > Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_layernorm as default > Injecting model.layers.37.self_attn.kv_b_proj as default > Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.37.mlp.shared_experts as default > Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.act_fn as default > Injecting model.layers.37.input_layernorm as default > Injecting model.layers.37.post_attention_layernorm as default > Injecting model.layers.38 as default > Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.q_a_layernorm as default > Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_layernorm as default > Injecting model.layers.38.self_attn.kv_b_proj as default > Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.38.mlp.shared_experts as default > Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.act_fn as default > Injecting model.layers.38.input_layernorm as default > Injecting model.layers.38.post_attention_layernorm as default > Injecting model.layers.39 as default > Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.q_a_layernorm as default > Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_layernorm as default > Injecting model.layers.39.self_attn.kv_b_proj as default > Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.39.mlp.shared_experts as default > Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.act_fn as default > Injecting model.layers.39.input_layernorm as default > Injecting model.layers.39.post_attention_layernorm as default > Injecting model.layers.40 as default > Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.q_a_layernorm as default > Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_layernorm as default > Injecting model.layers.40.self_attn.kv_b_proj as default > Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.40.mlp.shared_experts as default > Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.act_fn as default > Injecting model.layers.40.input_layernorm as default > Injecting model.layers.40.post_attention_layernorm as default > Injecting model.layers.41 as default > Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.q_a_layernorm as default > Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_layernorm as default > Injecting model.layers.41.self_attn.kv_b_proj as default > Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.41.mlp.shared_experts as default > Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.act_fn as default > Injecting model.layers.41.input_layernorm as default > Injecting model.layers.41.post_attention_layernorm as default > Injecting model.layers.42 as default > Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.q_a_layernorm as default > Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_layernorm as default > Injecting model.layers.42.self_attn.kv_b_proj as default > Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.42.mlp.shared_experts as default > Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.act_fn as default > Injecting model.layers.42.input_layernorm as default > Injecting model.layers.42.post_attention_layernorm as default > Injecting model.layers.43 as default > Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.q_a_layernorm as default > Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_layernorm as default > Injecting model.layers.43.self_attn.kv_b_proj as default > Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.43.mlp.shared_experts as default > Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.act_fn as default > Injecting model.layers.43.input_layernorm as default > Injecting model.layers.43.post_attention_layernorm as default > Injecting model.layers.44 as default > Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.q_a_layernorm as default > Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_layernorm as default > Injecting model.layers.44.self_attn.kv_b_proj as default > Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.44.mlp.shared_experts as default > Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.act_fn as default > Injecting model.layers.44.input_layernorm as default > Injecting model.layers.44.post_attention_layernorm as default > Injecting model.layers.45 as default > Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.q_a_layernorm as default > Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_layernorm as default > Injecting model.layers.45.self_attn.kv_b_proj as default > Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.45.mlp.shared_experts as default > Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.act_fn as default > Injecting model.layers.45.input_layernorm as default > Injecting model.layers.45.post_attention_layernorm as default > Injecting model.layers.46 as default > Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.q_a_layernorm as default > Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_layernorm as default > Injecting model.layers.46.self_attn.kv_b_proj as default > Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.46.mlp.shared_experts as default > Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.act_fn as default > Injecting model.layers.46.input_layernorm as default > Injecting model.layers.46.post_attention_layernorm as default > Injecting model.layers.47 as default > Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.q_a_layernorm as default > Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_layernorm as default > Injecting model.layers.47.self_attn.kv_b_proj as default > Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.47.mlp.shared_experts as default > Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.act_fn as default > Injecting model.layers.47.input_layernorm as default > Injecting model.layers.47.post_attention_layernorm as default > Injecting model.layers.48 as default > Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.q_a_layernorm as default > Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_layernorm as default > Injecting model.layers.48.self_attn.kv_b_proj as default > Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.48.mlp.shared_experts as default > Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.act_fn as default > Injecting model.layers.48.input_layernorm as default > Injecting model.layers.48.post_attention_layernorm as default > Injecting model.layers.49 as default > Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.q_a_layernorm as default > Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_layernorm as default > Injecting model.layers.49.self_attn.kv_b_proj as default > Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.49.mlp.shared_experts as default > Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.act_fn as default > Injecting model.layers.49.input_layernorm as default > Injecting model.layers.49.post_attention_layernorm as default > Injecting model.layers.50 as default > Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.q_a_layernorm as default > Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_layernorm as default > Injecting model.layers.50.self_attn.kv_b_proj as default > Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.50.mlp.shared_experts as default > Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.act_fn as default > Injecting model.layers.50.input_layernorm as default > Injecting model.layers.50.post_attention_layernorm as default > Injecting model.layers.51 as default > Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.q_a_layernorm as default > Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_layernorm as default > Injecting model.layers.51.self_attn.kv_b_proj as default > Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.51.mlp.shared_experts as default > Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.act_fn as default > Injecting model.layers.51.input_layernorm as default > Injecting model.layers.51.post_attention_layernorm as default > Injecting model.layers.52 as default > Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.q_a_layernorm as default > Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_layernorm as default > Injecting model.layers.52.self_attn.kv_b_proj as default > Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.52.mlp.shared_experts as default > Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.act_fn as default > Injecting model.layers.52.input_layernorm as default > Injecting model.layers.52.post_attention_layernorm as default > Injecting model.layers.53 as default > Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.q_a_layernorm as default > Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_layernorm as default > Injecting model.layers.53.self_attn.kv_b_proj as default > Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.53.mlp.shared_experts as default > Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.act_fn as default > Injecting model.layers.53.input_layernorm as default > Injecting model.layers.53.post_attention_layernorm as default > Injecting model.layers.54 as default > Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.q_a_layernorm as default > Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_layernorm as default > Injecting model.layers.54.self_attn.kv_b_proj as default > Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.54.mlp.shared_experts as default > Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.act_fn as default > Injecting model.layers.54.input_layernorm as default > Injecting model.layers.54.post_attention_layernorm as default > Injecting model.layers.55 as default > Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.q_a_layernorm as default > Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_layernorm as default > Injecting model.layers.55.self_attn.kv_b_proj as default > Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.55.mlp.shared_experts as default > Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.act_fn as default > Injecting model.layers.55.input_layernorm as default > Injecting model.layers.55.post_attention_layernorm as default > Injecting model.layers.56 as default > Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.q_a_layernorm as default > Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_layernorm as default > Injecting model.layers.56.self_attn.kv_b_proj as default > Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.56.mlp.shared_experts as default > Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.act_fn as default > Injecting model.layers.56.input_layernorm as default > Injecting model.layers.56.post_attention_layernorm as default > Injecting model.layers.57 as default > Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.q_a_layernorm as default > Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_layernorm as default > Injecting model.layers.57.self_attn.kv_b_proj as default > Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.57.mlp.shared_experts as default > Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.act_fn as default > Injecting model.layers.57.input_layernorm as default > Injecting model.layers.57.post_attention_layernorm as default > Injecting model.layers.58 as default > Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.q_a_layernorm as default > Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_layernorm as default > Injecting model.layers.58.self_attn.kv_b_proj as default > Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.58.mlp.shared_experts as default > Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.act_fn as default > Injecting model.layers.58.input_layernorm as default > Injecting model.layers.58.post_attention_layernorm as default > Injecting model.layers.59 as default > Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.q_a_layernorm as default > Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_layernorm as default > Injecting model.layers.59.self_attn.kv_b_proj as default > Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.59.mlp.shared_experts as default > Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.act_fn as default > Injecting model.layers.59.input_layernorm as default > Injecting model.layers.59.post_attention_layernorm as default > Injecting model.layers.60 as default > Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > /opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.) > data = torch.from_numpy(data) > Injecting model.layers.60.self_attn.q_a_layernorm as default > Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_layernorm as default > Injecting model.layers.60.self_attn.kv_b_proj as default > Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.60.mlp.shared_experts as default > Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.act_fn as default > Injecting model.layers.60.input_layernorm as default > Injecting model.layers.60.post_attention_layernorm as default > Injecting model.norm as default > Injecting lm_head as default > loading token_embd.weight to cpu > loading blk.0.attn_q_a_norm.weight to cuda:0 > loading blk.0.attn_kv_a_norm.weight to cuda:0 > loading blk.0.attn_kv_b.weight to cuda:0 > Traceback (most recent call last): > File "/opt/conda/bin/ktransformers", line 8, in <module> > sys.exit(main()) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main > create_interface(config=cfg, default_args=cfg) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface > GlobalInterface.interface = BackendInterface(default_args) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ > optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf > load_weights(module, gguf_loader) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights > load_cur_state_dict(module, gguf_loader, prefix) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict > weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype) > RuntimeError: CUDA error: no kernel image is available for execution on the device > CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. > For debugging consider passing CUDA_LAUNCH_BLOCKING=1. > Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. > ``` 建议你不要使用whl，直接在main上build试试，然后使用export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6。再检查下检查cuda和torch版本啥的
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@zhizi42 commented on GitHub (Feb 18, 2025):

@moneyisallyouneed 感谢你。GLIBCxxx的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。

root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output

�[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m
�[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.q_a_layernorm as default
Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_layernorm as default
Injecting model.layers.9.self_attn.kv_b_proj as default
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.9.mlp.shared_experts as default
Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.act_fn as default
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.q_a_layernorm as default
Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_layernorm as default
Injecting model.layers.10.self_attn.kv_b_proj as default
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.10.mlp.shared_experts as default
Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.act_fn as default
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.q_a_layernorm as default
Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_layernorm as default
Injecting model.layers.11.self_attn.kv_b_proj as default
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.11.mlp.shared_experts as default
Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.act_fn as default
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.q_a_layernorm as default
Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_layernorm as default
Injecting model.layers.12.self_attn.kv_b_proj as default
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.12.mlp.shared_experts as default
Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.act_fn as default
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.q_a_layernorm as default
Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_layernorm as default
Injecting model.layers.13.self_attn.kv_b_proj as default
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.13.mlp.shared_experts as default
Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.act_fn as default
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.q_a_layernorm as default
Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_layernorm as default
Injecting model.layers.14.self_attn.kv_b_proj as default
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.14.mlp.shared_experts as default
Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.act_fn as default
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.q_a_layernorm as default
Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_layernorm as default
Injecting model.layers.15.self_attn.kv_b_proj as default
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.15.mlp.shared_experts as default
Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.act_fn as default
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.q_a_layernorm as default
Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_layernorm as default
Injecting model.layers.16.self_attn.kv_b_proj as default
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.16.mlp.shared_experts as default
Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.act_fn as default
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.q_a_layernorm as default
Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_layernorm as default
Injecting model.layers.17.self_attn.kv_b_proj as default
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.17.mlp.shared_experts as default
Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.act_fn as default
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.q_a_layernorm as default
Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_layernorm as default
Injecting model.layers.18.self_attn.kv_b_proj as default
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.18.mlp.shared_experts as default
Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.act_fn as default
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.q_a_layernorm as default
Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_layernorm as default
Injecting model.layers.19.self_attn.kv_b_proj as default
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.19.mlp.shared_experts as default
Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.act_fn as default
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.q_a_layernorm as default
Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_layernorm as default
Injecting model.layers.20.self_attn.kv_b_proj as default
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.20.mlp.shared_experts as default
Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.act_fn as default
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.q_a_layernorm as default
Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_layernorm as default
Injecting model.layers.21.self_attn.kv_b_proj as default
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.21.mlp.shared_experts as default
Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.act_fn as default
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.q_a_layernorm as default
Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_layernorm as default
Injecting model.layers.22.self_attn.kv_b_proj as default
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.22.mlp.shared_experts as default
Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.act_fn as default
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.q_a_layernorm as default
Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_layernorm as default
Injecting model.layers.23.self_attn.kv_b_proj as default
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.23.mlp.shared_experts as default
Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.act_fn as default
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.q_a_layernorm as default
Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_layernorm as default
Injecting model.layers.24.self_attn.kv_b_proj as default
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.24.mlp.shared_experts as default
Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.act_fn as default
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.q_a_layernorm as default
Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_layernorm as default
Injecting model.layers.25.self_attn.kv_b_proj as default
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.25.mlp.shared_experts as default
Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.act_fn as default
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.q_a_layernorm as default
Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_layernorm as default
Injecting model.layers.26.self_attn.kv_b_proj as default
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.26.mlp.shared_experts as default
Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.act_fn as default
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.q_a_layernorm as default
Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_layernorm as default
Injecting model.layers.27.self_attn.kv_b_proj as default
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.27.mlp.shared_experts as default
Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.act_fn as default
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.q_a_layernorm as default
Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_layernorm as default
Injecting model.layers.28.self_attn.kv_b_proj as default
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.28.mlp.shared_experts as default
Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.act_fn as default
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.q_a_layernorm as default
Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_layernorm as default
Injecting model.layers.29.self_attn.kv_b_proj as default
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.29.mlp.shared_experts as default
Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.act_fn as default
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.q_a_layernorm as default
Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_layernorm as default
Injecting model.layers.30.self_attn.kv_b_proj as default
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.30.mlp.shared_experts as default
Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.act_fn as default
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.q_a_layernorm as default
Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_layernorm as default
Injecting model.layers.31.self_attn.kv_b_proj as default
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.31.mlp.shared_experts as default
Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.act_fn as default
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.layers.32 as default
Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.q_a_layernorm as default
Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_layernorm as default
Injecting model.layers.32.self_attn.kv_b_proj as default
Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.32.mlp.shared_experts as default
Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.act_fn as default
Injecting model.layers.32.input_layernorm as default
Injecting model.layers.32.post_attention_layernorm as default
Injecting model.layers.33 as default
Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.q_a_layernorm as default
Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_layernorm as default
Injecting model.layers.33.self_attn.kv_b_proj as default
Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.33.mlp.shared_experts as default
Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.act_fn as default
Injecting model.layers.33.input_layernorm as default
Injecting model.layers.33.post_attention_layernorm as default
Injecting model.layers.34 as default
Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.q_a_layernorm as default
Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_layernorm as default
Injecting model.layers.34.self_attn.kv_b_proj as default
Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.34.mlp.shared_experts as default
Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.act_fn as default
Injecting model.layers.34.input_layernorm as default
Injecting model.layers.34.post_attention_layernorm as default
Injecting model.layers.35 as default
Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.q_a_layernorm as default
Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_layernorm as default
Injecting model.layers.35.self_attn.kv_b_proj as default
Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.35.mlp.shared_experts as default
Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.act_fn as default
Injecting model.layers.35.input_layernorm as default
Injecting model.layers.35.post_attention_layernorm as default
Injecting model.layers.36 as default
Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.q_a_layernorm as default
Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_layernorm as default
Injecting model.layers.36.self_attn.kv_b_proj as default
Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.36.mlp.shared_experts as default
Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.act_fn as default
Injecting model.layers.36.input_layernorm as default
Injecting model.layers.36.post_attention_layernorm as default
Injecting model.layers.37 as default
Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.q_a_layernorm as default
Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_layernorm as default
Injecting model.layers.37.self_attn.kv_b_proj as default
Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.37.mlp.shared_experts as default
Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.act_fn as default
Injecting model.layers.37.input_layernorm as default
Injecting model.layers.37.post_attention_layernorm as default
Injecting model.layers.38 as default
Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.q_a_layernorm as default
Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_layernorm as default
Injecting model.layers.38.self_attn.kv_b_proj as default
Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.38.mlp.shared_experts as default
Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.act_fn as default
Injecting model.layers.38.input_layernorm as default
Injecting model.layers.38.post_attention_layernorm as default
Injecting model.layers.39 as default
Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.q_a_layernorm as default
Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_layernorm as default
Injecting model.layers.39.self_attn.kv_b_proj as default
Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.39.mlp.shared_experts as default
Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.act_fn as default
Injecting model.layers.39.input_layernorm as default
Injecting model.layers.39.post_attention_layernorm as default
Injecting model.layers.40 as default
Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.q_a_layernorm as default
Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_layernorm as default
Injecting model.layers.40.self_attn.kv_b_proj as default
Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.40.mlp.shared_experts as default
Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.act_fn as default
Injecting model.layers.40.input_layernorm as default
Injecting model.layers.40.post_attention_layernorm as default
Injecting model.layers.41 as default
Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.q_a_layernorm as default
Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_layernorm as default
Injecting model.layers.41.self_attn.kv_b_proj as default
Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.41.mlp.shared_experts as default
Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.act_fn as default
Injecting model.layers.41.input_layernorm as default
Injecting model.layers.41.post_attention_layernorm as default
Injecting model.layers.42 as default
Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.q_a_layernorm as default
Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_layernorm as default
Injecting model.layers.42.self_attn.kv_b_proj as default
Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.42.mlp.shared_experts as default
Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.act_fn as default
Injecting model.layers.42.input_layernorm as default
Injecting model.layers.42.post_attention_layernorm as default
Injecting model.layers.43 as default
Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.q_a_layernorm as default
Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_layernorm as default
Injecting model.layers.43.self_attn.kv_b_proj as default
Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.43.mlp.shared_experts as default
Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.act_fn as default
Injecting model.layers.43.input_layernorm as default
Injecting model.layers.43.post_attention_layernorm as default
Injecting model.layers.44 as default
Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.q_a_layernorm as default
Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_layernorm as default
Injecting model.layers.44.self_attn.kv_b_proj as default
Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.44.mlp.shared_experts as default
Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.act_fn as default
Injecting model.layers.44.input_layernorm as default
Injecting model.layers.44.post_attention_layernorm as default
Injecting model.layers.45 as default
Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.q_a_layernorm as default
Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_layernorm as default
Injecting model.layers.45.self_attn.kv_b_proj as default
Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.45.mlp.shared_experts as default
Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.act_fn as default
Injecting model.layers.45.input_layernorm as default
Injecting model.layers.45.post_attention_layernorm as default
Injecting model.layers.46 as default
Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.q_a_layernorm as default
Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_layernorm as default
Injecting model.layers.46.self_attn.kv_b_proj as default
Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.46.mlp.shared_experts as default
Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.act_fn as default
Injecting model.layers.46.input_layernorm as default
Injecting model.layers.46.post_attention_layernorm as default
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_a_layernorm as default
Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_layernorm as default
Injecting model.layers.47.self_attn.kv_b_proj as default
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.47.mlp.shared_experts as default
Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.act_fn as default
Injecting model.layers.47.input_layernorm as default
Injecting model.layers.47.post_attention_layernorm as default
Injecting model.layers.48 as default
Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.q_a_layernorm as default
Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_layernorm as default
Injecting model.layers.48.self_attn.kv_b_proj as default
Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.48.mlp.shared_experts as default
Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.act_fn as default
Injecting model.layers.48.input_layernorm as default
Injecting model.layers.48.post_attention_layernorm as default
Injecting model.layers.49 as default
Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.q_a_layernorm as default
Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_layernorm as default
Injecting model.layers.49.self_attn.kv_b_proj as default
Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.49.mlp.shared_experts as default
Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.act_fn as default
Injecting model.layers.49.input_layernorm as default
Injecting model.layers.49.post_attention_layernorm as default
Injecting model.layers.50 as default
Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.q_a_layernorm as default
Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_layernorm as default
Injecting model.layers.50.self_attn.kv_b_proj as default
Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.50.mlp.shared_experts as default
Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.act_fn as default
Injecting model.layers.50.input_layernorm as default
Injecting model.layers.50.post_attention_layernorm as default
Injecting model.layers.51 as default
Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.q_a_layernorm as default
Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_layernorm as default
Injecting model.layers.51.self_attn.kv_b_proj as default
Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.51.mlp.shared_experts as default
Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.act_fn as default
Injecting model.layers.51.input_layernorm as default
Injecting model.layers.51.post_attention_layernorm as default
Injecting model.layers.52 as default
Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.q_a_layernorm as default
Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_layernorm as default
Injecting model.layers.52.self_attn.kv_b_proj as default
Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.52.mlp.shared_experts as default
Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.act_fn as default
Injecting model.layers.52.input_layernorm as default
Injecting model.layers.52.post_attention_layernorm as default
Injecting model.layers.53 as default
Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.q_a_layernorm as default
Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_layernorm as default
Injecting model.layers.53.self_attn.kv_b_proj as default
Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.53.mlp.shared_experts as default
Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.act_fn as default
Injecting model.layers.53.input_layernorm as default
Injecting model.layers.53.post_attention_layernorm as default
Injecting model.layers.54 as default
Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.q_a_layernorm as default
Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_layernorm as default
Injecting model.layers.54.self_attn.kv_b_proj as default
Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.54.mlp.shared_experts as default
Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.act_fn as default
Injecting model.layers.54.input_layernorm as default
Injecting model.layers.54.post_attention_layernorm as default
Injecting model.layers.55 as default
Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.q_a_layernorm as default
Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_layernorm as default
Injecting model.layers.55.self_attn.kv_b_proj as default
Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.55.mlp.shared_experts as default
Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.act_fn as default
Injecting model.layers.55.input_layernorm as default
Injecting model.layers.55.post_attention_layernorm as default
Injecting model.layers.56 as default
Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.q_a_layernorm as default
Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_layernorm as default
Injecting model.layers.56.self_attn.kv_b_proj as default
Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.56.mlp.shared_experts as default
Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.act_fn as default
Injecting model.layers.56.input_layernorm as default
Injecting model.layers.56.post_attention_layernorm as default
Injecting model.layers.57 as default
Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.q_a_layernorm as default
Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_layernorm as default
Injecting model.layers.57.self_attn.kv_b_proj as default
Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.57.mlp.shared_experts as default
Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.act_fn as default
Injecting model.layers.57.input_layernorm as default
Injecting model.layers.57.post_attention_layernorm as default
Injecting model.layers.58 as default
Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.q_a_layernorm as default
Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_layernorm as default
Injecting model.layers.58.self_attn.kv_b_proj as default
Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.58.mlp.shared_experts as default
Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.act_fn as default
Injecting model.layers.58.input_layernorm as default
Injecting model.layers.58.post_attention_layernorm as default
Injecting model.layers.59 as default
Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.q_a_layernorm as default
Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_layernorm as default
Injecting model.layers.59.self_attn.kv_b_proj as default
Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.59.mlp.shared_experts as default
Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.act_fn as default
Injecting model.layers.59.input_layernorm as default
Injecting model.layers.59.post_attention_layernorm as default
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
/opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.)
  data = torch.from_numpy(data)
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict
    weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

看上去相同的问题。
装的torch是cu124，2.4.1的，cuda是12.4.1，flash_attn是cu124torch2.4.0
gpu是2070super

  data = torch.from_numpy(data)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\local_chat.py", line 278, in <module>
    fire.Fire(local_chat)
  File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\local_chat.py", line 215, in local_chat
    optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\optimize\optimize.py", line 129, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 85, in load_weights
    module.load()
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  [Previous line repeated 1 more time]
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 85, in load_weights
    module.load()
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 422, in load
    self.generate_linear.load(w=w)
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 138, in load
    if w is None: w = self.load_weight(device=device)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 80, in load_weight
    tensors = self.load_multi(key, ["weight", "bias"], device=device)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 97, in load_multi
    tensors[k] = self.gguf_loader.load_gguf_tensor(key + "." + k, device=device)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\custom_gguf.py", line 293, in load_gguf_tensor
    values = GGML_DEQUANTIZE_GPU[ggml_name](data, device)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\custom_gguf.py", line 714, in dequantize_f32_gpu
    res_gpu = torch.empty_like(res, device=device)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.```
<!-- gh-comment-id:2665783225 --> @zhizi42 commented on GitHub (Feb 18, 2025): > [@moneyisallyouneed](https://github.com/moneyisallyouneed) 感谢你。`GLIBCxxx`的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。 > > ``` > root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output > > �[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m > �[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m > Injecting model as ktransformers.operators.models . KDeepseekV2Model > Injecting model.embed_tokens as default > Injecting model.layers as default > Injecting model.layers.0 as default > Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.q_a_layernorm as default > Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_layernorm as default > Injecting model.layers.0.self_attn.kv_b_proj as default > Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.0.mlp as default > Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.act_fn as default > Injecting model.layers.0.input_layernorm as default > Injecting model.layers.0.post_attention_layernorm as default > Injecting model.layers.1 as default > Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.q_a_layernorm as default > Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_layernorm as default > Injecting model.layers.1.self_attn.kv_b_proj as default > Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.1.mlp as default > Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.act_fn as default > Injecting model.layers.1.input_layernorm as default > Injecting model.layers.1.post_attention_layernorm as default > Injecting model.layers.2 as default > Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.q_a_layernorm as default > Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_layernorm as default > Injecting model.layers.2.self_attn.kv_b_proj as default > Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.2.mlp as default > Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.act_fn as default > Injecting model.layers.2.input_layernorm as default > Injecting model.layers.2.post_attention_layernorm as default > Injecting model.layers.3 as default > Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.q_a_layernorm as default > Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_layernorm as default > Injecting model.layers.3.self_attn.kv_b_proj as default > Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.3.mlp.shared_experts as default > Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.act_fn as default > Injecting model.layers.3.input_layernorm as default > Injecting model.layers.3.post_attention_layernorm as default > Injecting model.layers.4 as default > Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.q_a_layernorm as default > Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_layernorm as default > Injecting model.layers.4.self_attn.kv_b_proj as default > Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.4.mlp.shared_experts as default > Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.act_fn as default > Injecting model.layers.4.input_layernorm as default > Injecting model.layers.4.post_attention_layernorm as default > Injecting model.layers.5 as default > Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.q_a_layernorm as default > Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_layernorm as default > Injecting model.layers.5.self_attn.kv_b_proj as default > Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.5.mlp.shared_experts as default > Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.act_fn as default > Injecting model.layers.5.input_layernorm as default > Injecting model.layers.5.post_attention_layernorm as default > Injecting model.layers.6 as default > Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.q_a_layernorm as default > Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_layernorm as default > Injecting model.layers.6.self_attn.kv_b_proj as default > Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.6.mlp.shared_experts as default > Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.act_fn as default > Injecting model.layers.6.input_layernorm as default > Injecting model.layers.6.post_attention_layernorm as default > Injecting model.layers.7 as default > Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.q_a_layernorm as default > Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_layernorm as default > Injecting model.layers.7.self_attn.kv_b_proj as default > Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.7.mlp.shared_experts as default > Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.act_fn as default > Injecting model.layers.7.input_layernorm as default > Injecting model.layers.7.post_attention_layernorm as default > Injecting model.layers.8 as default > Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.q_a_layernorm as default > Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_layernorm as default > Injecting model.layers.8.self_attn.kv_b_proj as default > Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.8.mlp.shared_experts as default > Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.act_fn as default > Injecting model.layers.8.input_layernorm as default > Injecting model.layers.8.post_attention_layernorm as default > Injecting model.layers.9 as default > Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.q_a_layernorm as default > Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_layernorm as default > Injecting model.layers.9.self_attn.kv_b_proj as default > Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.9.mlp.shared_experts as default > Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.act_fn as default > Injecting model.layers.9.input_layernorm as default > Injecting model.layers.9.post_attention_layernorm as default > Injecting model.layers.10 as default > Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.q_a_layernorm as default > Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_layernorm as default > Injecting model.layers.10.self_attn.kv_b_proj as default > Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.10.mlp.shared_experts as default > Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.act_fn as default > Injecting model.layers.10.input_layernorm as default > Injecting model.layers.10.post_attention_layernorm as default > Injecting model.layers.11 as default > Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.q_a_layernorm as default > Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_layernorm as default > Injecting model.layers.11.self_attn.kv_b_proj as default > Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.11.mlp.shared_experts as default > Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.act_fn as default > Injecting model.layers.11.input_layernorm as default > Injecting model.layers.11.post_attention_layernorm as default > Injecting model.layers.12 as default > Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.q_a_layernorm as default > Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_layernorm as default > Injecting model.layers.12.self_attn.kv_b_proj as default > Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.12.mlp.shared_experts as default > Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.act_fn as default > Injecting model.layers.12.input_layernorm as default > Injecting model.layers.12.post_attention_layernorm as default > Injecting model.layers.13 as default > Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.q_a_layernorm as default > Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_layernorm as default > Injecting model.layers.13.self_attn.kv_b_proj as default > Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.13.mlp.shared_experts as default > Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.act_fn as default > Injecting model.layers.13.input_layernorm as default > Injecting model.layers.13.post_attention_layernorm as default > Injecting model.layers.14 as default > Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.q_a_layernorm as default > Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_layernorm as default > Injecting model.layers.14.self_attn.kv_b_proj as default > Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.14.mlp.shared_experts as default > Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.act_fn as default > Injecting model.layers.14.input_layernorm as default > Injecting model.layers.14.post_attention_layernorm as default > Injecting model.layers.15 as default > Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.q_a_layernorm as default > Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_layernorm as default > Injecting model.layers.15.self_attn.kv_b_proj as default > Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.15.mlp.shared_experts as default > Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.act_fn as default > Injecting model.layers.15.input_layernorm as default > Injecting model.layers.15.post_attention_layernorm as default > Injecting model.layers.16 as default > Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.q_a_layernorm as default > Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_layernorm as default > Injecting model.layers.16.self_attn.kv_b_proj as default > Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.16.mlp.shared_experts as default > Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.act_fn as default > Injecting model.layers.16.input_layernorm as default > Injecting model.layers.16.post_attention_layernorm as default > Injecting model.layers.17 as default > Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.q_a_layernorm as default > Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_layernorm as default > Injecting model.layers.17.self_attn.kv_b_proj as default > Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.17.mlp.shared_experts as default > Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.act_fn as default > Injecting model.layers.17.input_layernorm as default > Injecting model.layers.17.post_attention_layernorm as default > Injecting model.layers.18 as default > Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.q_a_layernorm as default > Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_layernorm as default > Injecting model.layers.18.self_attn.kv_b_proj as default > Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.18.mlp.shared_experts as default > Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.act_fn as default > Injecting model.layers.18.input_layernorm as default > Injecting model.layers.18.post_attention_layernorm as default > Injecting model.layers.19 as default > Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.q_a_layernorm as default > Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_layernorm as default > Injecting model.layers.19.self_attn.kv_b_proj as default > Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.19.mlp.shared_experts as default > Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.act_fn as default > Injecting model.layers.19.input_layernorm as default > Injecting model.layers.19.post_attention_layernorm as default > Injecting model.layers.20 as default > Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.q_a_layernorm as default > Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_layernorm as default > Injecting model.layers.20.self_attn.kv_b_proj as default > Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.20.mlp.shared_experts as default > Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.act_fn as default > Injecting model.layers.20.input_layernorm as default > Injecting model.layers.20.post_attention_layernorm as default > Injecting model.layers.21 as default > Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.q_a_layernorm as default > Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_layernorm as default > Injecting model.layers.21.self_attn.kv_b_proj as default > Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.21.mlp.shared_experts as default > Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.act_fn as default > Injecting model.layers.21.input_layernorm as default > Injecting model.layers.21.post_attention_layernorm as default > Injecting model.layers.22 as default > Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.q_a_layernorm as default > Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_layernorm as default > Injecting model.layers.22.self_attn.kv_b_proj as default > Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.22.mlp.shared_experts as default > Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.act_fn as default > Injecting model.layers.22.input_layernorm as default > Injecting model.layers.22.post_attention_layernorm as default > Injecting model.layers.23 as default > Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.q_a_layernorm as default > Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_layernorm as default > Injecting model.layers.23.self_attn.kv_b_proj as default > Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.23.mlp.shared_experts as default > Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.act_fn as default > Injecting model.layers.23.input_layernorm as default > Injecting model.layers.23.post_attention_layernorm as default > Injecting model.layers.24 as default > Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.q_a_layernorm as default > Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_layernorm as default > Injecting model.layers.24.self_attn.kv_b_proj as default > Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.24.mlp.shared_experts as default > Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.act_fn as default > Injecting model.layers.24.input_layernorm as default > Injecting model.layers.24.post_attention_layernorm as default > Injecting model.layers.25 as default > Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.q_a_layernorm as default > Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_layernorm as default > Injecting model.layers.25.self_attn.kv_b_proj as default > Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.25.mlp.shared_experts as default > Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.act_fn as default > Injecting model.layers.25.input_layernorm as default > Injecting model.layers.25.post_attention_layernorm as default > Injecting model.layers.26 as default > Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.q_a_layernorm as default > Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_layernorm as default > Injecting model.layers.26.self_attn.kv_b_proj as default > Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.26.mlp.shared_experts as default > Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.act_fn as default > Injecting model.layers.26.input_layernorm as default > Injecting model.layers.26.post_attention_layernorm as default > Injecting model.layers.27 as default > Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.q_a_layernorm as default > Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_layernorm as default > Injecting model.layers.27.self_attn.kv_b_proj as default > Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.27.mlp.shared_experts as default > Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.act_fn as default > Injecting model.layers.27.input_layernorm as default > Injecting model.layers.27.post_attention_layernorm as default > Injecting model.layers.28 as default > Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.q_a_layernorm as default > Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_layernorm as default > Injecting model.layers.28.self_attn.kv_b_proj as default > Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.28.mlp.shared_experts as default > Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.act_fn as default > Injecting model.layers.28.input_layernorm as default > Injecting model.layers.28.post_attention_layernorm as default > Injecting model.layers.29 as default > Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.q_a_layernorm as default > Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_layernorm as default > Injecting model.layers.29.self_attn.kv_b_proj as default > Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.29.mlp.shared_experts as default > Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.act_fn as default > Injecting model.layers.29.input_layernorm as default > Injecting model.layers.29.post_attention_layernorm as default > Injecting model.layers.30 as default > Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.q_a_layernorm as default > Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_layernorm as default > Injecting model.layers.30.self_attn.kv_b_proj as default > Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.30.mlp.shared_experts as default > Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.act_fn as default > Injecting model.layers.30.input_layernorm as default > Injecting model.layers.30.post_attention_layernorm as default > Injecting model.layers.31 as default > Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.q_a_layernorm as default > Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_layernorm as default > Injecting model.layers.31.self_attn.kv_b_proj as default > Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.31.mlp.shared_experts as default > Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.act_fn as default > Injecting model.layers.31.input_layernorm as default > Injecting model.layers.31.post_attention_layernorm as default > Injecting model.layers.32 as default > Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.q_a_layernorm as default > Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_layernorm as default > Injecting model.layers.32.self_attn.kv_b_proj as default > Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.32.mlp.shared_experts as default > Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.act_fn as default > Injecting model.layers.32.input_layernorm as default > Injecting model.layers.32.post_attention_layernorm as default > Injecting model.layers.33 as default > Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.q_a_layernorm as default > Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_layernorm as default > Injecting model.layers.33.self_attn.kv_b_proj as default > Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.33.mlp.shared_experts as default > Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.act_fn as default > Injecting model.layers.33.input_layernorm as default > Injecting model.layers.33.post_attention_layernorm as default > Injecting model.layers.34 as default > Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.q_a_layernorm as default > Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_layernorm as default > Injecting model.layers.34.self_attn.kv_b_proj as default > Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.34.mlp.shared_experts as default > Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.act_fn as default > Injecting model.layers.34.input_layernorm as default > Injecting model.layers.34.post_attention_layernorm as default > Injecting model.layers.35 as default > Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.q_a_layernorm as default > Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_layernorm as default > Injecting model.layers.35.self_attn.kv_b_proj as default > Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.35.mlp.shared_experts as default > Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.act_fn as default > Injecting model.layers.35.input_layernorm as default > Injecting model.layers.35.post_attention_layernorm as default > Injecting model.layers.36 as default > Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.q_a_layernorm as default > Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_layernorm as default > Injecting model.layers.36.self_attn.kv_b_proj as default > Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.36.mlp.shared_experts as default > Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.act_fn as default > Injecting model.layers.36.input_layernorm as default > Injecting model.layers.36.post_attention_layernorm as default > Injecting model.layers.37 as default > Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.q_a_layernorm as default > Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_layernorm as default > Injecting model.layers.37.self_attn.kv_b_proj as default > Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.37.mlp.shared_experts as default > Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.act_fn as default > Injecting model.layers.37.input_layernorm as default > Injecting model.layers.37.post_attention_layernorm as default > Injecting model.layers.38 as default > Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.q_a_layernorm as default > Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_layernorm as default > Injecting model.layers.38.self_attn.kv_b_proj as default > Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.38.mlp.shared_experts as default > Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.act_fn as default > Injecting model.layers.38.input_layernorm as default > Injecting model.layers.38.post_attention_layernorm as default > Injecting model.layers.39 as default > Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.q_a_layernorm as default > Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_layernorm as default > Injecting model.layers.39.self_attn.kv_b_proj as default > Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.39.mlp.shared_experts as default > Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.act_fn as default > Injecting model.layers.39.input_layernorm as default > Injecting model.layers.39.post_attention_layernorm as default > Injecting model.layers.40 as default > Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.q_a_layernorm as default > Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_layernorm as default > Injecting model.layers.40.self_attn.kv_b_proj as default > Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.40.mlp.shared_experts as default > Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.act_fn as default > Injecting model.layers.40.input_layernorm as default > Injecting model.layers.40.post_attention_layernorm as default > Injecting model.layers.41 as default > Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.q_a_layernorm as default > Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_layernorm as default > Injecting model.layers.41.self_attn.kv_b_proj as default > Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.41.mlp.shared_experts as default > Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.act_fn as default > Injecting model.layers.41.input_layernorm as default > Injecting model.layers.41.post_attention_layernorm as default > Injecting model.layers.42 as default > Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.q_a_layernorm as default > Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_layernorm as default > Injecting model.layers.42.self_attn.kv_b_proj as default > Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.42.mlp.shared_experts as default > Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.act_fn as default > Injecting model.layers.42.input_layernorm as default > Injecting model.layers.42.post_attention_layernorm as default > Injecting model.layers.43 as default > Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.q_a_layernorm as default > Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_layernorm as default > Injecting model.layers.43.self_attn.kv_b_proj as default > Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.43.mlp.shared_experts as default > Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.act_fn as default > Injecting model.layers.43.input_layernorm as default > Injecting model.layers.43.post_attention_layernorm as default > Injecting model.layers.44 as default > Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.q_a_layernorm as default > Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_layernorm as default > Injecting model.layers.44.self_attn.kv_b_proj as default > Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.44.mlp.shared_experts as default > Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.act_fn as default > Injecting model.layers.44.input_layernorm as default > Injecting model.layers.44.post_attention_layernorm as default > Injecting model.layers.45 as default > Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.q_a_layernorm as default > Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_layernorm as default > Injecting model.layers.45.self_attn.kv_b_proj as default > Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.45.mlp.shared_experts as default > Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.act_fn as default > Injecting model.layers.45.input_layernorm as default > Injecting model.layers.45.post_attention_layernorm as default > Injecting model.layers.46 as default > Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.q_a_layernorm as default > Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_layernorm as default > Injecting model.layers.46.self_attn.kv_b_proj as default > Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.46.mlp.shared_experts as default > Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.act_fn as default > Injecting model.layers.46.input_layernorm as default > Injecting model.layers.46.post_attention_layernorm as default > Injecting model.layers.47 as default > Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.q_a_layernorm as default > Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_layernorm as default > Injecting model.layers.47.self_attn.kv_b_proj as default > Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.47.mlp.shared_experts as default > Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.act_fn as default > Injecting model.layers.47.input_layernorm as default > Injecting model.layers.47.post_attention_layernorm as default > Injecting model.layers.48 as default > Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.q_a_layernorm as default > Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_layernorm as default > Injecting model.layers.48.self_attn.kv_b_proj as default > Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.48.mlp.shared_experts as default > Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.act_fn as default > Injecting model.layers.48.input_layernorm as default > Injecting model.layers.48.post_attention_layernorm as default > Injecting model.layers.49 as default > Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.q_a_layernorm as default > Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_layernorm as default > Injecting model.layers.49.self_attn.kv_b_proj as default > Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.49.mlp.shared_experts as default > Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.act_fn as default > Injecting model.layers.49.input_layernorm as default > Injecting model.layers.49.post_attention_layernorm as default > Injecting model.layers.50 as default > Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.q_a_layernorm as default > Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_layernorm as default > Injecting model.layers.50.self_attn.kv_b_proj as default > Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.50.mlp.shared_experts as default > Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.act_fn as default > Injecting model.layers.50.input_layernorm as default > Injecting model.layers.50.post_attention_layernorm as default > Injecting model.layers.51 as default > Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.q_a_layernorm as default > Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_layernorm as default > Injecting model.layers.51.self_attn.kv_b_proj as default > Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.51.mlp.shared_experts as default > Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.act_fn as default > Injecting model.layers.51.input_layernorm as default > Injecting model.layers.51.post_attention_layernorm as default > Injecting model.layers.52 as default > Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.q_a_layernorm as default > Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_layernorm as default > Injecting model.layers.52.self_attn.kv_b_proj as default > Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.52.mlp.shared_experts as default > Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.act_fn as default > Injecting model.layers.52.input_layernorm as default > Injecting model.layers.52.post_attention_layernorm as default > Injecting model.layers.53 as default > Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.q_a_layernorm as default > Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_layernorm as default > Injecting model.layers.53.self_attn.kv_b_proj as default > Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.53.mlp.shared_experts as default > Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.act_fn as default > Injecting model.layers.53.input_layernorm as default > Injecting model.layers.53.post_attention_layernorm as default > Injecting model.layers.54 as default > Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.q_a_layernorm as default > Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_layernorm as default > Injecting model.layers.54.self_attn.kv_b_proj as default > Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.54.mlp.shared_experts as default > Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.act_fn as default > Injecting model.layers.54.input_layernorm as default > Injecting model.layers.54.post_attention_layernorm as default > Injecting model.layers.55 as default > Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.q_a_layernorm as default > Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_layernorm as default > Injecting model.layers.55.self_attn.kv_b_proj as default > Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.55.mlp.shared_experts as default > Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.act_fn as default > Injecting model.layers.55.input_layernorm as default > Injecting model.layers.55.post_attention_layernorm as default > Injecting model.layers.56 as default > Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.q_a_layernorm as default > Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_layernorm as default > Injecting model.layers.56.self_attn.kv_b_proj as default > Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.56.mlp.shared_experts as default > Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.act_fn as default > Injecting model.layers.56.input_layernorm as default > Injecting model.layers.56.post_attention_layernorm as default > Injecting model.layers.57 as default > Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.q_a_layernorm as default > Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_layernorm as default > Injecting model.layers.57.self_attn.kv_b_proj as default > Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.57.mlp.shared_experts as default > Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.act_fn as default > Injecting model.layers.57.input_layernorm as default > Injecting model.layers.57.post_attention_layernorm as default > Injecting model.layers.58 as default > Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.q_a_layernorm as default > Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_layernorm as default > Injecting model.layers.58.self_attn.kv_b_proj as default > Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.58.mlp.shared_experts as default > Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.act_fn as default > Injecting model.layers.58.input_layernorm as default > Injecting model.layers.58.post_attention_layernorm as default > Injecting model.layers.59 as default > Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.q_a_layernorm as default > Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_layernorm as default > Injecting model.layers.59.self_attn.kv_b_proj as default > Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.59.mlp.shared_experts as default > Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.act_fn as default > Injecting model.layers.59.input_layernorm as default > Injecting model.layers.59.post_attention_layernorm as default > Injecting model.layers.60 as default > Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > /opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.) > data = torch.from_numpy(data) > Injecting model.layers.60.self_attn.q_a_layernorm as default > Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_layernorm as default > Injecting model.layers.60.self_attn.kv_b_proj as default > Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.60.mlp.shared_experts as default > Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.act_fn as default > Injecting model.layers.60.input_layernorm as default > Injecting model.layers.60.post_attention_layernorm as default > Injecting model.norm as default > Injecting lm_head as default > loading token_embd.weight to cpu > loading blk.0.attn_q_a_norm.weight to cuda:0 > loading blk.0.attn_kv_a_norm.weight to cuda:0 > loading blk.0.attn_kv_b.weight to cuda:0 > Traceback (most recent call last): > File "/opt/conda/bin/ktransformers", line 8, in <module> > sys.exit(main()) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main > create_interface(config=cfg, default_args=cfg) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface > GlobalInterface.interface = BackendInterface(default_args) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ > optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf > load_weights(module, gguf_loader) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights > load_cur_state_dict(module, gguf_loader, prefix) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict > weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype) > RuntimeError: CUDA error: no kernel image is available for execution on the device > CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. > For debugging consider passing CUDA_LAUNCH_BLOCKING=1. > Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. > ``` 看上去相同的问题。 装的torch是cu124，2.4.1的，cuda是12.4.1，flash_attn是cu124torch2.4.0 gpu是2070super ```C:\Program Files\Python311\Lib\site-packages\ktransformers\util\custom_gguf.py:482: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:212.) data = torch.from_numpy(data) Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Program Files\Python311\Lib\site-packages\ktransformers\local_chat.py", line 278, in <module> fire.Fire(local_chat) File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\ktransformers\local_chat.py", line 215, in local_chat optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config) File "C:\Program Files\Python311\Lib\site-packages\ktransformers\optimize\optimize.py", line 129, in optimize_and_load_gguf load_weights(module, gguf_loader) File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 85, in load_weights module.load() File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\base_operator.py", line 60, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") [Previous line repeated 1 more time] File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\utils.py", line 85, in load_weights module.load() File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 422, in load self.generate_linear.load(w=w) File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 138, in load if w is None: w = self.load_weight(device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 80, in load_weight tensors = self.load_multi(key, ["weight", "bias"], device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\ktransformers\operators\linear.py", line 97, in load_multi tensors[k] = self.gguf_loader.load_gguf_tensor(key + "." + k, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\custom_gguf.py", line 293, in load_gguf_tensor values = GGML_DEQUANTIZE_GPU[ggml_name](data, device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\site-packages\ktransformers\util\custom_gguf.py", line 714, in dequantize_f32_gpu res_gpu = torch.empty_like(res, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.```
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@xiongsp commented on GitHub (Feb 18, 2025):

我已经成功跑起来了，贴一下解决的帖子。我用的环境是chatteohr/ktransformers这个镜像

  1. GLIBCxx not found：export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
  2. no kernel...：#374、https://github.com/kvcache-ai/ktransformers/issues/278#issuecomment-2658674519
<!-- gh-comment-id:2665807954 --> @xiongsp commented on GitHub (Feb 18, 2025): 我已经成功跑起来了，贴一下解决的帖子。我用的环境是`chatteohr/ktransformers`这个镜像 1. `GLIBCxx not found`：`export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6` 2. `no kernel...`：#374、https://github.com/kvcache-ai/ktransformers/issues/278#issuecomment-2658674519
bsakaguchi commented 2026-10-29 14:35:40 +00:00
Author
Owner
Copy link

@montagetao commented on GitHub (Feb 20, 2025):

GLIBCXX not found这个问题我是通过这样解决的，不知道你能不能用： export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6

我也是借鉴了你的方法解决，谢谢

<!-- gh-comment-id:2670123212 --> @montagetao commented on GitHub (Feb 20, 2025): > GLIBCXX not found这个问题我是通过这样解决的，不知道你能不能用： export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 我也是借鉴了你的方法解决，谢谢
bsakaguchi commented 2026-10-29 14:35:41 +00:00
Author
Owner
Copy link

@jercylew commented on GitHub (Nov 10, 2025):

@moneyisallyouneed 感谢你。GLIBCxxx的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。

root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output

�[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m
�[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.q_a_layernorm as default
Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_layernorm as default
Injecting model.layers.9.self_attn.kv_b_proj as default
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.9.mlp.shared_experts as default
Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.act_fn as default
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.q_a_layernorm as default
Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_layernorm as default
Injecting model.layers.10.self_attn.kv_b_proj as default
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.10.mlp.shared_experts as default
Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.act_fn as default
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.q_a_layernorm as default
Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_layernorm as default
Injecting model.layers.11.self_attn.kv_b_proj as default
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.11.mlp.shared_experts as default
Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.act_fn as default
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.q_a_layernorm as default
Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_layernorm as default
Injecting model.layers.12.self_attn.kv_b_proj as default
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.12.mlp.shared_experts as default
Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.act_fn as default
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.q_a_layernorm as default
Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_layernorm as default
Injecting model.layers.13.self_attn.kv_b_proj as default
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.13.mlp.shared_experts as default
Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.act_fn as default
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.q_a_layernorm as default
Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_layernorm as default
Injecting model.layers.14.self_attn.kv_b_proj as default
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.14.mlp.shared_experts as default
Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.act_fn as default
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.q_a_layernorm as default
Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_layernorm as default
Injecting model.layers.15.self_attn.kv_b_proj as default
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.15.mlp.shared_experts as default
Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.act_fn as default
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.q_a_layernorm as default
Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_layernorm as default
Injecting model.layers.16.self_attn.kv_b_proj as default
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.16.mlp.shared_experts as default
Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.act_fn as default
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.q_a_layernorm as default
Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_layernorm as default
Injecting model.layers.17.self_attn.kv_b_proj as default
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.17.mlp.shared_experts as default
Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.act_fn as default
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.q_a_layernorm as default
Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_layernorm as default
Injecting model.layers.18.self_attn.kv_b_proj as default
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.18.mlp.shared_experts as default
Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.act_fn as default
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.q_a_layernorm as default
Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_layernorm as default
Injecting model.layers.19.self_attn.kv_b_proj as default
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.19.mlp.shared_experts as default
Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.act_fn as default
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.q_a_layernorm as default
Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_layernorm as default
Injecting model.layers.20.self_attn.kv_b_proj as default
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.20.mlp.shared_experts as default
Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.act_fn as default
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.q_a_layernorm as default
Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_layernorm as default
Injecting model.layers.21.self_attn.kv_b_proj as default
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.21.mlp.shared_experts as default
Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.act_fn as default
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.q_a_layernorm as default
Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_layernorm as default
Injecting model.layers.22.self_attn.kv_b_proj as default
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.22.mlp.shared_experts as default
Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.act_fn as default
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.q_a_layernorm as default
Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_layernorm as default
Injecting model.layers.23.self_attn.kv_b_proj as default
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.23.mlp.shared_experts as default
Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.act_fn as default
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.q_a_layernorm as default
Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_layernorm as default
Injecting model.layers.24.self_attn.kv_b_proj as default
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.24.mlp.shared_experts as default
Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.act_fn as default
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.q_a_layernorm as default
Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_layernorm as default
Injecting model.layers.25.self_attn.kv_b_proj as default
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.25.mlp.shared_experts as default
Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.act_fn as default
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.q_a_layernorm as default
Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_layernorm as default
Injecting model.layers.26.self_attn.kv_b_proj as default
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.26.mlp.shared_experts as default
Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.act_fn as default
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.q_a_layernorm as default
Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_layernorm as default
Injecting model.layers.27.self_attn.kv_b_proj as default
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.27.mlp.shared_experts as default
Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.act_fn as default
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.q_a_layernorm as default
Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_layernorm as default
Injecting model.layers.28.self_attn.kv_b_proj as default
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.28.mlp.shared_experts as default
Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.act_fn as default
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.q_a_layernorm as default
Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_layernorm as default
Injecting model.layers.29.self_attn.kv_b_proj as default
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.29.mlp.shared_experts as default
Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.act_fn as default
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.q_a_layernorm as default
Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_layernorm as default
Injecting model.layers.30.self_attn.kv_b_proj as default
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.30.mlp.shared_experts as default
Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.act_fn as default
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.q_a_layernorm as default
Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_layernorm as default
Injecting model.layers.31.self_attn.kv_b_proj as default
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.31.mlp.shared_experts as default
Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.act_fn as default
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.layers.32 as default
Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.q_a_layernorm as default
Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_layernorm as default
Injecting model.layers.32.self_attn.kv_b_proj as default
Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.32.mlp.shared_experts as default
Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.act_fn as default
Injecting model.layers.32.input_layernorm as default
Injecting model.layers.32.post_attention_layernorm as default
Injecting model.layers.33 as default
Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.q_a_layernorm as default
Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_layernorm as default
Injecting model.layers.33.self_attn.kv_b_proj as default
Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.33.mlp.shared_experts as default
Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.act_fn as default
Injecting model.layers.33.input_layernorm as default
Injecting model.layers.33.post_attention_layernorm as default
Injecting model.layers.34 as default
Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.q_a_layernorm as default
Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_layernorm as default
Injecting model.layers.34.self_attn.kv_b_proj as default
Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.34.mlp.shared_experts as default
Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.act_fn as default
Injecting model.layers.34.input_layernorm as default
Injecting model.layers.34.post_attention_layernorm as default
Injecting model.layers.35 as default
Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.q_a_layernorm as default
Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_layernorm as default
Injecting model.layers.35.self_attn.kv_b_proj as default
Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.35.mlp.shared_experts as default
Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.act_fn as default
Injecting model.layers.35.input_layernorm as default
Injecting model.layers.35.post_attention_layernorm as default
Injecting model.layers.36 as default
Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.q_a_layernorm as default
Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_layernorm as default
Injecting model.layers.36.self_attn.kv_b_proj as default
Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.36.mlp.shared_experts as default
Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.act_fn as default
Injecting model.layers.36.input_layernorm as default
Injecting model.layers.36.post_attention_layernorm as default
Injecting model.layers.37 as default
Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.q_a_layernorm as default
Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_layernorm as default
Injecting model.layers.37.self_attn.kv_b_proj as default
Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.37.mlp.shared_experts as default
Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.act_fn as default
Injecting model.layers.37.input_layernorm as default
Injecting model.layers.37.post_attention_layernorm as default
Injecting model.layers.38 as default
Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.q_a_layernorm as default
Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_layernorm as default
Injecting model.layers.38.self_attn.kv_b_proj as default
Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.38.mlp.shared_experts as default
Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.act_fn as default
Injecting model.layers.38.input_layernorm as default
Injecting model.layers.38.post_attention_layernorm as default
Injecting model.layers.39 as default
Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.q_a_layernorm as default
Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_layernorm as default
Injecting model.layers.39.self_attn.kv_b_proj as default
Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.39.mlp.shared_experts as default
Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.act_fn as default
Injecting model.layers.39.input_layernorm as default
Injecting model.layers.39.post_attention_layernorm as default
Injecting model.layers.40 as default
Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.q_a_layernorm as default
Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_layernorm as default
Injecting model.layers.40.self_attn.kv_b_proj as default
Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.40.mlp.shared_experts as default
Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.act_fn as default
Injecting model.layers.40.input_layernorm as default
Injecting model.layers.40.post_attention_layernorm as default
Injecting model.layers.41 as default
Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.q_a_layernorm as default
Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_layernorm as default
Injecting model.layers.41.self_attn.kv_b_proj as default
Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.41.mlp.shared_experts as default
Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.act_fn as default
Injecting model.layers.41.input_layernorm as default
Injecting model.layers.41.post_attention_layernorm as default
Injecting model.layers.42 as default
Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.q_a_layernorm as default
Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_layernorm as default
Injecting model.layers.42.self_attn.kv_b_proj as default
Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.42.mlp.shared_experts as default
Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.act_fn as default
Injecting model.layers.42.input_layernorm as default
Injecting model.layers.42.post_attention_layernorm as default
Injecting model.layers.43 as default
Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.q_a_layernorm as default
Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_layernorm as default
Injecting model.layers.43.self_attn.kv_b_proj as default
Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.43.mlp.shared_experts as default
Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.act_fn as default
Injecting model.layers.43.input_layernorm as default
Injecting model.layers.43.post_attention_layernorm as default
Injecting model.layers.44 as default
Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.q_a_layernorm as default
Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_layernorm as default
Injecting model.layers.44.self_attn.kv_b_proj as default
Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.44.mlp.shared_experts as default
Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.act_fn as default
Injecting model.layers.44.input_layernorm as default
Injecting model.layers.44.post_attention_layernorm as default
Injecting model.layers.45 as default
Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.q_a_layernorm as default
Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_layernorm as default
Injecting model.layers.45.self_attn.kv_b_proj as default
Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.45.mlp.shared_experts as default
Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.act_fn as default
Injecting model.layers.45.input_layernorm as default
Injecting model.layers.45.post_attention_layernorm as default
Injecting model.layers.46 as default
Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.q_a_layernorm as default
Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_layernorm as default
Injecting model.layers.46.self_attn.kv_b_proj as default
Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.46.mlp.shared_experts as default
Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.act_fn as default
Injecting model.layers.46.input_layernorm as default
Injecting model.layers.46.post_attention_layernorm as default
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_a_layernorm as default
Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_layernorm as default
Injecting model.layers.47.self_attn.kv_b_proj as default
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.47.mlp.shared_experts as default
Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.act_fn as default
Injecting model.layers.47.input_layernorm as default
Injecting model.layers.47.post_attention_layernorm as default
Injecting model.layers.48 as default
Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.q_a_layernorm as default
Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_layernorm as default
Injecting model.layers.48.self_attn.kv_b_proj as default
Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.48.mlp.shared_experts as default
Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.act_fn as default
Injecting model.layers.48.input_layernorm as default
Injecting model.layers.48.post_attention_layernorm as default
Injecting model.layers.49 as default
Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.q_a_layernorm as default
Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_layernorm as default
Injecting model.layers.49.self_attn.kv_b_proj as default
Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.49.mlp.shared_experts as default
Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.act_fn as default
Injecting model.layers.49.input_layernorm as default
Injecting model.layers.49.post_attention_layernorm as default
Injecting model.layers.50 as default
Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.q_a_layernorm as default
Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_layernorm as default
Injecting model.layers.50.self_attn.kv_b_proj as default
Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.50.mlp.shared_experts as default
Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.act_fn as default
Injecting model.layers.50.input_layernorm as default
Injecting model.layers.50.post_attention_layernorm as default
Injecting model.layers.51 as default
Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.q_a_layernorm as default
Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_layernorm as default
Injecting model.layers.51.self_attn.kv_b_proj as default
Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.51.mlp.shared_experts as default
Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.act_fn as default
Injecting model.layers.51.input_layernorm as default
Injecting model.layers.51.post_attention_layernorm as default
Injecting model.layers.52 as default
Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.q_a_layernorm as default
Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_layernorm as default
Injecting model.layers.52.self_attn.kv_b_proj as default
Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.52.mlp.shared_experts as default
Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.act_fn as default
Injecting model.layers.52.input_layernorm as default
Injecting model.layers.52.post_attention_layernorm as default
Injecting model.layers.53 as default
Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.q_a_layernorm as default
Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_layernorm as default
Injecting model.layers.53.self_attn.kv_b_proj as default
Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.53.mlp.shared_experts as default
Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.act_fn as default
Injecting model.layers.53.input_layernorm as default
Injecting model.layers.53.post_attention_layernorm as default
Injecting model.layers.54 as default
Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.q_a_layernorm as default
Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_layernorm as default
Injecting model.layers.54.self_attn.kv_b_proj as default
Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.54.mlp.shared_experts as default
Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.act_fn as default
Injecting model.layers.54.input_layernorm as default
Injecting model.layers.54.post_attention_layernorm as default
Injecting model.layers.55 as default
Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.q_a_layernorm as default
Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_layernorm as default
Injecting model.layers.55.self_attn.kv_b_proj as default
Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.55.mlp.shared_experts as default
Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.act_fn as default
Injecting model.layers.55.input_layernorm as default
Injecting model.layers.55.post_attention_layernorm as default
Injecting model.layers.56 as default
Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.q_a_layernorm as default
Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_layernorm as default
Injecting model.layers.56.self_attn.kv_b_proj as default
Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.56.mlp.shared_experts as default
Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.act_fn as default
Injecting model.layers.56.input_layernorm as default
Injecting model.layers.56.post_attention_layernorm as default
Injecting model.layers.57 as default
Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.q_a_layernorm as default
Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_layernorm as default
Injecting model.layers.57.self_attn.kv_b_proj as default
Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.57.mlp.shared_experts as default
Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.act_fn as default
Injecting model.layers.57.input_layernorm as default
Injecting model.layers.57.post_attention_layernorm as default
Injecting model.layers.58 as default
Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.q_a_layernorm as default
Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_layernorm as default
Injecting model.layers.58.self_attn.kv_b_proj as default
Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.58.mlp.shared_experts as default
Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.act_fn as default
Injecting model.layers.58.input_layernorm as default
Injecting model.layers.58.post_attention_layernorm as default
Injecting model.layers.59 as default
Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.q_a_layernorm as default
Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_layernorm as default
Injecting model.layers.59.self_attn.kv_b_proj as default
Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.59.mlp.shared_experts as default
Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.act_fn as default
Injecting model.layers.59.input_layernorm as default
Injecting model.layers.59.post_attention_layernorm as default
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
/opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.)
  data = torch.from_numpy(data)
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights
    module.load()
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict
    weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Hi，请问你的现在跑起来了吗？我的也是V100, 折腾了好久，查了一下貌似是PyTorch新版不再支持sm_70算力的架构，也就是Tesla V100, 也许需要自己从源码编译PyTorch

<!-- gh-comment-id:3509128841 --> @jercylew commented on GitHub (Nov 10, 2025): > [@moneyisallyouneed](https://github.com/moneyisallyouneed) 感谢你。`GLIBCxxx`的报错已经通过使用avx512的whl修复，设置LD_PRELOAD环境变量也能解决。但是CUDA Error始终无法修复，我按照其他issues指示将KLinearMarlin修改为KLinearTorch，仍不起作用。具体的CUDA Error如下。 > > ``` > root@3f1f6abfc329:/workspace/ktransformers# ktransformers --port 10002 --gguf_path /models/ --model_path /models/DeepSeek-R1/ --optimize_config_path myrule/DeepSeek-V3-Chat-multi-gpu.yaml 2>&1 | tee output > > �[32m2025-02-15 14:52:23,762 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables�[0m > �[32m2025-02-15 14:52:23,765 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant�[0m > Injecting model as ktransformers.operators.models . KDeepseekV2Model > Injecting model.embed_tokens as default > Injecting model.layers as default > Injecting model.layers.0 as default > Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.q_a_layernorm as default > Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.kv_a_layernorm as default > Injecting model.layers.0.self_attn.kv_b_proj as default > Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.0.mlp as default > Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.0.mlp.act_fn as default > Injecting model.layers.0.input_layernorm as default > Injecting model.layers.0.post_attention_layernorm as default > Injecting model.layers.1 as default > Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.q_a_layernorm as default > Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.kv_a_layernorm as default > Injecting model.layers.1.self_attn.kv_b_proj as default > Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.1.mlp as default > Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.1.mlp.act_fn as default > Injecting model.layers.1.input_layernorm as default > Injecting model.layers.1.post_attention_layernorm as default > Injecting model.layers.2 as default > Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.q_a_layernorm as default > Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.kv_a_layernorm as default > Injecting model.layers.2.self_attn.kv_b_proj as default > Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.2.mlp as default > Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.2.mlp.act_fn as default > Injecting model.layers.2.input_layernorm as default > Injecting model.layers.2.post_attention_layernorm as default > Injecting model.layers.3 as default > Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.q_a_layernorm as default > Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.kv_a_layernorm as default > Injecting model.layers.3.self_attn.kv_b_proj as default > Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.3.mlp.shared_experts as default > Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.3.mlp.shared_experts.act_fn as default > Injecting model.layers.3.input_layernorm as default > Injecting model.layers.3.post_attention_layernorm as default > Injecting model.layers.4 as default > Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.q_a_layernorm as default > Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.kv_a_layernorm as default > Injecting model.layers.4.self_attn.kv_b_proj as default > Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.4.mlp.shared_experts as default > Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.4.mlp.shared_experts.act_fn as default > Injecting model.layers.4.input_layernorm as default > Injecting model.layers.4.post_attention_layernorm as default > Injecting model.layers.5 as default > Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.q_a_layernorm as default > Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.kv_a_layernorm as default > Injecting model.layers.5.self_attn.kv_b_proj as default > Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.5.mlp.shared_experts as default > Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.5.mlp.shared_experts.act_fn as default > Injecting model.layers.5.input_layernorm as default > Injecting model.layers.5.post_attention_layernorm as default > Injecting model.layers.6 as default > Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.q_a_layernorm as default > Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.kv_a_layernorm as default > Injecting model.layers.6.self_attn.kv_b_proj as default > Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.6.mlp.shared_experts as default > Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.6.mlp.shared_experts.act_fn as default > Injecting model.layers.6.input_layernorm as default > Injecting model.layers.6.post_attention_layernorm as default > Injecting model.layers.7 as default > Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.q_a_layernorm as default > Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.kv_a_layernorm as default > Injecting model.layers.7.self_attn.kv_b_proj as default > Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.7.mlp.shared_experts as default > Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.7.mlp.shared_experts.act_fn as default > Injecting model.layers.7.input_layernorm as default > Injecting model.layers.7.post_attention_layernorm as default > Injecting model.layers.8 as default > Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.q_a_layernorm as default > Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.kv_a_layernorm as default > Injecting model.layers.8.self_attn.kv_b_proj as default > Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.8.mlp.shared_experts as default > Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.8.mlp.shared_experts.act_fn as default > Injecting model.layers.8.input_layernorm as default > Injecting model.layers.8.post_attention_layernorm as default > Injecting model.layers.9 as default > Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.q_a_layernorm as default > Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.kv_a_layernorm as default > Injecting model.layers.9.self_attn.kv_b_proj as default > Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.9.mlp.shared_experts as default > Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.9.mlp.shared_experts.act_fn as default > Injecting model.layers.9.input_layernorm as default > Injecting model.layers.9.post_attention_layernorm as default > Injecting model.layers.10 as default > Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.q_a_layernorm as default > Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.kv_a_layernorm as default > Injecting model.layers.10.self_attn.kv_b_proj as default > Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.10.mlp.shared_experts as default > Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.10.mlp.shared_experts.act_fn as default > Injecting model.layers.10.input_layernorm as default > Injecting model.layers.10.post_attention_layernorm as default > Injecting model.layers.11 as default > Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.q_a_layernorm as default > Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.kv_a_layernorm as default > Injecting model.layers.11.self_attn.kv_b_proj as default > Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.11.mlp.shared_experts as default > Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.11.mlp.shared_experts.act_fn as default > Injecting model.layers.11.input_layernorm as default > Injecting model.layers.11.post_attention_layernorm as default > Injecting model.layers.12 as default > Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.q_a_layernorm as default > Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.kv_a_layernorm as default > Injecting model.layers.12.self_attn.kv_b_proj as default > Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.12.mlp.shared_experts as default > Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.12.mlp.shared_experts.act_fn as default > Injecting model.layers.12.input_layernorm as default > Injecting model.layers.12.post_attention_layernorm as default > Injecting model.layers.13 as default > Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.q_a_layernorm as default > Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.kv_a_layernorm as default > Injecting model.layers.13.self_attn.kv_b_proj as default > Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.13.mlp.shared_experts as default > Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.13.mlp.shared_experts.act_fn as default > Injecting model.layers.13.input_layernorm as default > Injecting model.layers.13.post_attention_layernorm as default > Injecting model.layers.14 as default > Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.q_a_layernorm as default > Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.kv_a_layernorm as default > Injecting model.layers.14.self_attn.kv_b_proj as default > Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.14.mlp.shared_experts as default > Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.14.mlp.shared_experts.act_fn as default > Injecting model.layers.14.input_layernorm as default > Injecting model.layers.14.post_attention_layernorm as default > Injecting model.layers.15 as default > Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.q_a_layernorm as default > Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.kv_a_layernorm as default > Injecting model.layers.15.self_attn.kv_b_proj as default > Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.15.mlp.shared_experts as default > Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.15.mlp.shared_experts.act_fn as default > Injecting model.layers.15.input_layernorm as default > Injecting model.layers.15.post_attention_layernorm as default > Injecting model.layers.16 as default > Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.q_a_layernorm as default > Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.kv_a_layernorm as default > Injecting model.layers.16.self_attn.kv_b_proj as default > Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.16.mlp.shared_experts as default > Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.16.mlp.shared_experts.act_fn as default > Injecting model.layers.16.input_layernorm as default > Injecting model.layers.16.post_attention_layernorm as default > Injecting model.layers.17 as default > Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.q_a_layernorm as default > Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.kv_a_layernorm as default > Injecting model.layers.17.self_attn.kv_b_proj as default > Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.17.mlp.shared_experts as default > Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.17.mlp.shared_experts.act_fn as default > Injecting model.layers.17.input_layernorm as default > Injecting model.layers.17.post_attention_layernorm as default > Injecting model.layers.18 as default > Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.q_a_layernorm as default > Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.kv_a_layernorm as default > Injecting model.layers.18.self_attn.kv_b_proj as default > Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.18.mlp.shared_experts as default > Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.18.mlp.shared_experts.act_fn as default > Injecting model.layers.18.input_layernorm as default > Injecting model.layers.18.post_attention_layernorm as default > Injecting model.layers.19 as default > Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.q_a_layernorm as default > Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.kv_a_layernorm as default > Injecting model.layers.19.self_attn.kv_b_proj as default > Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.19.mlp.shared_experts as default > Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.19.mlp.shared_experts.act_fn as default > Injecting model.layers.19.input_layernorm as default > Injecting model.layers.19.post_attention_layernorm as default > Injecting model.layers.20 as default > Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.q_a_layernorm as default > Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.kv_a_layernorm as default > Injecting model.layers.20.self_attn.kv_b_proj as default > Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.20.mlp.shared_experts as default > Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.20.mlp.shared_experts.act_fn as default > Injecting model.layers.20.input_layernorm as default > Injecting model.layers.20.post_attention_layernorm as default > Injecting model.layers.21 as default > Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.q_a_layernorm as default > Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.kv_a_layernorm as default > Injecting model.layers.21.self_attn.kv_b_proj as default > Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.21.mlp.shared_experts as default > Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.21.mlp.shared_experts.act_fn as default > Injecting model.layers.21.input_layernorm as default > Injecting model.layers.21.post_attention_layernorm as default > Injecting model.layers.22 as default > Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.q_a_layernorm as default > Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.kv_a_layernorm as default > Injecting model.layers.22.self_attn.kv_b_proj as default > Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.22.mlp.shared_experts as default > Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.22.mlp.shared_experts.act_fn as default > Injecting model.layers.22.input_layernorm as default > Injecting model.layers.22.post_attention_layernorm as default > Injecting model.layers.23 as default > Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.q_a_layernorm as default > Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.kv_a_layernorm as default > Injecting model.layers.23.self_attn.kv_b_proj as default > Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.23.mlp.shared_experts as default > Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.23.mlp.shared_experts.act_fn as default > Injecting model.layers.23.input_layernorm as default > Injecting model.layers.23.post_attention_layernorm as default > Injecting model.layers.24 as default > Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.q_a_layernorm as default > Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.kv_a_layernorm as default > Injecting model.layers.24.self_attn.kv_b_proj as default > Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.24.mlp.shared_experts as default > Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.24.mlp.shared_experts.act_fn as default > Injecting model.layers.24.input_layernorm as default > Injecting model.layers.24.post_attention_layernorm as default > Injecting model.layers.25 as default > Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.q_a_layernorm as default > Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.kv_a_layernorm as default > Injecting model.layers.25.self_attn.kv_b_proj as default > Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.25.mlp.shared_experts as default > Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.25.mlp.shared_experts.act_fn as default > Injecting model.layers.25.input_layernorm as default > Injecting model.layers.25.post_attention_layernorm as default > Injecting model.layers.26 as default > Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.q_a_layernorm as default > Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.kv_a_layernorm as default > Injecting model.layers.26.self_attn.kv_b_proj as default > Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.26.mlp.shared_experts as default > Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.26.mlp.shared_experts.act_fn as default > Injecting model.layers.26.input_layernorm as default > Injecting model.layers.26.post_attention_layernorm as default > Injecting model.layers.27 as default > Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.q_a_layernorm as default > Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.kv_a_layernorm as default > Injecting model.layers.27.self_attn.kv_b_proj as default > Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.27.mlp.shared_experts as default > Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.27.mlp.shared_experts.act_fn as default > Injecting model.layers.27.input_layernorm as default > Injecting model.layers.27.post_attention_layernorm as default > Injecting model.layers.28 as default > Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.q_a_layernorm as default > Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.kv_a_layernorm as default > Injecting model.layers.28.self_attn.kv_b_proj as default > Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.28.mlp.shared_experts as default > Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.28.mlp.shared_experts.act_fn as default > Injecting model.layers.28.input_layernorm as default > Injecting model.layers.28.post_attention_layernorm as default > Injecting model.layers.29 as default > Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.q_a_layernorm as default > Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.kv_a_layernorm as default > Injecting model.layers.29.self_attn.kv_b_proj as default > Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.29.mlp.shared_experts as default > Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.29.mlp.shared_experts.act_fn as default > Injecting model.layers.29.input_layernorm as default > Injecting model.layers.29.post_attention_layernorm as default > Injecting model.layers.30 as default > Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.q_a_layernorm as default > Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.kv_a_layernorm as default > Injecting model.layers.30.self_attn.kv_b_proj as default > Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.30.mlp.shared_experts as default > Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.30.mlp.shared_experts.act_fn as default > Injecting model.layers.30.input_layernorm as default > Injecting model.layers.30.post_attention_layernorm as default > Injecting model.layers.31 as default > Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.q_a_layernorm as default > Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.kv_a_layernorm as default > Injecting model.layers.31.self_attn.kv_b_proj as default > Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.31.mlp.shared_experts as default > Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.31.mlp.shared_experts.act_fn as default > Injecting model.layers.31.input_layernorm as default > Injecting model.layers.31.post_attention_layernorm as default > Injecting model.layers.32 as default > Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.q_a_layernorm as default > Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.kv_a_layernorm as default > Injecting model.layers.32.self_attn.kv_b_proj as default > Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.32.mlp.shared_experts as default > Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.32.mlp.shared_experts.act_fn as default > Injecting model.layers.32.input_layernorm as default > Injecting model.layers.32.post_attention_layernorm as default > Injecting model.layers.33 as default > Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.q_a_layernorm as default > Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.kv_a_layernorm as default > Injecting model.layers.33.self_attn.kv_b_proj as default > Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.33.mlp.shared_experts as default > Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.33.mlp.shared_experts.act_fn as default > Injecting model.layers.33.input_layernorm as default > Injecting model.layers.33.post_attention_layernorm as default > Injecting model.layers.34 as default > Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.q_a_layernorm as default > Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.kv_a_layernorm as default > Injecting model.layers.34.self_attn.kv_b_proj as default > Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.34.mlp.shared_experts as default > Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.34.mlp.shared_experts.act_fn as default > Injecting model.layers.34.input_layernorm as default > Injecting model.layers.34.post_attention_layernorm as default > Injecting model.layers.35 as default > Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.q_a_layernorm as default > Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.kv_a_layernorm as default > Injecting model.layers.35.self_attn.kv_b_proj as default > Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.35.mlp.shared_experts as default > Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.35.mlp.shared_experts.act_fn as default > Injecting model.layers.35.input_layernorm as default > Injecting model.layers.35.post_attention_layernorm as default > Injecting model.layers.36 as default > Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.q_a_layernorm as default > Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.kv_a_layernorm as default > Injecting model.layers.36.self_attn.kv_b_proj as default > Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.36.mlp.shared_experts as default > Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.36.mlp.shared_experts.act_fn as default > Injecting model.layers.36.input_layernorm as default > Injecting model.layers.36.post_attention_layernorm as default > Injecting model.layers.37 as default > Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.q_a_layernorm as default > Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.kv_a_layernorm as default > Injecting model.layers.37.self_attn.kv_b_proj as default > Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.37.mlp.shared_experts as default > Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.37.mlp.shared_experts.act_fn as default > Injecting model.layers.37.input_layernorm as default > Injecting model.layers.37.post_attention_layernorm as default > Injecting model.layers.38 as default > Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.q_a_layernorm as default > Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.kv_a_layernorm as default > Injecting model.layers.38.self_attn.kv_b_proj as default > Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.38.mlp.shared_experts as default > Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.38.mlp.shared_experts.act_fn as default > Injecting model.layers.38.input_layernorm as default > Injecting model.layers.38.post_attention_layernorm as default > Injecting model.layers.39 as default > Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.q_a_layernorm as default > Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.kv_a_layernorm as default > Injecting model.layers.39.self_attn.kv_b_proj as default > Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.39.mlp.shared_experts as default > Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.39.mlp.shared_experts.act_fn as default > Injecting model.layers.39.input_layernorm as default > Injecting model.layers.39.post_attention_layernorm as default > Injecting model.layers.40 as default > Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.q_a_layernorm as default > Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.kv_a_layernorm as default > Injecting model.layers.40.self_attn.kv_b_proj as default > Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.40.mlp.shared_experts as default > Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.40.mlp.shared_experts.act_fn as default > Injecting model.layers.40.input_layernorm as default > Injecting model.layers.40.post_attention_layernorm as default > Injecting model.layers.41 as default > Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.q_a_layernorm as default > Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.kv_a_layernorm as default > Injecting model.layers.41.self_attn.kv_b_proj as default > Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.41.mlp.shared_experts as default > Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.41.mlp.shared_experts.act_fn as default > Injecting model.layers.41.input_layernorm as default > Injecting model.layers.41.post_attention_layernorm as default > Injecting model.layers.42 as default > Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.q_a_layernorm as default > Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.kv_a_layernorm as default > Injecting model.layers.42.self_attn.kv_b_proj as default > Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.42.mlp.shared_experts as default > Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.42.mlp.shared_experts.act_fn as default > Injecting model.layers.42.input_layernorm as default > Injecting model.layers.42.post_attention_layernorm as default > Injecting model.layers.43 as default > Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.q_a_layernorm as default > Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.kv_a_layernorm as default > Injecting model.layers.43.self_attn.kv_b_proj as default > Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.43.mlp.shared_experts as default > Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.43.mlp.shared_experts.act_fn as default > Injecting model.layers.43.input_layernorm as default > Injecting model.layers.43.post_attention_layernorm as default > Injecting model.layers.44 as default > Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.q_a_layernorm as default > Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.kv_a_layernorm as default > Injecting model.layers.44.self_attn.kv_b_proj as default > Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.44.mlp.shared_experts as default > Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.44.mlp.shared_experts.act_fn as default > Injecting model.layers.44.input_layernorm as default > Injecting model.layers.44.post_attention_layernorm as default > Injecting model.layers.45 as default > Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.q_a_layernorm as default > Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.kv_a_layernorm as default > Injecting model.layers.45.self_attn.kv_b_proj as default > Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.45.mlp.shared_experts as default > Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.45.mlp.shared_experts.act_fn as default > Injecting model.layers.45.input_layernorm as default > Injecting model.layers.45.post_attention_layernorm as default > Injecting model.layers.46 as default > Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.q_a_layernorm as default > Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.kv_a_layernorm as default > Injecting model.layers.46.self_attn.kv_b_proj as default > Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.46.mlp.shared_experts as default > Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.46.mlp.shared_experts.act_fn as default > Injecting model.layers.46.input_layernorm as default > Injecting model.layers.46.post_attention_layernorm as default > Injecting model.layers.47 as default > Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.q_a_layernorm as default > Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.kv_a_layernorm as default > Injecting model.layers.47.self_attn.kv_b_proj as default > Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.47.mlp.shared_experts as default > Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.47.mlp.shared_experts.act_fn as default > Injecting model.layers.47.input_layernorm as default > Injecting model.layers.47.post_attention_layernorm as default > Injecting model.layers.48 as default > Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.q_a_layernorm as default > Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.kv_a_layernorm as default > Injecting model.layers.48.self_attn.kv_b_proj as default > Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.48.mlp.shared_experts as default > Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.48.mlp.shared_experts.act_fn as default > Injecting model.layers.48.input_layernorm as default > Injecting model.layers.48.post_attention_layernorm as default > Injecting model.layers.49 as default > Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.q_a_layernorm as default > Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.kv_a_layernorm as default > Injecting model.layers.49.self_attn.kv_b_proj as default > Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.49.mlp.shared_experts as default > Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.49.mlp.shared_experts.act_fn as default > Injecting model.layers.49.input_layernorm as default > Injecting model.layers.49.post_attention_layernorm as default > Injecting model.layers.50 as default > Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.q_a_layernorm as default > Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.kv_a_layernorm as default > Injecting model.layers.50.self_attn.kv_b_proj as default > Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.50.mlp.shared_experts as default > Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.50.mlp.shared_experts.act_fn as default > Injecting model.layers.50.input_layernorm as default > Injecting model.layers.50.post_attention_layernorm as default > Injecting model.layers.51 as default > Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.q_a_layernorm as default > Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.kv_a_layernorm as default > Injecting model.layers.51.self_attn.kv_b_proj as default > Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.51.mlp.shared_experts as default > Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.51.mlp.shared_experts.act_fn as default > Injecting model.layers.51.input_layernorm as default > Injecting model.layers.51.post_attention_layernorm as default > Injecting model.layers.52 as default > Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.q_a_layernorm as default > Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.kv_a_layernorm as default > Injecting model.layers.52.self_attn.kv_b_proj as default > Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.52.mlp.shared_experts as default > Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.52.mlp.shared_experts.act_fn as default > Injecting model.layers.52.input_layernorm as default > Injecting model.layers.52.post_attention_layernorm as default > Injecting model.layers.53 as default > Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.q_a_layernorm as default > Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.kv_a_layernorm as default > Injecting model.layers.53.self_attn.kv_b_proj as default > Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.53.mlp.shared_experts as default > Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.53.mlp.shared_experts.act_fn as default > Injecting model.layers.53.input_layernorm as default > Injecting model.layers.53.post_attention_layernorm as default > Injecting model.layers.54 as default > Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.q_a_layernorm as default > Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.kv_a_layernorm as default > Injecting model.layers.54.self_attn.kv_b_proj as default > Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.54.mlp.shared_experts as default > Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.54.mlp.shared_experts.act_fn as default > Injecting model.layers.54.input_layernorm as default > Injecting model.layers.54.post_attention_layernorm as default > Injecting model.layers.55 as default > Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.q_a_layernorm as default > Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.kv_a_layernorm as default > Injecting model.layers.55.self_attn.kv_b_proj as default > Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.55.mlp.shared_experts as default > Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.55.mlp.shared_experts.act_fn as default > Injecting model.layers.55.input_layernorm as default > Injecting model.layers.55.post_attention_layernorm as default > Injecting model.layers.56 as default > Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.q_a_layernorm as default > Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.kv_a_layernorm as default > Injecting model.layers.56.self_attn.kv_b_proj as default > Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.56.mlp.shared_experts as default > Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.56.mlp.shared_experts.act_fn as default > Injecting model.layers.56.input_layernorm as default > Injecting model.layers.56.post_attention_layernorm as default > Injecting model.layers.57 as default > Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.q_a_layernorm as default > Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.kv_a_layernorm as default > Injecting model.layers.57.self_attn.kv_b_proj as default > Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.57.mlp.shared_experts as default > Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.57.mlp.shared_experts.act_fn as default > Injecting model.layers.57.input_layernorm as default > Injecting model.layers.57.post_attention_layernorm as default > Injecting model.layers.58 as default > Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.q_a_layernorm as default > Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.kv_a_layernorm as default > Injecting model.layers.58.self_attn.kv_b_proj as default > Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.58.mlp.shared_experts as default > Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.58.mlp.shared_experts.act_fn as default > Injecting model.layers.58.input_layernorm as default > Injecting model.layers.58.post_attention_layernorm as default > Injecting model.layers.59 as default > Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.q_a_layernorm as default > Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.kv_a_layernorm as default > Injecting model.layers.59.self_attn.kv_b_proj as default > Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.59.mlp.shared_experts as default > Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.59.mlp.shared_experts.act_fn as default > Injecting model.layers.59.input_layernorm as default > Injecting model.layers.59.post_attention_layernorm as default > Injecting model.layers.60 as default > Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention > Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear > /opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:401: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.) > data = torch.from_numpy(data) > Injecting model.layers.60.self_attn.q_a_layernorm as default > Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.kv_a_layernorm as default > Injecting model.layers.60.self_attn.kv_b_proj as default > Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 > Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE > Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts > Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate > Injecting model.layers.60.mlp.shared_experts as default > Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear > Injecting model.layers.60.mlp.shared_experts.act_fn as default > Injecting model.layers.60.input_layernorm as default > Injecting model.layers.60.post_attention_layernorm as default > Injecting model.norm as default > Injecting lm_head as default > loading token_embd.weight to cpu > loading blk.0.attn_q_a_norm.weight to cuda:0 > loading blk.0.attn_kv_a_norm.weight to cuda:0 > loading blk.0.attn_kv_b.weight to cuda:0 > Traceback (most recent call last): > File "/opt/conda/bin/ktransformers", line 8, in <module> > sys.exit(main()) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main > create_interface(config=cfg, default_args=cfg) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface > GlobalInterface.interface = BackendInterface(default_args) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ > optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf > load_weights(module, gguf_loader) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 85, in load_weights > module.load() > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 60, in load > utils.load_weights(child, self.gguf_loader, self.key+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 83, in load_weights > load_weights(child, gguf_loader, prefix+name+".") > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 81, in load_weights > load_cur_state_dict(module, gguf_loader, prefix) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 71, in load_cur_state_dict > weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype) > RuntimeError: CUDA error: no kernel image is available for execution on the device > CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. > For debugging consider passing CUDA_LAUNCH_BLOCKING=1. > Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. > ``` Hi，请问你的现在跑起来了吗？我的也是V100, 折腾了好久，查了一下貌似是PyTorch新版不再支持sm_70算力的架构，也就是Tesla V100, 也许需要自己从源码编译PyTorch
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#193
No description provided.