mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#646
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @gengchaogit on GitHub (Mar 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/903
具体的问题有几个
1.从硬盘加载巨慢
2.缓存到内存后第二次加载，也就1G每秒，也很慢，我的内存带宽600G，大小有1T多
3.加载完后输入任意一句话，直接卡死退出
4.AMD的卡只往16G显存加载了14%也就3G多点，理论上不应该这么点。
5.回测llama.cpp加载速度，10秒从内存加载400G
不确定是否跟我手动跳过编译有关
Chat: hi
/root/ktransformers/ktransformers/operators/RoPE.py:269: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:310.)
freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
corrupted size vs. prev_size
Every 1.0s: rocm-smi
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
^[3m (DID, GUID) (Edge) (Socket) (Mem, Compute, ID)
0 1 0x66a1, 52718 35.0°C 21.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 14% 0%
1 2 0x66a1, 46486 37.0°C 20.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 0% 0%
@gengchaogit commented on GitHub (Mar 15, 2025):
@gengchaogit commented on GitHub (Mar 15, 2025):
(ktransformers) root@epyc:
/ktransformers# cat ./start671bq4.sh/ktransformers# python ktransformers/local_chat.pypython ktransformers/local_chat.py
--model_path deepseek-ai/DeepSeek-R1
--gguf_path /mnt/windows/Users/chao/.lmstudio/models/deepseek671b/deepseek671b-q4/DeepSeek-R1-Q4_K_M-00001-of-00009.gguf
--optimize_config_path ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml
--cpu_infer 65
(ktransformers) root@epyc:
--model_path deepseek-ai/DeepSeek-R1
--gguf_path /mnt/windows/Users/chao/.lmstudio/models/deepseek671b/deepseek671b-q4/DeepSeek-R1-Q4_K_M-00001-of-00009.gguf
--optimize_config_path ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml
--cpu_infer 65
flashinfer not found, use triton for linux
using custom modeling_xxx.py.
falsh attn not found
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
Injecting model.layers.9.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.9.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.q_a_layernorm as default
Injecting model.layers.9.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.kv_a_layernorm as default
Injecting model.layers.9.self_attn.kv_b_proj as default
Injecting model.layers.9.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.9.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.9.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.9.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.9.mlp.shared_experts as default
Injecting model.layers.9.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.9.mlp.shared_experts.act_fn as default
Injecting model.layers.9.input_layernorm as default
Injecting model.layers.9.post_attention_layernorm as default
Injecting model.layers.10 as default
Injecting model.layers.10.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.10.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.q_a_layernorm as default
Injecting model.layers.10.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.kv_a_layernorm as default
Injecting model.layers.10.self_attn.kv_b_proj as default
Injecting model.layers.10.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.10.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.10.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.10.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.10.mlp.shared_experts as default
Injecting model.layers.10.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.10.mlp.shared_experts.act_fn as default
Injecting model.layers.10.input_layernorm as default
Injecting model.layers.10.post_attention_layernorm as default
Injecting model.layers.11 as default
Injecting model.layers.11.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.11.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.q_a_layernorm as default
Injecting model.layers.11.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.kv_a_layernorm as default
Injecting model.layers.11.self_attn.kv_b_proj as default
Injecting model.layers.11.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.11.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.11.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.11.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.11.mlp.shared_experts as default
Injecting model.layers.11.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.11.mlp.shared_experts.act_fn as default
Injecting model.layers.11.input_layernorm as default
Injecting model.layers.11.post_attention_layernorm as default
Injecting model.layers.12 as default
Injecting model.layers.12.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.12.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.q_a_layernorm as default
Injecting model.layers.12.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.kv_a_layernorm as default
Injecting model.layers.12.self_attn.kv_b_proj as default
Injecting model.layers.12.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.12.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.12.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.12.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.12.mlp.shared_experts as default
Injecting model.layers.12.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.12.mlp.shared_experts.act_fn as default
Injecting model.layers.12.input_layernorm as default
Injecting model.layers.12.post_attention_layernorm as default
Injecting model.layers.13 as default
Injecting model.layers.13.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.13.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.q_a_layernorm as default
Injecting model.layers.13.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.kv_a_layernorm as default
Injecting model.layers.13.self_attn.kv_b_proj as default
Injecting model.layers.13.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.13.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.13.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.13.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.13.mlp.shared_experts as default
Injecting model.layers.13.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.13.mlp.shared_experts.act_fn as default
Injecting model.layers.13.input_layernorm as default
Injecting model.layers.13.post_attention_layernorm as default
Injecting model.layers.14 as default
Injecting model.layers.14.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.14.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.q_a_layernorm as default
Injecting model.layers.14.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.kv_a_layernorm as default
Injecting model.layers.14.self_attn.kv_b_proj as default
Injecting model.layers.14.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.14.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.14.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.14.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.14.mlp.shared_experts as default
Injecting model.layers.14.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.14.mlp.shared_experts.act_fn as default
Injecting model.layers.14.input_layernorm as default
Injecting model.layers.14.post_attention_layernorm as default
Injecting model.layers.15 as default
Injecting model.layers.15.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.15.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.q_a_layernorm as default
Injecting model.layers.15.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.kv_a_layernorm as default
Injecting model.layers.15.self_attn.kv_b_proj as default
Injecting model.layers.15.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.15.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.15.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.15.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.15.mlp.shared_experts as default
Injecting model.layers.15.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.15.mlp.shared_experts.act_fn as default
Injecting model.layers.15.input_layernorm as default
Injecting model.layers.15.post_attention_layernorm as default
Injecting model.layers.16 as default
Injecting model.layers.16.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.16.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.q_a_layernorm as default
Injecting model.layers.16.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.kv_a_layernorm as default
Injecting model.layers.16.self_attn.kv_b_proj as default
Injecting model.layers.16.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.16.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.16.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.16.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.16.mlp.shared_experts as default
Injecting model.layers.16.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.16.mlp.shared_experts.act_fn as default
Injecting model.layers.16.input_layernorm as default
Injecting model.layers.16.post_attention_layernorm as default
Injecting model.layers.17 as default
Injecting model.layers.17.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.17.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.q_a_layernorm as default
Injecting model.layers.17.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.kv_a_layernorm as default
Injecting model.layers.17.self_attn.kv_b_proj as default
Injecting model.layers.17.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.17.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.17.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.17.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.17.mlp.shared_experts as default
Injecting model.layers.17.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.17.mlp.shared_experts.act_fn as default
Injecting model.layers.17.input_layernorm as default
Injecting model.layers.17.post_attention_layernorm as default
Injecting model.layers.18 as default
Injecting model.layers.18.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.18.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.q_a_layernorm as default
Injecting model.layers.18.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.kv_a_layernorm as default
Injecting model.layers.18.self_attn.kv_b_proj as default
Injecting model.layers.18.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.18.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.18.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.18.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.18.mlp.shared_experts as default
Injecting model.layers.18.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.18.mlp.shared_experts.act_fn as default
Injecting model.layers.18.input_layernorm as default
Injecting model.layers.18.post_attention_layernorm as default
Injecting model.layers.19 as default
Injecting model.layers.19.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.19.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.q_a_layernorm as default
Injecting model.layers.19.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.kv_a_layernorm as default
Injecting model.layers.19.self_attn.kv_b_proj as default
Injecting model.layers.19.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.19.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.19.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.19.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.19.mlp.shared_experts as default
Injecting model.layers.19.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.19.mlp.shared_experts.act_fn as default
Injecting model.layers.19.input_layernorm as default
Injecting model.layers.19.post_attention_layernorm as default
Injecting model.layers.20 as default
Injecting model.layers.20.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.20.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.q_a_layernorm as default
Injecting model.layers.20.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.kv_a_layernorm as default
Injecting model.layers.20.self_attn.kv_b_proj as default
Injecting model.layers.20.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.20.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.20.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.20.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.20.mlp.shared_experts as default
Injecting model.layers.20.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.20.mlp.shared_experts.act_fn as default
Injecting model.layers.20.input_layernorm as default
Injecting model.layers.20.post_attention_layernorm as default
Injecting model.layers.21 as default
Injecting model.layers.21.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.21.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.q_a_layernorm as default
Injecting model.layers.21.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.kv_a_layernorm as default
Injecting model.layers.21.self_attn.kv_b_proj as default
Injecting model.layers.21.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.21.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.21.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.21.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.21.mlp.shared_experts as default
Injecting model.layers.21.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.21.mlp.shared_experts.act_fn as default
Injecting model.layers.21.input_layernorm as default
Injecting model.layers.21.post_attention_layernorm as default
Injecting model.layers.22 as default
Injecting model.layers.22.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.22.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.q_a_layernorm as default
Injecting model.layers.22.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.kv_a_layernorm as default
Injecting model.layers.22.self_attn.kv_b_proj as default
Injecting model.layers.22.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.22.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.22.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.22.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.22.mlp.shared_experts as default
Injecting model.layers.22.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.22.mlp.shared_experts.act_fn as default
Injecting model.layers.22.input_layernorm as default
Injecting model.layers.22.post_attention_layernorm as default
Injecting model.layers.23 as default
Injecting model.layers.23.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.23.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.q_a_layernorm as default
Injecting model.layers.23.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.kv_a_layernorm as default
Injecting model.layers.23.self_attn.kv_b_proj as default
Injecting model.layers.23.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.23.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.23.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.23.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.23.mlp.shared_experts as default
Injecting model.layers.23.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.23.mlp.shared_experts.act_fn as default
Injecting model.layers.23.input_layernorm as default
Injecting model.layers.23.post_attention_layernorm as default
Injecting model.layers.24 as default
Injecting model.layers.24.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.24.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.q_a_layernorm as default
Injecting model.layers.24.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.kv_a_layernorm as default
Injecting model.layers.24.self_attn.kv_b_proj as default
Injecting model.layers.24.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.24.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.24.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.24.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.24.mlp.shared_experts as default
Injecting model.layers.24.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.24.mlp.shared_experts.act_fn as default
Injecting model.layers.24.input_layernorm as default
Injecting model.layers.24.post_attention_layernorm as default
Injecting model.layers.25 as default
Injecting model.layers.25.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.25.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.q_a_layernorm as default
Injecting model.layers.25.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.kv_a_layernorm as default
Injecting model.layers.25.self_attn.kv_b_proj as default
Injecting model.layers.25.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.25.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.25.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.25.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.25.mlp.shared_experts as default
Injecting model.layers.25.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.25.mlp.shared_experts.act_fn as default
Injecting model.layers.25.input_layernorm as default
Injecting model.layers.25.post_attention_layernorm as default
Injecting model.layers.26 as default
Injecting model.layers.26.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.26.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.q_a_layernorm as default
Injecting model.layers.26.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.kv_a_layernorm as default
Injecting model.layers.26.self_attn.kv_b_proj as default
Injecting model.layers.26.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.26.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.26.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.26.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.26.mlp.shared_experts as default
Injecting model.layers.26.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.26.mlp.shared_experts.act_fn as default
Injecting model.layers.26.input_layernorm as default
Injecting model.layers.26.post_attention_layernorm as default
Injecting model.layers.27 as default
Injecting model.layers.27.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.27.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.q_a_layernorm as default
Injecting model.layers.27.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.kv_a_layernorm as default
Injecting model.layers.27.self_attn.kv_b_proj as default
Injecting model.layers.27.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.27.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.27.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.27.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.27.mlp.shared_experts as default
Injecting model.layers.27.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.27.mlp.shared_experts.act_fn as default
Injecting model.layers.27.input_layernorm as default
Injecting model.layers.27.post_attention_layernorm as default
Injecting model.layers.28 as default
Injecting model.layers.28.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.28.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.q_a_layernorm as default
Injecting model.layers.28.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.kv_a_layernorm as default
Injecting model.layers.28.self_attn.kv_b_proj as default
Injecting model.layers.28.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.28.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.28.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.28.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.28.mlp.shared_experts as default
Injecting model.layers.28.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.28.mlp.shared_experts.act_fn as default
Injecting model.layers.28.input_layernorm as default
Injecting model.layers.28.post_attention_layernorm as default
Injecting model.layers.29 as default
Injecting model.layers.29.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.29.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.q_a_layernorm as default
Injecting model.layers.29.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.kv_a_layernorm as default
Injecting model.layers.29.self_attn.kv_b_proj as default
Injecting model.layers.29.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.29.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.29.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.29.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.29.mlp.shared_experts as default
Injecting model.layers.29.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.29.mlp.shared_experts.act_fn as default
Injecting model.layers.29.input_layernorm as default
Injecting model.layers.29.post_attention_layernorm as default
Injecting model.layers.30 as default
Injecting model.layers.30.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.30.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.q_a_layernorm as default
Injecting model.layers.30.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.kv_a_layernorm as default
Injecting model.layers.30.self_attn.kv_b_proj as default
Injecting model.layers.30.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.30.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.30.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.30.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.30.mlp.shared_experts as default
Injecting model.layers.30.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.30.mlp.shared_experts.act_fn as default
Injecting model.layers.30.input_layernorm as default
Injecting model.layers.30.post_attention_layernorm as default
Injecting model.layers.31 as default
Injecting model.layers.31.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.31.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.q_a_layernorm as default
Injecting model.layers.31.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.kv_a_layernorm as default
Injecting model.layers.31.self_attn.kv_b_proj as default
Injecting model.layers.31.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.31.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.31.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.31.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.31.mlp.shared_experts as default
Injecting model.layers.31.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.31.mlp.shared_experts.act_fn as default
Injecting model.layers.31.input_layernorm as default
Injecting model.layers.31.post_attention_layernorm as default
Injecting model.layers.32 as default
Injecting model.layers.32.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.32.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.q_a_layernorm as default
Injecting model.layers.32.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.kv_a_layernorm as default
Injecting model.layers.32.self_attn.kv_b_proj as default
Injecting model.layers.32.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.32.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.32.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.32.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.32.mlp.shared_experts as default
Injecting model.layers.32.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.32.mlp.shared_experts.act_fn as default
Injecting model.layers.32.input_layernorm as default
Injecting model.layers.32.post_attention_layernorm as default
Injecting model.layers.33 as default
Injecting model.layers.33.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.33.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.q_a_layernorm as default
Injecting model.layers.33.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.kv_a_layernorm as default
Injecting model.layers.33.self_attn.kv_b_proj as default
Injecting model.layers.33.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.33.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.33.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.33.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.33.mlp.shared_experts as default
Injecting model.layers.33.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.33.mlp.shared_experts.act_fn as default
Injecting model.layers.33.input_layernorm as default
Injecting model.layers.33.post_attention_layernorm as default
Injecting model.layers.34 as default
Injecting model.layers.34.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.34.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.q_a_layernorm as default
Injecting model.layers.34.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.kv_a_layernorm as default
Injecting model.layers.34.self_attn.kv_b_proj as default
Injecting model.layers.34.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.34.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.34.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.34.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.34.mlp.shared_experts as default
Injecting model.layers.34.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.34.mlp.shared_experts.act_fn as default
Injecting model.layers.34.input_layernorm as default
Injecting model.layers.34.post_attention_layernorm as default
Injecting model.layers.35 as default
Injecting model.layers.35.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.35.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.q_a_layernorm as default
Injecting model.layers.35.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.kv_a_layernorm as default
Injecting model.layers.35.self_attn.kv_b_proj as default
Injecting model.layers.35.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.35.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.35.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.35.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.35.mlp.shared_experts as default
Injecting model.layers.35.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.35.mlp.shared_experts.act_fn as default
Injecting model.layers.35.input_layernorm as default
Injecting model.layers.35.post_attention_layernorm as default
Injecting model.layers.36 as default
Injecting model.layers.36.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.36.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.q_a_layernorm as default
Injecting model.layers.36.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.kv_a_layernorm as default
Injecting model.layers.36.self_attn.kv_b_proj as default
Injecting model.layers.36.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.36.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.36.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.36.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.36.mlp.shared_experts as default
Injecting model.layers.36.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.36.mlp.shared_experts.act_fn as default
Injecting model.layers.36.input_layernorm as default
Injecting model.layers.36.post_attention_layernorm as default
Injecting model.layers.37 as default
Injecting model.layers.37.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.37.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.q_a_layernorm as default
Injecting model.layers.37.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.kv_a_layernorm as default
Injecting model.layers.37.self_attn.kv_b_proj as default
Injecting model.layers.37.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.37.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.37.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.37.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.37.mlp.shared_experts as default
Injecting model.layers.37.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.37.mlp.shared_experts.act_fn as default
Injecting model.layers.37.input_layernorm as default
Injecting model.layers.37.post_attention_layernorm as default
Injecting model.layers.38 as default
Injecting model.layers.38.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.38.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.q_a_layernorm as default
Injecting model.layers.38.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.kv_a_layernorm as default
Injecting model.layers.38.self_attn.kv_b_proj as default
Injecting model.layers.38.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.38.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.38.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.38.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.38.mlp.shared_experts as default
Injecting model.layers.38.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.38.mlp.shared_experts.act_fn as default
Injecting model.layers.38.input_layernorm as default
Injecting model.layers.38.post_attention_layernorm as default
Injecting model.layers.39 as default
Injecting model.layers.39.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.39.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.q_a_layernorm as default
Injecting model.layers.39.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.kv_a_layernorm as default
Injecting model.layers.39.self_attn.kv_b_proj as default
Injecting model.layers.39.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.39.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.39.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.39.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.39.mlp.shared_experts as default
Injecting model.layers.39.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.39.mlp.shared_experts.act_fn as default
Injecting model.layers.39.input_layernorm as default
Injecting model.layers.39.post_attention_layernorm as default
Injecting model.layers.40 as default
Injecting model.layers.40.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.40.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.q_a_layernorm as default
Injecting model.layers.40.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.kv_a_layernorm as default
Injecting model.layers.40.self_attn.kv_b_proj as default
Injecting model.layers.40.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.40.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.40.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.40.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.40.mlp.shared_experts as default
Injecting model.layers.40.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.40.mlp.shared_experts.act_fn as default
Injecting model.layers.40.input_layernorm as default
Injecting model.layers.40.post_attention_layernorm as default
Injecting model.layers.41 as default
Injecting model.layers.41.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.41.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.q_a_layernorm as default
Injecting model.layers.41.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.kv_a_layernorm as default
Injecting model.layers.41.self_attn.kv_b_proj as default
Injecting model.layers.41.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.41.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.41.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.41.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.41.mlp.shared_experts as default
Injecting model.layers.41.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.41.mlp.shared_experts.act_fn as default
Injecting model.layers.41.input_layernorm as default
Injecting model.layers.41.post_attention_layernorm as default
Injecting model.layers.42 as default
Injecting model.layers.42.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.42.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.q_a_layernorm as default
Injecting model.layers.42.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.kv_a_layernorm as default
Injecting model.layers.42.self_attn.kv_b_proj as default
Injecting model.layers.42.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.42.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.42.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.42.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.42.mlp.shared_experts as default
Injecting model.layers.42.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.42.mlp.shared_experts.act_fn as default
Injecting model.layers.42.input_layernorm as default
Injecting model.layers.42.post_attention_layernorm as default
Injecting model.layers.43 as default
Injecting model.layers.43.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.43.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.q_a_layernorm as default
Injecting model.layers.43.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.kv_a_layernorm as default
Injecting model.layers.43.self_attn.kv_b_proj as default
Injecting model.layers.43.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.43.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.43.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.43.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.43.mlp.shared_experts as default
Injecting model.layers.43.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.43.mlp.shared_experts.act_fn as default
Injecting model.layers.43.input_layernorm as default
Injecting model.layers.43.post_attention_layernorm as default
Injecting model.layers.44 as default
Injecting model.layers.44.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.44.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.q_a_layernorm as default
Injecting model.layers.44.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.kv_a_layernorm as default
Injecting model.layers.44.self_attn.kv_b_proj as default
Injecting model.layers.44.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.44.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.44.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.44.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.44.mlp.shared_experts as default
Injecting model.layers.44.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.44.mlp.shared_experts.act_fn as default
Injecting model.layers.44.input_layernorm as default
Injecting model.layers.44.post_attention_layernorm as default
Injecting model.layers.45 as default
Injecting model.layers.45.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.45.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.q_a_layernorm as default
Injecting model.layers.45.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.kv_a_layernorm as default
Injecting model.layers.45.self_attn.kv_b_proj as default
Injecting model.layers.45.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.45.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.45.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.45.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.45.mlp.shared_experts as default
Injecting model.layers.45.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.45.mlp.shared_experts.act_fn as default
Injecting model.layers.45.input_layernorm as default
Injecting model.layers.45.post_attention_layernorm as default
Injecting model.layers.46 as default
Injecting model.layers.46.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.46.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.q_a_layernorm as default
Injecting model.layers.46.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.kv_a_layernorm as default
Injecting model.layers.46.self_attn.kv_b_proj as default
Injecting model.layers.46.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.46.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.46.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.46.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.46.mlp.shared_experts as default
Injecting model.layers.46.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.46.mlp.shared_experts.act_fn as default
Injecting model.layers.46.input_layernorm as default
Injecting model.layers.46.post_attention_layernorm as default
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.47.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_a_layernorm as default
Injecting model.layers.47.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.kv_a_layernorm as default
Injecting model.layers.47.self_attn.kv_b_proj as default
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.47.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.47.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.47.mlp.shared_experts as default
Injecting model.layers.47.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.shared_experts.act_fn as default
Injecting model.layers.47.input_layernorm as default
Injecting model.layers.47.post_attention_layernorm as default
Injecting model.layers.48 as default
Injecting model.layers.48.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.48.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.q_a_layernorm as default
Injecting model.layers.48.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.kv_a_layernorm as default
Injecting model.layers.48.self_attn.kv_b_proj as default
Injecting model.layers.48.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.48.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.48.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.48.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.48.mlp.shared_experts as default
Injecting model.layers.48.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.48.mlp.shared_experts.act_fn as default
Injecting model.layers.48.input_layernorm as default
Injecting model.layers.48.post_attention_layernorm as default
Injecting model.layers.49 as default
Injecting model.layers.49.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.49.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.q_a_layernorm as default
Injecting model.layers.49.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.kv_a_layernorm as default
Injecting model.layers.49.self_attn.kv_b_proj as default
Injecting model.layers.49.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.49.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.49.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.49.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.49.mlp.shared_experts as default
Injecting model.layers.49.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.49.mlp.shared_experts.act_fn as default
Injecting model.layers.49.input_layernorm as default
Injecting model.layers.49.post_attention_layernorm as default
Injecting model.layers.50 as default
Injecting model.layers.50.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.50.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.q_a_layernorm as default
Injecting model.layers.50.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.kv_a_layernorm as default
Injecting model.layers.50.self_attn.kv_b_proj as default
Injecting model.layers.50.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.50.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.50.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.50.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.50.mlp.shared_experts as default
Injecting model.layers.50.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.50.mlp.shared_experts.act_fn as default
Injecting model.layers.50.input_layernorm as default
Injecting model.layers.50.post_attention_layernorm as default
Injecting model.layers.51 as default
Injecting model.layers.51.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.51.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.q_a_layernorm as default
Injecting model.layers.51.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.kv_a_layernorm as default
Injecting model.layers.51.self_attn.kv_b_proj as default
Injecting model.layers.51.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.51.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.51.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.51.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.51.mlp.shared_experts as default
Injecting model.layers.51.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.51.mlp.shared_experts.act_fn as default
Injecting model.layers.51.input_layernorm as default
Injecting model.layers.51.post_attention_layernorm as default
Injecting model.layers.52 as default
Injecting model.layers.52.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.52.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.q_a_layernorm as default
Injecting model.layers.52.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.kv_a_layernorm as default
Injecting model.layers.52.self_attn.kv_b_proj as default
Injecting model.layers.52.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.52.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.52.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.52.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.52.mlp.shared_experts as default
Injecting model.layers.52.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.52.mlp.shared_experts.act_fn as default
Injecting model.layers.52.input_layernorm as default
Injecting model.layers.52.post_attention_layernorm as default
Injecting model.layers.53 as default
Injecting model.layers.53.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.53.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.q_a_layernorm as default
Injecting model.layers.53.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.kv_a_layernorm as default
Injecting model.layers.53.self_attn.kv_b_proj as default
Injecting model.layers.53.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.53.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.53.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.53.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.53.mlp.shared_experts as default
Injecting model.layers.53.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.53.mlp.shared_experts.act_fn as default
Injecting model.layers.53.input_layernorm as default
Injecting model.layers.53.post_attention_layernorm as default
Injecting model.layers.54 as default
Injecting model.layers.54.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.54.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.q_a_layernorm as default
Injecting model.layers.54.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.kv_a_layernorm as default
Injecting model.layers.54.self_attn.kv_b_proj as default
Injecting model.layers.54.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.54.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.54.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.54.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.54.mlp.shared_experts as default
Injecting model.layers.54.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.54.mlp.shared_experts.act_fn as default
Injecting model.layers.54.input_layernorm as default
Injecting model.layers.54.post_attention_layernorm as default
Injecting model.layers.55 as default
Injecting model.layers.55.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.55.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.q_a_layernorm as default
Injecting model.layers.55.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.kv_a_layernorm as default
Injecting model.layers.55.self_attn.kv_b_proj as default
Injecting model.layers.55.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.55.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.55.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.55.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.55.mlp.shared_experts as default
Injecting model.layers.55.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.55.mlp.shared_experts.act_fn as default
Injecting model.layers.55.input_layernorm as default
Injecting model.layers.55.post_attention_layernorm as default
Injecting model.layers.56 as default
Injecting model.layers.56.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.56.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.q_a_layernorm as default
Injecting model.layers.56.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.kv_a_layernorm as default
Injecting model.layers.56.self_attn.kv_b_proj as default
Injecting model.layers.56.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.56.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.56.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.56.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.56.mlp.shared_experts as default
Injecting model.layers.56.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.56.mlp.shared_experts.act_fn as default
Injecting model.layers.56.input_layernorm as default
Injecting model.layers.56.post_attention_layernorm as default
Injecting model.layers.57 as default
Injecting model.layers.57.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.57.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.q_a_layernorm as default
Injecting model.layers.57.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.kv_a_layernorm as default
Injecting model.layers.57.self_attn.kv_b_proj as default
Injecting model.layers.57.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.57.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.57.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.57.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.57.mlp.shared_experts as default
Injecting model.layers.57.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.57.mlp.shared_experts.act_fn as default
Injecting model.layers.57.input_layernorm as default
Injecting model.layers.57.post_attention_layernorm as default
Injecting model.layers.58 as default
Injecting model.layers.58.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.58.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.q_a_layernorm as default
Injecting model.layers.58.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.kv_a_layernorm as default
Injecting model.layers.58.self_attn.kv_b_proj as default
Injecting model.layers.58.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.58.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.58.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.58.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.58.mlp.shared_experts as default
Injecting model.layers.58.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.58.mlp.shared_experts.act_fn as default
Injecting model.layers.58.input_layernorm as default
Injecting model.layers.58.post_attention_layernorm as default
Injecting model.layers.59 as default
Injecting model.layers.59.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.59.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.q_a_layernorm as default
Injecting model.layers.59.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.kv_a_layernorm as default
Injecting model.layers.59.self_attn.kv_b_proj as default
Injecting model.layers.59.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.59.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.59.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.59.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.59.mlp.shared_experts as default
Injecting model.layers.59.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.59.mlp.shared_experts.act_fn as default
Injecting model.layers.59.input_layernorm as default
Injecting model.layers.59.post_attention_layernorm as default
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading output to cuda using CPUInfer
loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_q_a.weight with CPU
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_q_b.weight with CPU
loading blk.0.attn_kv_a_mqa.weight with CPU
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_output.weight with CPU
loading blk.0.ffn_gate.weight with CPU
loading blk.0.ffn_up.weight with CPU
loading blk.0.ffn_down.weight with CPU
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_q_a.weight with CPU
loading blk.1.attn_q_a_norm.weight to cuda:0
loading blk.1.attn_q_b.weight with CPU
loading blk.1.attn_kv_a_mqa.weight with CPU
loading blk.1.attn_kv_a_norm.weight to cuda:0
loading blk.1.attn_kv_b.weight to cuda:0
loading blk.1.attn_output.weight with CPU
loading blk.1.ffn_gate.weight with CPU
loading blk.1.ffn_up.weight with CPU
loading blk.1.ffn_down.weight with CPU
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_q_a.weight with CPU
loading blk.2.attn_q_a_norm.weight to cuda:0
loading blk.2.attn_q_b.weight with CPU
loading blk.2.attn_kv_a_mqa.weight with CPU
loading blk.2.attn_kv_a_norm.weight to cuda:0
loading blk.2.attn_kv_b.weight to cuda:0
loading blk.2.attn_output.weight with CPU
loading blk.2.ffn_gate.weight with CPU
loading blk.2.ffn_up.weight with CPU
loading blk.2.ffn_down.weight with CPU
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_q_a.weight with CPU
loading blk.3.attn_q_a_norm.weight to cuda:0
loading blk.3.attn_q_b.weight with CPU
loading blk.3.attn_kv_a_mqa.weight with CPU
loading blk.3.attn_kv_a_norm.weight to cuda:0
loading blk.3.attn_kv_b.weight to cuda:0
loading blk.3.attn_output.weight with CPU
loading blk.3.ffn_gate_shexp.weight with CPU
loading blk.3.ffn_up_shexp.weight with CPU
loading blk.3.ffn_down_shexp.weight with CPU
loading blk.3.attn_norm.weight to cuda:0
loading blk.3.ffn_norm.weight to cuda:0
loading blk.4.attn_q_a.weight with CPU
loading blk.4.attn_q_a_norm.weight to cuda:0
loading blk.4.attn_q_b.weight with CPU
loading blk.4.attn_kv_a_mqa.weight with CPU
loading blk.4.attn_kv_a_norm.weight to cuda:0
loading blk.4.attn_kv_b.weight to cuda:0
loading blk.4.attn_output.weight with CPU
loading blk.4.ffn_gate_shexp.weight with CPU
loading blk.4.ffn_up_shexp.weight with CPU
loading blk.4.ffn_down_shexp.weight with CPU
loading blk.4.attn_norm.weight to cuda:0
loading blk.4.ffn_norm.weight to cuda:0
loading blk.5.attn_q_a.weight with CPU
loading blk.5.attn_q_a_norm.weight to cuda:0
loading blk.5.attn_q_b.weight with CPU
loading blk.5.attn_kv_a_mqa.weight with CPU
loading blk.5.attn_kv_a_norm.weight to cuda:0
loading blk.5.attn_kv_b.weight to cuda:0
loading blk.5.attn_output.weight with CPU
loading blk.5.ffn_gate_shexp.weight with CPU
loading blk.5.ffn_up_shexp.weight with CPU
loading blk.5.ffn_down_shexp.weight with CPU
loading blk.5.attn_norm.weight to cuda:0
loading blk.5.ffn_norm.weight to cuda:0
loading blk.6.attn_q_a.weight with CPU
loading blk.6.attn_q_a_norm.weight to cuda:0
loading blk.6.attn_q_b.weight with CPU
loading blk.6.attn_kv_a_mqa.weight with CPU
loading blk.6.attn_kv_a_norm.weight to cuda:0
loading blk.6.attn_kv_b.weight to cuda:0
loading blk.6.attn_output.weight with CPU
loading blk.6.ffn_gate_shexp.weight with CPU
loading blk.6.ffn_up_shexp.weight with CPU
loading blk.6.ffn_down_shexp.weight with CPU
loading blk.6.attn_norm.weight to cuda:0
loading blk.6.ffn_norm.weight to cuda:0
loading blk.7.attn_q_a.weight with CPU
loading blk.7.attn_q_a_norm.weight to cuda:0
loading blk.7.attn_q_b.weight with CPU
loading blk.7.attn_kv_a_mqa.weight with CPU
loading blk.7.attn_kv_a_norm.weight to cuda:0
loading blk.7.attn_kv_b.weight to cuda:0
loading blk.7.attn_output.weight with CPU
loading blk.7.ffn_gate_shexp.weight with CPU
loading blk.7.ffn_up_shexp.weight with CPU
loading blk.7.ffn_down_shexp.weight with CPU
loading blk.7.attn_norm.weight to cuda:0
loading blk.7.ffn_norm.weight to cuda:0
loading blk.8.attn_q_a.weight with CPU
loading blk.8.attn_q_a_norm.weight to cuda:0
loading blk.8.attn_q_b.weight with CPU
loading blk.8.attn_kv_a_mqa.weight with CPU
loading blk.8.attn_kv_a_norm.weight to cuda:0
loading blk.8.attn_kv_b.weight to cuda:0
loading blk.8.attn_output.weight with CPU
loading blk.8.ffn_gate_shexp.weight with CPU
loading blk.8.ffn_up_shexp.weight with CPU
loading blk.8.ffn_down_shexp.weight with CPU
loading blk.8.attn_norm.weight to cuda:0
loading blk.8.ffn_norm.weight to cuda:0
loading blk.9.attn_q_a.weight with CPU
loading blk.9.attn_q_a_norm.weight to cuda:0
loading blk.9.attn_q_b.weight with CPU
loading blk.9.attn_kv_a_mqa.weight with CPU
loading blk.9.attn_kv_a_norm.weight to cuda:0
loading blk.9.attn_kv_b.weight to cuda:0
loading blk.9.attn_output.weight with CPU
loading blk.9.ffn_gate_shexp.weight with CPU
loading blk.9.ffn_up_shexp.weight with CPU
loading blk.9.ffn_down_shexp.weight with CPU
loading blk.9.attn_norm.weight to cuda:0
loading blk.9.ffn_norm.weight to cuda:0
loading blk.10.attn_q_a.weight with CPU
loading blk.10.attn_q_a_norm.weight to cuda:0
loading blk.10.attn_q_b.weight with CPU
loading blk.10.attn_kv_a_mqa.weight with CPU
loading blk.10.attn_kv_a_norm.weight to cuda:0
loading blk.10.attn_kv_b.weight to cuda:0
loading blk.10.attn_output.weight with CPU
loading blk.10.ffn_gate_shexp.weight with CPU
loading blk.10.ffn_up_shexp.weight with CPU
loading blk.10.ffn_down_shexp.weight with CPU
loading blk.10.attn_norm.weight to cuda:0
loading blk.10.ffn_norm.weight to cuda:0
loading blk.11.attn_q_a.weight with CPU
loading blk.11.attn_q_a_norm.weight to cuda:0
loading blk.11.attn_q_b.weight with CPU
loading blk.11.attn_kv_a_mqa.weight with CPU
loading blk.11.attn_kv_a_norm.weight to cuda:0
loading blk.11.attn_kv_b.weight to cuda:0
loading blk.11.attn_output.weight with CPU
loading blk.11.ffn_gate_shexp.weight with CPU
loading blk.11.ffn_up_shexp.weight with CPU
loading blk.11.ffn_down_shexp.weight with CPU
loading blk.11.attn_norm.weight to cuda:0
loading blk.11.ffn_norm.weight to cuda:0
loading blk.12.attn_q_a.weight with CPU
loading blk.12.attn_q_a_norm.weight to cuda:0
loading blk.12.attn_q_b.weight with CPU
loading blk.12.attn_kv_a_mqa.weight with CPU
loading blk.12.attn_kv_a_norm.weight to cuda:0
loading blk.12.attn_kv_b.weight to cuda:0
loading blk.12.attn_output.weight with CPU
loading blk.12.ffn_gate_shexp.weight with CPU
loading blk.12.ffn_up_shexp.weight with CPU
loading blk.12.ffn_down_shexp.weight with CPU
loading blk.12.attn_norm.weight to cuda:0
loading blk.12.ffn_norm.weight to cuda:0
loading blk.13.attn_q_a.weight with CPU
loading blk.13.attn_q_a_norm.weight to cuda:0
loading blk.13.attn_q_b.weight with CPU
loading blk.13.attn_kv_a_mqa.weight with CPU
loading blk.13.attn_kv_a_norm.weight to cuda:0
loading blk.13.attn_kv_b.weight to cuda:0
loading blk.13.attn_output.weight with CPU
loading blk.13.ffn_gate_shexp.weight with CPU
loading blk.13.ffn_up_shexp.weight with CPU
loading blk.13.ffn_down_shexp.weight with CPU
loading blk.13.attn_norm.weight to cuda:0
loading blk.13.ffn_norm.weight to cuda:0
loading blk.14.attn_q_a.weight with CPU
loading blk.14.attn_q_a_norm.weight to cuda:0
loading blk.14.attn_q_b.weight with CPU
loading blk.14.attn_kv_a_mqa.weight with CPU
loading blk.14.attn_kv_a_norm.weight to cuda:0
loading blk.14.attn_kv_b.weight to cuda:0
loading blk.14.attn_output.weight with CPU
loading blk.14.ffn_gate_shexp.weight with CPU
loading blk.14.ffn_up_shexp.weight with CPU
loading blk.14.ffn_down_shexp.weight with CPU
loading blk.14.attn_norm.weight to cuda:0
loading blk.14.ffn_norm.weight to cuda:0
loading blk.15.attn_q_a.weight with CPU
loading blk.15.attn_q_a_norm.weight to cuda:0
loading blk.15.attn_q_b.weight with CPU
loading blk.15.attn_kv_a_mqa.weight with CPU
loading blk.15.attn_kv_a_norm.weight to cuda:0
loading blk.15.attn_kv_b.weight to cuda:0
loading blk.15.attn_output.weight with CPU
loading blk.15.ffn_gate_shexp.weight with CPU
loading blk.15.ffn_up_shexp.weight with CPU
loading blk.15.ffn_down_shexp.weight with CPU
loading blk.15.attn_norm.weight to cuda:0
loading blk.15.ffn_norm.weight to cuda:0
loading blk.16.attn_q_a.weight with CPU
loading blk.16.attn_q_a_norm.weight to cuda:0
loading blk.16.attn_q_b.weight with CPU
loading blk.16.attn_kv_a_mqa.weight with CPU
loading blk.16.attn_kv_a_norm.weight to cuda:0
loading blk.16.attn_kv_b.weight to cuda:0
loading blk.16.attn_output.weight with CPU
loading blk.16.ffn_gate_shexp.weight with CPU
loading blk.16.ffn_up_shexp.weight with CPU
loading blk.16.ffn_down_shexp.weight with CPU
loading blk.16.attn_norm.weight to cuda:0
loading blk.16.ffn_norm.weight to cuda:0
loading blk.17.attn_q_a.weight with CPU
loading blk.17.attn_q_a_norm.weight to cuda:0
loading blk.17.attn_q_b.weight with CPU
loading blk.17.attn_kv_a_mqa.weight with CPU
loading blk.17.attn_kv_a_norm.weight to cuda:0
loading blk.17.attn_kv_b.weight to cuda:0
loading blk.17.attn_output.weight with CPU
loading blk.17.ffn_gate_shexp.weight with CPU
loading blk.17.ffn_up_shexp.weight with CPU
loading blk.17.ffn_down_shexp.weight with CPU
loading blk.17.attn_norm.weight to cuda:0
loading blk.17.ffn_norm.weight to cuda:0
loading blk.18.attn_q_a.weight with CPU
loading blk.18.attn_q_a_norm.weight to cuda:0
loading blk.18.attn_q_b.weight with CPU
loading blk.18.attn_kv_a_mqa.weight with CPU
loading blk.18.attn_kv_a_norm.weight to cuda:0
loading blk.18.attn_kv_b.weight to cuda:0
loading blk.18.attn_output.weight with CPU
loading blk.18.ffn_gate_shexp.weight with CPU
loading blk.18.ffn_up_shexp.weight with CPU
loading blk.18.ffn_down_shexp.weight with CPU
loading blk.18.attn_norm.weight to cuda:0
loading blk.18.ffn_norm.weight to cuda:0
loading blk.19.attn_q_a.weight with CPU
loading blk.19.attn_q_a_norm.weight to cuda:0
loading blk.19.attn_q_b.weight with CPU
loading blk.19.attn_kv_a_mqa.weight with CPU
loading blk.19.attn_kv_a_norm.weight to cuda:0
loading blk.19.attn_kv_b.weight to cuda:0
loading blk.19.attn_output.weight with CPU
loading blk.19.ffn_gate_shexp.weight with CPU
loading blk.19.ffn_up_shexp.weight with CPU
loading blk.19.ffn_down_shexp.weight with CPU
loading blk.19.attn_norm.weight to cuda:0
loading blk.19.ffn_norm.weight to cuda:0
loading blk.20.attn_q_a.weight with CPU
loading blk.20.attn_q_a_norm.weight to cuda:0
loading blk.20.attn_q_b.weight with CPU
loading blk.20.attn_kv_a_mqa.weight with CPU
loading blk.20.attn_kv_a_norm.weight to cuda:0
loading blk.20.attn_kv_b.weight to cuda:0
loading blk.20.attn_output.weight with CPU
loading blk.20.ffn_gate_shexp.weight with CPU
loading blk.20.ffn_up_shexp.weight with CPU
loading blk.20.ffn_down_shexp.weight with CPU
loading blk.20.attn_norm.weight to cuda:0
loading blk.20.ffn_norm.weight to cuda:0
loading blk.21.attn_q_a.weight with CPU
loading blk.21.attn_q_a_norm.weight to cuda:0
loading blk.21.attn_q_b.weight with CPU
loading blk.21.attn_kv_a_mqa.weight with CPU
loading blk.21.attn_kv_a_norm.weight to cuda:0
loading blk.21.attn_kv_b.weight to cuda:0
loading blk.21.attn_output.weight with CPU
loading blk.21.ffn_gate_shexp.weight with CPU
loading blk.21.ffn_up_shexp.weight with CPU
loading blk.21.ffn_down_shexp.weight with CPU
loading blk.21.attn_norm.weight to cuda:0
loading blk.21.ffn_norm.weight to cuda:0
loading blk.22.attn_q_a.weight with CPU
loading blk.22.attn_q_a_norm.weight to cuda:0
loading blk.22.attn_q_b.weight with CPU
loading blk.22.attn_kv_a_mqa.weight with CPU
loading blk.22.attn_kv_a_norm.weight to cuda:0
loading blk.22.attn_kv_b.weight to cuda:0
loading blk.22.attn_output.weight with CPU
loading blk.22.ffn_gate_shexp.weight with CPU
loading blk.22.ffn_up_shexp.weight with CPU
loading blk.22.ffn_down_shexp.weight with CPU
loading blk.22.attn_norm.weight to cuda:0
loading blk.22.ffn_norm.weight to cuda:0
loading blk.23.attn_q_a.weight with CPU
loading blk.23.attn_q_a_norm.weight to cuda:0
loading blk.23.attn_q_b.weight with CPU
loading blk.23.attn_kv_a_mqa.weight with CPU
loading blk.23.attn_kv_a_norm.weight to cuda:0
loading blk.23.attn_kv_b.weight to cuda:0
loading blk.23.attn_output.weight with CPU
loading blk.23.ffn_gate_shexp.weight with CPU
loading blk.23.ffn_up_shexp.weight with CPU
loading blk.23.ffn_down_shexp.weight with CPU
loading blk.23.attn_norm.weight to cuda:0
loading blk.23.ffn_norm.weight to cuda:0
loading blk.24.attn_q_a.weight with CPU
loading blk.24.attn_q_a_norm.weight to cuda:0
loading blk.24.attn_q_b.weight with CPU
loading blk.24.attn_kv_a_mqa.weight with CPU
loading blk.24.attn_kv_a_norm.weight to cuda:0
loading blk.24.attn_kv_b.weight to cuda:0
loading blk.24.attn_output.weight with CPU
loading blk.24.ffn_gate_shexp.weight with CPU
loading blk.24.ffn_up_shexp.weight with CPU
loading blk.24.ffn_down_shexp.weight with CPU
loading blk.24.attn_norm.weight to cuda:0
loading blk.24.ffn_norm.weight to cuda:0
loading blk.25.attn_q_a.weight with CPU
loading blk.25.attn_q_a_norm.weight to cuda:0
loading blk.25.attn_q_b.weight with CPU
loading blk.25.attn_kv_a_mqa.weight with CPU
loading blk.25.attn_kv_a_norm.weight to cuda:0
loading blk.25.attn_kv_b.weight to cuda:0
loading blk.25.attn_output.weight with CPU
loading blk.25.ffn_gate_shexp.weight with CPU
loading blk.25.ffn_up_shexp.weight with CPU
loading blk.25.ffn_down_shexp.weight with CPU
loading blk.25.attn_norm.weight to cuda:0
loading blk.25.ffn_norm.weight to cuda:0
loading blk.26.attn_q_a.weight with CPU
loading blk.26.attn_q_a_norm.weight to cuda:0
loading blk.26.attn_q_b.weight with CPU
loading blk.26.attn_kv_a_mqa.weight with CPU
loading blk.26.attn_kv_a_norm.weight to cuda:0
loading blk.26.attn_kv_b.weight to cuda:0
loading blk.26.attn_output.weight with CPU
loading blk.26.ffn_gate_shexp.weight with CPU
loading blk.26.ffn_up_shexp.weight with CPU
loading blk.26.ffn_down_shexp.weight with CPU
loading blk.26.attn_norm.weight to cuda:0
loading blk.26.ffn_norm.weight to cuda:0
loading blk.27.attn_q_a.weight with CPU
loading blk.27.attn_q_a_norm.weight to cuda:0
loading blk.27.attn_q_b.weight with CPU
loading blk.27.attn_kv_a_mqa.weight with CPU
loading blk.27.attn_kv_a_norm.weight to cuda:0
loading blk.27.attn_kv_b.weight to cuda:0
loading blk.27.attn_output.weight with CPU
loading blk.27.ffn_gate_shexp.weight with CPU
loading blk.27.ffn_up_shexp.weight with CPU
loading blk.27.ffn_down_shexp.weight with CPU
loading blk.27.attn_norm.weight to cuda:0
loading blk.27.ffn_norm.weight to cuda:0
loading blk.28.attn_q_a.weight with CPU
loading blk.28.attn_q_a_norm.weight to cuda:0
loading blk.28.attn_q_b.weight with CPU
loading blk.28.attn_kv_a_mqa.weight with CPU
loading blk.28.attn_kv_a_norm.weight to cuda:0
loading blk.28.attn_kv_b.weight to cuda:0
loading blk.28.attn_output.weight with CPU
loading blk.28.ffn_gate_shexp.weight with CPU
loading blk.28.ffn_up_shexp.weight with CPU
loading blk.28.ffn_down_shexp.weight with CPU
loading blk.28.attn_norm.weight to cuda:0
loading blk.28.ffn_norm.weight to cuda:0
loading blk.29.attn_q_a.weight with CPU
loading blk.29.attn_q_a_norm.weight to cuda:0
loading blk.29.attn_q_b.weight with CPU
loading blk.29.attn_kv_a_mqa.weight with CPU
loading blk.29.attn_kv_a_norm.weight to cuda:0
loading blk.29.attn_kv_b.weight to cuda:0
loading blk.29.attn_output.weight with CPU
loading blk.29.ffn_gate_shexp.weight with CPU
loading blk.29.ffn_up_shexp.weight with CPU
loading blk.29.ffn_down_shexp.weight with CPU
loading blk.29.attn_norm.weight to cuda:0
loading blk.29.ffn_norm.weight to cuda:0
loading blk.30.attn_q_a.weight with CPU
loading blk.30.attn_q_a_norm.weight to cuda:0
loading blk.30.attn_q_b.weight with CPU
loading blk.30.attn_kv_a_mqa.weight with CPU
loading blk.30.attn_kv_a_norm.weight to cuda:0
loading blk.30.attn_kv_b.weight to cuda:0
loading blk.30.attn_output.weight with CPU
loading blk.30.ffn_gate_shexp.weight with CPU
loading blk.30.ffn_up_shexp.weight with CPU
loading blk.30.ffn_down_shexp.weight with CPU
loading blk.30.attn_norm.weight to cuda:0
loading blk.30.ffn_norm.weight to cuda:0
loading blk.31.attn_q_a.weight with CPU
loading blk.31.attn_q_a_norm.weight to cuda:0
loading blk.31.attn_q_b.weight with CPU
loading blk.31.attn_kv_a_mqa.weight with CPU
loading blk.31.attn_kv_a_norm.weight to cuda:0
loading blk.31.attn_kv_b.weight to cuda:0
loading blk.31.attn_output.weight with CPU
loading blk.31.ffn_gate_shexp.weight with CPU
loading blk.31.ffn_up_shexp.weight with CPU
loading blk.31.ffn_down_shexp.weight with CPU
loading blk.31.attn_norm.weight to cuda:0
loading blk.31.ffn_norm.weight to cuda:0
loading blk.32.attn_q_a.weight with CPU
loading blk.32.attn_q_a_norm.weight to cuda:0
loading blk.32.attn_q_b.weight with CPU
loading blk.32.attn_kv_a_mqa.weight with CPU
loading blk.32.attn_kv_a_norm.weight to cuda:0
loading blk.32.attn_kv_b.weight to cuda:0
loading blk.32.attn_output.weight with CPU
loading blk.32.ffn_gate_shexp.weight with CPU
loading blk.32.ffn_up_shexp.weight with CPU
loading blk.32.ffn_down_shexp.weight with CPU
loading blk.32.attn_norm.weight to cuda:0
loading blk.32.ffn_norm.weight to cuda:0
loading blk.33.attn_q_a.weight with CPU
loading blk.33.attn_q_a_norm.weight to cuda:0
loading blk.33.attn_q_b.weight with CPU
loading blk.33.attn_kv_a_mqa.weight with CPU
loading blk.33.attn_kv_a_norm.weight to cuda:0
loading blk.33.attn_kv_b.weight to cuda:0
loading blk.33.attn_output.weight with CPU
loading blk.33.ffn_gate_shexp.weight with CPU
loading blk.33.ffn_up_shexp.weight with CPU
loading blk.33.ffn_down_shexp.weight with CPU
loading blk.33.attn_norm.weight to cuda:0
loading blk.33.ffn_norm.weight to cuda:0
loading blk.34.attn_q_a.weight with CPU
loading blk.34.attn_q_a_norm.weight to cuda:0
loading blk.34.attn_q_b.weight with CPU
loading blk.34.attn_kv_a_mqa.weight with CPU
loading blk.34.attn_kv_a_norm.weight to cuda:0
loading blk.34.attn_kv_b.weight to cuda:0
loading blk.34.attn_output.weight with CPU
loading blk.34.ffn_gate_shexp.weight with CPU
loading blk.34.ffn_up_shexp.weight with CPU
loading blk.34.ffn_down_shexp.weight with CPU
loading blk.34.attn_norm.weight to cuda:0
loading blk.34.ffn_norm.weight to cuda:0
loading blk.35.attn_q_a.weight with CPU
loading blk.35.attn_q_a_norm.weight to cuda:0
loading blk.35.attn_q_b.weight with CPU
loading blk.35.attn_kv_a_mqa.weight with CPU
loading blk.35.attn_kv_a_norm.weight to cuda:0
loading blk.35.attn_kv_b.weight to cuda:0
loading blk.35.attn_output.weight with CPU
loading blk.35.ffn_gate_shexp.weight with CPU
loading blk.35.ffn_up_shexp.weight with CPU
loading blk.35.ffn_down_shexp.weight with CPU
loading blk.35.attn_norm.weight to cuda:0
loading blk.35.ffn_norm.weight to cuda:0
loading blk.36.attn_q_a.weight with CPU
loading blk.36.attn_q_a_norm.weight to cuda:0
loading blk.36.attn_q_b.weight with CPU
loading blk.36.attn_kv_a_mqa.weight with CPU
loading blk.36.attn_kv_a_norm.weight to cuda:0
loading blk.36.attn_kv_b.weight to cuda:0
loading blk.36.attn_output.weight with CPU
loading blk.36.ffn_gate_shexp.weight with CPU
loading blk.36.ffn_up_shexp.weight with CPU
loading blk.36.ffn_down_shexp.weight with CPU
loading blk.36.attn_norm.weight to cuda:0
loading blk.36.ffn_norm.weight to cuda:0
loading blk.37.attn_q_a.weight with CPU
loading blk.37.attn_q_a_norm.weight to cuda:0
loading blk.37.attn_q_b.weight with CPU
loading blk.37.attn_kv_a_mqa.weight with CPU
loading blk.37.attn_kv_a_norm.weight to cuda:0
loading blk.37.attn_kv_b.weight to cuda:0
loading blk.37.attn_output.weight with CPU
loading blk.37.ffn_gate_shexp.weight with CPU
loading blk.37.ffn_up_shexp.weight with CPU
loading blk.37.ffn_down_shexp.weight with CPU
loading blk.37.attn_norm.weight to cuda:0
loading blk.37.ffn_norm.weight to cuda:0
loading blk.38.attn_q_a.weight with CPU
loading blk.38.attn_q_a_norm.weight to cuda:0
loading blk.38.attn_q_b.weight with CPU
loading blk.38.attn_kv_a_mqa.weight with CPU
loading blk.38.attn_kv_a_norm.weight to cuda:0
loading blk.38.attn_kv_b.weight to cuda:0
loading blk.38.attn_output.weight with CPU
loading blk.38.ffn_gate_shexp.weight with CPU
loading blk.38.ffn_up_shexp.weight with CPU
loading blk.38.ffn_down_shexp.weight with CPU
loading blk.38.attn_norm.weight to cuda:0
loading blk.38.ffn_norm.weight to cuda:0
loading blk.39.attn_q_a.weight with CPU
loading blk.39.attn_q_a_norm.weight to cuda:0
loading blk.39.attn_q_b.weight with CPU
loading blk.39.attn_kv_a_mqa.weight with CPU
loading blk.39.attn_kv_a_norm.weight to cuda:0
loading blk.39.attn_kv_b.weight to cuda:0
loading blk.39.attn_output.weight with CPU
loading blk.39.ffn_gate_shexp.weight with CPU
loading blk.39.ffn_up_shexp.weight with CPU
loading blk.39.ffn_down_shexp.weight with CPU
loading blk.39.attn_norm.weight to cuda:0
loading blk.39.ffn_norm.weight to cuda:0
loading blk.40.attn_q_a.weight with CPU
loading blk.40.attn_q_a_norm.weight to cuda:0
loading blk.40.attn_q_b.weight with CPU
loading blk.40.attn_kv_a_mqa.weight with CPU
loading blk.40.attn_kv_a_norm.weight to cuda:0
loading blk.40.attn_kv_b.weight to cuda:0
loading blk.40.attn_output.weight with CPU
loading blk.40.ffn_gate_shexp.weight with CPU
loading blk.40.ffn_up_shexp.weight with CPU
loading blk.40.ffn_down_shexp.weight with CPU
loading blk.40.attn_norm.weight to cuda:0
loading blk.40.ffn_norm.weight to cuda:0
loading blk.41.attn_q_a.weight with CPU
loading blk.41.attn_q_a_norm.weight to cuda:0
loading blk.41.attn_q_b.weight with CPU
loading blk.41.attn_kv_a_mqa.weight with CPU
loading blk.41.attn_kv_a_norm.weight to cuda:0
loading blk.41.attn_kv_b.weight to cuda:0
loading blk.41.attn_output.weight with CPU
loading blk.41.ffn_gate_shexp.weight with CPU
loading blk.41.ffn_up_shexp.weight with CPU
loading blk.41.ffn_down_shexp.weight with CPU
loading blk.41.attn_norm.weight to cuda:0
loading blk.41.ffn_norm.weight to cuda:0
loading blk.42.attn_q_a.weight with CPU
loading blk.42.attn_q_a_norm.weight to cuda:0
loading blk.42.attn_q_b.weight with CPU
loading blk.42.attn_kv_a_mqa.weight with CPU
loading blk.42.attn_kv_a_norm.weight to cuda:0
loading blk.42.attn_kv_b.weight to cuda:0
loading blk.42.attn_output.weight with CPU
loading blk.42.ffn_gate_shexp.weight with CPU
loading blk.42.ffn_up_shexp.weight with CPU
loading blk.42.ffn_down_shexp.weight with CPU
loading blk.42.attn_norm.weight to cuda:0
loading blk.42.ffn_norm.weight to cuda:0
loading blk.43.attn_q_a.weight with CPU
loading blk.43.attn_q_a_norm.weight to cuda:0
loading blk.43.attn_q_b.weight with CPU
loading blk.43.attn_kv_a_mqa.weight with CPU
loading blk.43.attn_kv_a_norm.weight to cuda:0
loading blk.43.attn_kv_b.weight to cuda:0
loading blk.43.attn_output.weight with CPU
loading blk.43.ffn_gate_shexp.weight with CPU
loading blk.43.ffn_up_shexp.weight with CPU
loading blk.43.ffn_down_shexp.weight with CPU
loading blk.43.attn_norm.weight to cuda:0
loading blk.43.ffn_norm.weight to cuda:0
loading blk.44.attn_q_a.weight with CPU
loading blk.44.attn_q_a_norm.weight to cuda:0
loading blk.44.attn_q_b.weight with CPU
loading blk.44.attn_kv_a_mqa.weight with CPU
loading blk.44.attn_kv_a_norm.weight to cuda:0
loading blk.44.attn_kv_b.weight to cuda:0
loading blk.44.attn_output.weight with CPU
loading blk.44.ffn_gate_shexp.weight with CPU
loading blk.44.ffn_up_shexp.weight with CPU
loading blk.44.ffn_down_shexp.weight with CPU
loading blk.44.attn_norm.weight to cuda:0
loading blk.44.ffn_norm.weight to cuda:0
loading blk.45.attn_q_a.weight with CPU
loading blk.45.attn_q_a_norm.weight to cuda:0
loading blk.45.attn_q_b.weight with CPU
loading blk.45.attn_kv_a_mqa.weight with CPU
loading blk.45.attn_kv_a_norm.weight to cuda:0
loading blk.45.attn_kv_b.weight to cuda:0
loading blk.45.attn_output.weight with CPU
loading blk.45.ffn_gate_shexp.weight with CPU
loading blk.45.ffn_up_shexp.weight with CPU
loading blk.45.ffn_down_shexp.weight with CPU
loading blk.45.attn_norm.weight to cuda:0
loading blk.45.ffn_norm.weight to cuda:0
loading blk.46.attn_q_a.weight with CPU
loading blk.46.attn_q_a_norm.weight to cuda:0
loading blk.46.attn_q_b.weight with CPU
loading blk.46.attn_kv_a_mqa.weight with CPU
loading blk.46.attn_kv_a_norm.weight to cuda:0
loading blk.46.attn_kv_b.weight to cuda:0
loading blk.46.attn_output.weight with CPU
loading blk.46.ffn_gate_shexp.weight with CPU
loading blk.46.ffn_up_shexp.weight with CPU
loading blk.46.ffn_down_shexp.weight with CPU
loading blk.46.attn_norm.weight to cuda:0
loading blk.46.ffn_norm.weight to cuda:0
loading blk.47.attn_q_a.weight with CPU
loading blk.47.attn_q_a_norm.weight to cuda:0
loading blk.47.attn_q_b.weight with CPU
loading blk.47.attn_kv_a_mqa.weight with CPU
loading blk.47.attn_kv_a_norm.weight to cuda:0
loading blk.47.attn_kv_b.weight to cuda:0
loading blk.47.attn_output.weight with CPU
loading blk.47.ffn_gate_shexp.weight with CPU
loading blk.47.ffn_up_shexp.weight with CPU
loading blk.47.ffn_down_shexp.weight with CPU
loading blk.47.attn_norm.weight to cuda:0
loading blk.47.ffn_norm.weight to cuda:0
loading blk.48.attn_q_a.weight with CPU
loading blk.48.attn_q_a_norm.weight to cuda:0
loading blk.48.attn_q_b.weight with CPU
loading blk.48.attn_kv_a_mqa.weight with CPU
loading blk.48.attn_kv_a_norm.weight to cuda:0
loading blk.48.attn_kv_b.weight to cuda:0
loading blk.48.attn_output.weight with CPU
loading blk.48.ffn_gate_shexp.weight with CPU
loading blk.48.ffn_up_shexp.weight with CPU
loading blk.48.ffn_down_shexp.weight with CPU
loading blk.48.attn_norm.weight to cuda:0
loading blk.48.ffn_norm.weight to cuda:0
loading blk.49.attn_q_a.weight with CPU
loading blk.49.attn_q_a_norm.weight to cuda:0
loading blk.49.attn_q_b.weight with CPU
loading blk.49.attn_kv_a_mqa.weight with CPU
loading blk.49.attn_kv_a_norm.weight to cuda:0
loading blk.49.attn_kv_b.weight to cuda:0
loading blk.49.attn_output.weight with CPU
loading blk.49.ffn_gate_shexp.weight with CPU
loading blk.49.ffn_up_shexp.weight with CPU
loading blk.49.ffn_down_shexp.weight with CPU
loading blk.49.attn_norm.weight to cuda:0
loading blk.49.ffn_norm.weight to cuda:0
loading blk.50.attn_q_a.weight with CPU
loading blk.50.attn_q_a_norm.weight to cuda:0
loading blk.50.attn_q_b.weight with CPU
loading blk.50.attn_kv_a_mqa.weight with CPU
loading blk.50.attn_kv_a_norm.weight to cuda:0
loading blk.50.attn_kv_b.weight to cuda:0
loading blk.50.attn_output.weight with CPU
loading blk.50.ffn_gate_shexp.weight with CPU
loading blk.50.ffn_up_shexp.weight with CPU
loading blk.50.ffn_down_shexp.weight with CPU
loading blk.50.attn_norm.weight to cuda:0
loading blk.50.ffn_norm.weight to cuda:0
loading blk.51.attn_q_a.weight with CPU
loading blk.51.attn_q_a_norm.weight to cuda:0
loading blk.51.attn_q_b.weight with CPU
loading blk.51.attn_kv_a_mqa.weight with CPU
loading blk.51.attn_kv_a_norm.weight to cuda:0
loading blk.51.attn_kv_b.weight to cuda:0
loading blk.51.attn_output.weight with CPU
loading blk.51.ffn_gate_shexp.weight with CPU
loading blk.51.ffn_up_shexp.weight with CPU
loading blk.51.ffn_down_shexp.weight with CPU
loading blk.51.attn_norm.weight to cuda:0
loading blk.51.ffn_norm.weight to cuda:0
loading blk.52.attn_q_a.weight with CPU
loading blk.52.attn_q_a_norm.weight to cuda:0
loading blk.52.attn_q_b.weight with CPU
loading blk.52.attn_kv_a_mqa.weight with CPU
loading blk.52.attn_kv_a_norm.weight to cuda:0
loading blk.52.attn_kv_b.weight to cuda:0
loading blk.52.attn_output.weight with CPU
loading blk.52.ffn_gate_shexp.weight with CPU
loading blk.52.ffn_up_shexp.weight with CPU
loading blk.52.ffn_down_shexp.weight with CPU
loading blk.52.attn_norm.weight to cuda:0
loading blk.52.ffn_norm.weight to cuda:0
loading blk.53.attn_q_a.weight with CPU
loading blk.53.attn_q_a_norm.weight to cuda:0
loading blk.53.attn_q_b.weight with CPU
loading blk.53.attn_kv_a_mqa.weight with CPU
loading blk.53.attn_kv_a_norm.weight to cuda:0
loading blk.53.attn_kv_b.weight to cuda:0
loading blk.53.attn_output.weight with CPU
loading blk.53.ffn_gate_shexp.weight with CPU
loading blk.53.ffn_up_shexp.weight with CPU
loading blk.53.ffn_down_shexp.weight with CPU
loading blk.53.attn_norm.weight to cuda:0
loading blk.53.ffn_norm.weight to cuda:0
loading blk.54.attn_q_a.weight with CPU
loading blk.54.attn_q_a_norm.weight to cuda:0
loading blk.54.attn_q_b.weight with CPU
loading blk.54.attn_kv_a_mqa.weight with CPU
loading blk.54.attn_kv_a_norm.weight to cuda:0
loading blk.54.attn_kv_b.weight to cuda:0
loading blk.54.attn_output.weight with CPU
loading blk.54.ffn_gate_shexp.weight with CPU
loading blk.54.ffn_up_shexp.weight with CPU
loading blk.54.ffn_down_shexp.weight with CPU
loading blk.54.attn_norm.weight to cuda:0
loading blk.54.ffn_norm.weight to cuda:0
loading blk.55.attn_q_a.weight with CPU
loading blk.55.attn_q_a_norm.weight to cuda:0
loading blk.55.attn_q_b.weight with CPU
loading blk.55.attn_kv_a_mqa.weight with CPU
loading blk.55.attn_kv_a_norm.weight to cuda:0
loading blk.55.attn_kv_b.weight to cuda:0
loading blk.55.attn_output.weight with CPU
loading blk.55.ffn_gate_shexp.weight with CPU
loading blk.55.ffn_up_shexp.weight with CPU
loading blk.55.ffn_down_shexp.weight with CPU
loading blk.55.attn_norm.weight to cuda:0
loading blk.55.ffn_norm.weight to cuda:0
loading blk.56.attn_q_a.weight with CPU
loading blk.56.attn_q_a_norm.weight to cuda:0
loading blk.56.attn_q_b.weight with CPU
loading blk.56.attn_kv_a_mqa.weight with CPU
loading blk.56.attn_kv_a_norm.weight to cuda:0
loading blk.56.attn_kv_b.weight to cuda:0
loading blk.56.attn_output.weight with CPU
loading blk.56.ffn_gate_shexp.weight with CPU
loading blk.56.ffn_up_shexp.weight with CPU
loading blk.56.ffn_down_shexp.weight with CPU
loading blk.56.attn_norm.weight to cuda:0
loading blk.56.ffn_norm.weight to cuda:0
loading blk.57.attn_q_a.weight with CPU
loading blk.57.attn_q_a_norm.weight to cuda:0
loading blk.57.attn_q_b.weight with CPU
loading blk.57.attn_kv_a_mqa.weight with CPU
loading blk.57.attn_kv_a_norm.weight to cuda:0
loading blk.57.attn_kv_b.weight to cuda:0
loading blk.57.attn_output.weight with CPU
loading blk.57.ffn_gate_shexp.weight with CPU
loading blk.57.ffn_up_shexp.weight with CPU
loading blk.57.ffn_down_shexp.weight with CPU
loading blk.57.attn_norm.weight to cuda:0
loading blk.57.ffn_norm.weight to cuda:0
loading blk.58.attn_q_a.weight with CPU
loading blk.58.attn_q_a_norm.weight to cuda:0
loading blk.58.attn_q_b.weight with CPU
loading blk.58.attn_kv_a_mqa.weight with CPU
loading blk.58.attn_kv_a_norm.weight to cuda:0
loading blk.58.attn_kv_b.weight to cuda:0
loading blk.58.attn_output.weight with CPU
loading blk.58.ffn_gate_shexp.weight with CPU
loading blk.58.ffn_up_shexp.weight with CPU
loading blk.58.ffn_down_shexp.weight with CPU
loading blk.58.attn_norm.weight to cuda:0
loading blk.58.ffn_norm.weight to cuda:0
loading blk.59.attn_q_a.weight with CPU
loading blk.59.attn_q_a_norm.weight to cuda:0
loading blk.59.attn_q_b.weight with CPU
loading blk.59.attn_kv_a_mqa.weight with CPU
loading blk.59.attn_kv_a_norm.weight to cuda:0
loading blk.59.attn_kv_b.weight to cuda:0
loading blk.59.attn_output.weight with CPU
loading blk.59.ffn_gate_shexp.weight with CPU
loading blk.59.ffn_up_shexp.weight with CPU
loading blk.59.ffn_down_shexp.weight with CPU
loading blk.59.attn_norm.weight to cuda:0
loading blk.59.ffn_norm.weight to cuda:0
loading blk.60.attn_q_a.weight with CPU
loading blk.60.attn_q_a_norm.weight to cuda:0
loading blk.60.attn_q_b.weight with CPU
loading blk.60.attn_kv_a_mqa.weight with CPU
loading blk.60.attn_kv_a_norm.weight to cuda:0
loading blk.60.attn_kv_b.weight to cuda:0
loading blk.60.attn_output.weight with CPU
loading blk.60.ffn_gate_shexp.weight with CPU
loading blk.60.ffn_up_shexp.weight with CPU
loading blk.60.ffn_down_shexp.weight with CPU
loading blk.60.attn_norm.weight to cuda:0
loading blk.60.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output to cuda using CPUInfer
Chat: hi
/root/ktransformers/ktransformers/operators/RoPE.py:269: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:310.)
freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
corrupted size vs. prev_size
Aborted (core dumped)
(ktransformers) root@epyc:~/ktransformers# history
@Azure-Tang commented on GitHub (Mar 15, 2025):
您是在amd的什么设备上跑的？
加载比llamacpp慢合理，因为gpu部分需要做在线的dequant和quant
@gengchaogit commented on GitHub (Mar 15, 2025):
双路epyc 9334 cpu, 显卡是amd mi50, 对应的型号应该是gx906
@gengchaogit commented on GitHub (Mar 16, 2025):
loading blk.60.attn_kv_b.weight to cuda:1
loading blk.60.attn_norm.weight to cuda:1
loading blk.60.ffn_norm.weight to cuda:1
loading output_norm.weight to cuda:1
Chat: hi
/root/ktransformers/ktransformers/operators/linear.py:211: UserWarning: Attempting torted architecture! Overriding blas backend to hipblas (Triggered internally at /pytpp:310.)
out = x @ weight_dequant.T
Traceback (most recent call last):
File "/root/ktransformers/ktransformers/util/cuda_graph_runner.py", line 41, in capture
logits=model(inputs_embeds=inputs_embeds,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1702, in forward
logits = self.lm_head(hidden_states[:,-1:,:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/operators/linear.py", line 677, in forward
y = self.generate_linear.forward(x)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/operators/linear.py", line 210, in forward
weight_dequant = self._dequantize_weight(self.weight, self.weight_scale, bits=8)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/operators/linear.py", line 249, in _dequantize_weight
dequant_matrix = dequant_matrix * scales_broadcast
~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
RuntimeError: HIP error: operation not permitted when stream is capturing
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with
TORCH_USE_HIP_DSAto enable device-side assertions.During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/ktransformers/ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/root/ktransformers/ktransformers/util/utils.py", line 247, in prefill_and_generate
cuda_graph_runner.capture(model, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, torch_device, return_dict=False, use_cache=True)
File "/root/ktransformers/ktransformers/util/cuda_graph_runner.py", line 40, in capture
with torch.cuda.graph(self.graph, stream = capture_stream):
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 186, in exit
self.cuda_graph.capture_end()
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 84, in capture_end
super().capture_end()
RuntimeError: Capture must end on the same stream it began on.
terminate called after throwing an instance of 'c10::Error'
what(): HIP error: operation not permitted when stream is capturing
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with
TORCH_USE_HIP_DSAto enable device-side assertions.Exception raised from c10_hip_check_implementation at /pytorch/c10/hip/HIPException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7faced3ba968 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe0 (0x7faced363f78 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::hip::c10_hip_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7fada7e70952 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10_hip.so)
frame #3: at::cuda::CUDAGraph::~CUDAGraph() + 0xb9 (0x7fad46746b69 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_hip.so)
frame #4: + 0xc80392 (0x7fada8be9392 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0xc8052a (0x7fada8be952a in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x42c400 (0x7fada8395400 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x42ca41 (0x7fada8395a41 in /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: python() [0x55f0be]
frame #9: python() [0x55ef70]
frame #10: python() [0x541355]
frame #11: python() [0x53fd88]
frame #12: python() [0x53fdc4]
frame #13: python() [0x53fdc4]
frame #14: python() [0x53fdc4]
frame #15: python() [0x53fdc4]
frame #16: python() [0x53fdc4]
frame #17: python() [0x4f770b]
frame #19: python() [0x5fcc84]
frame #23: + 0x29d90 (0x7fadaa426d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #24: __libc_start_main + 0x80 (0x7fadaa426e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #25: python() [0x5bbf93]
Aborted (core dumped)
(ktransformers) root@epyc:~/ktransformers#
@gengchaogit commented on GitHub (Mar 16, 2025):
更新代码后遇到的新问题，双显卡占用58% 47%，对话chat crash。
python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/windows/Users/chao/.lmstudio/models/deepseek671b/deepseek671b-q4/DeepSeek-R1-Q4_K_M-00001-of-00009.gguf --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --cpu_infer 65
@gengchaogit commented on GitHub (Mar 17, 2025):
感谢帮助，我今晚成功跑完一次了，虽然速度慢点，但是好歹算跑起来了。、
export TORCH_BLAS_PREFER_HIPBLASLT=0
我通过这个命令把HIPBLASLT给禁用了，然后那个warning提示就不出了。
下面是amd mi50*2的测试成绩
Chat: hi
Okay, the user is asking for help with their homework. They mentioned "hi" but didn't specify the subject or the exact problem. I need to prompt them for more details. Maybe they're struggling with math, science, or literature? I should ask politely to get the necessary information to assist effectively. Keep the response friendly and encouraging so they feel comfortable sharing more.
Alright, I need to respond in a helpful way. Let's start by acknowledging their request and then ask for specifics. Maybe say something like, "Hi! I'd be happy to help with your homework. Could you tell me more about what you're working on? Like the subject, the problem, or any questions you have?" That way they can provide the details needed. Keep it positive to encourage engagement.
prompt eval count: 6 token(s)
prompt eval duration: 10.09068250656128s
prompt eval rate: 0.594607946102616 tokens/s
eval count: 162 token(s)
eval duration: 481.8623569011688s
eval rate: 0.3361955912925288 tokens/s
@Azure-Tang commented on GitHub (Mar 18, 2025):
mi50用不了hipblas还是影响比较大
@fxzjshm commented on GitHub (Mar 20, 2025):
hipblaslt 目前只适配了很新的 MI 系列专业卡; 另外比如 gfx1100 虽然支持但性能还不如 hipblas (hipblaslt 3.27 tokens/s vs. hipblas 4.49 tokens/s)
hipblaslt has only tuned for recent MI series only; and for gfx1100 although hipblaslt supports it, performance is worse than hipblas (hipblaslt 3.27 tokens/s vs. hipblas 4.49 tokens/s)
@gengchaogit commented on GitHub (Apr 13, 2025):
您的意思是支持hipblaslt也才3.27 token/s? 那目前amd的卡还是不太行
@fxzjshm commented on GitHub (Apr 13, 2025):
看是和什么卡对比. 目前其性能与 2080 等不支持 Marlin 的卡性能类似, 要与 30 和 40 系比较需要先让 AMD 的卡支持 Marlin 才能公平比较.
@gengchaogit commented on GitHub (Apr 14, 2025):
好吧，我们这边3070的卡单路671b q4也能跑到11t