[GH-ISSUE #1285] [Bug] Getting inference context from sched_client. sched_rpc started with PID: 1011 后无输出 #872

New issue

Closed

opened 2026-10-29 14:45:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:45:37 +00:00

Owner

Copy link

Originally created by @venssy on GitHub (May 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1285

检查清单

• 1. 我已经搜索过相关问题，但未能获得预期的帮助
• 2. 该问题在最新版本中尚未修复
• 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
• 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
• 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

Qwen3-30B-A3B 卡在 Getting inference context from sched_client. sched_rpc started with PID: 1011 没有往下运行，api调用无响应

2025-05-12 14:59:25,855 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
found flash_attn
set start method
Connected to server at tcp://localhost:53335
2025-05-12 14:59:50,143 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
found flash_attn
start method already set to spawn
Connected to server at tcp://localhost:53335
Injecting model as ktransformers.operators.models . KQwen2MoeModel
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention
Injecting model.layers.0.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.0.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.0.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2
Injecting model.layers.0.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.0.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.0.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
......
Injecting model.layers.47 as default
Injecting model.layers.47.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention
Injecting model.layers.47.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.47.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.47.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2
Injecting model.layers.47.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.47.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.47.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.rotary_emb as default
Injecting cache as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_q_norm.weight to cuda
loading blk.0.attn_k_norm.weight to cuda
loading blk.0.attn_norm.weight to cuda
loading blk.0.ffn_norm.weight to cuda
loading blk.1.attn_q_norm.weight to cuda
loading blk.1.attn_k_norm.weight to cuda
loading blk.1.attn_norm.weight to cuda
loading blk.1.ffn_norm.weight to cuda
loading blk.2.attn_q_norm.weight to cuda
loading blk.2.attn_k_norm.weight to cuda
loading blk.2.attn_norm.weight to cuda
loading blk.2.ffn_norm.weight to cuda
loading blk.3.attn_q_norm.weight to cuda
loading blk.3.attn_k_norm.weight to cuda
loading blk.3.attn_norm.weight to cuda
loading blk.3.ffn_norm.weight to cuda
loading blk.4.attn_q_norm.weight to cuda
loading blk.4.attn_k_norm.weight to cuda
loading blk.4.attn_norm.weight to cuda
loading blk.4.ffn_norm.weight to cuda
loading blk.5.attn_q_norm.weight to cuda
loading blk.5.attn_k_norm.weight to cuda
loading blk.5.attn_norm.weight to cuda
loading blk.5.ffn_norm.weight to cuda
loading blk.6.attn_q_norm.weight to cuda
loading blk.6.attn_k_norm.weight to cuda
loading blk.6.attn_norm.weight to cuda
loading blk.6.ffn_norm.weight to cuda
loading blk.7.attn_q_norm.weight to cuda
loading blk.7.attn_k_norm.weight to cuda
loading blk.7.attn_norm.weight to cuda
loading blk.7.ffn_norm.weight to cuda
loading blk.8.attn_q_norm.weight to cuda
loading blk.8.attn_k_norm.weight to cuda
loading blk.8.attn_norm.weight to cuda
loading blk.8.ffn_norm.weight to cuda
loading blk.9.attn_q_norm.weight to cuda
loading blk.9.attn_k_norm.weight to cuda
loading blk.9.attn_norm.weight to cuda
loading blk.9.ffn_norm.weight to cuda
loading blk.10.attn_q_norm.weight to cuda
loading blk.10.attn_k_norm.weight to cuda
loading blk.10.attn_norm.weight to cuda
loading blk.10.ffn_norm.weight to cuda
loading blk.11.attn_q_norm.weight to cuda
loading blk.11.attn_k_norm.weight to cuda
loading blk.11.attn_norm.weight to cuda
loading blk.11.ffn_norm.weight to cuda
loading blk.12.attn_q_norm.weight to cuda
loading blk.12.attn_k_norm.weight to cuda
loading blk.12.attn_norm.weight to cuda
loading blk.12.ffn_norm.weight to cuda
loading blk.13.attn_q_norm.weight to cuda
loading blk.13.attn_k_norm.weight to cuda
loading blk.13.attn_norm.weight to cuda
loading blk.13.ffn_norm.weight to cuda
loading blk.14.attn_q_norm.weight to cuda
loading blk.14.attn_k_norm.weight to cuda
loading blk.14.attn_norm.weight to cuda
loading blk.14.ffn_norm.weight to cuda
loading blk.15.attn_q_norm.weight to cuda
loading blk.15.attn_k_norm.weight to cuda
loading blk.15.attn_norm.weight to cuda
loading blk.15.ffn_norm.weight to cuda
loading blk.16.attn_q_norm.weight to cuda
loading blk.16.attn_k_norm.weight to cuda
loading blk.16.attn_norm.weight to cuda
loading blk.16.ffn_norm.weight to cuda
loading blk.17.attn_q_norm.weight to cuda
loading blk.17.attn_k_norm.weight to cuda
loading blk.17.attn_norm.weight to cuda
loading blk.17.ffn_norm.weight to cuda
loading blk.18.attn_q_norm.weight to cuda
loading blk.18.attn_k_norm.weight to cuda
loading blk.18.attn_norm.weight to cuda
loading blk.18.ffn_norm.weight to cuda
loading blk.19.attn_q_norm.weight to cuda
loading blk.19.attn_k_norm.weight to cuda
loading blk.19.attn_norm.weight to cuda
loading blk.19.ffn_norm.weight to cuda
loading blk.20.attn_q_norm.weight to cuda
loading blk.20.attn_k_norm.weight to cuda
loading blk.20.attn_norm.weight to cuda
loading blk.20.ffn_norm.weight to cuda
loading blk.21.attn_q_norm.weight to cuda
loading blk.21.attn_k_norm.weight to cuda
loading blk.21.attn_norm.weight to cuda
loading blk.21.ffn_norm.weight to cuda
loading blk.22.attn_q_norm.weight to cuda
loading blk.22.attn_k_norm.weight to cuda
loading blk.22.attn_norm.weight to cuda
loading blk.22.ffn_norm.weight to cuda
loading blk.23.attn_q_norm.weight to cuda
loading blk.23.attn_k_norm.weight to cuda
loading blk.23.attn_norm.weight to cuda
loading blk.23.ffn_norm.weight to cuda
loading blk.24.attn_q_norm.weight to cuda
loading blk.24.attn_k_norm.weight to cuda
loading blk.24.attn_norm.weight to cuda
loading blk.24.ffn_norm.weight to cuda
loading blk.25.attn_q_norm.weight to cuda
loading blk.25.attn_k_norm.weight to cuda
loading blk.25.attn_norm.weight to cuda
loading blk.25.ffn_norm.weight to cuda
loading blk.26.attn_q_norm.weight to cuda
loading blk.26.attn_k_norm.weight to cuda
loading blk.26.attn_norm.weight to cuda
loading blk.26.ffn_norm.weight to cuda
loading blk.27.attn_q_norm.weight to cuda
loading blk.27.attn_k_norm.weight to cuda
loading blk.27.attn_norm.weight to cuda
loading blk.27.ffn_norm.weight to cuda
loading blk.28.attn_q_norm.weight to cuda
loading blk.28.attn_k_norm.weight to cuda
loading blk.28.attn_norm.weight to cuda
loading blk.28.ffn_norm.weight to cuda
loading blk.29.attn_q_norm.weight to cuda
loading blk.29.attn_k_norm.weight to cuda
loading blk.29.attn_norm.weight to cuda
loading blk.29.ffn_norm.weight to cuda
loading blk.30.attn_q_norm.weight to cuda
loading blk.30.attn_k_norm.weight to cuda
loading blk.30.attn_norm.weight to cuda
loading blk.30.ffn_norm.weight to cuda
loading blk.31.attn_q_norm.weight to cuda
loading blk.31.attn_k_norm.weight to cuda
loading blk.31.attn_norm.weight to cuda
loading blk.31.ffn_norm.weight to cuda
loading blk.32.attn_q_norm.weight to cuda
loading blk.32.attn_k_norm.weight to cuda
loading blk.32.attn_norm.weight to cuda
loading blk.32.ffn_norm.weight to cuda
loading blk.33.attn_q_norm.weight to cuda
loading blk.33.attn_k_norm.weight to cuda
loading blk.33.attn_norm.weight to cuda
loading blk.33.ffn_norm.weight to cuda
loading blk.34.attn_q_norm.weight to cuda
loading blk.34.attn_k_norm.weight to cuda
loading blk.34.attn_norm.weight to cuda
loading blk.34.ffn_norm.weight to cuda
loading blk.35.attn_q_norm.weight to cuda
loading blk.35.attn_k_norm.weight to cuda
loading blk.35.attn_norm.weight to cuda
loading blk.35.ffn_norm.weight to cuda
loading blk.36.attn_q_norm.weight to cuda
loading blk.36.attn_k_norm.weight to cuda
loading blk.36.attn_norm.weight to cuda
loading blk.36.ffn_norm.weight to cuda
loading blk.37.attn_q_norm.weight to cuda
loading blk.37.attn_k_norm.weight to cuda
loading blk.37.attn_norm.weight to cuda
loading blk.37.ffn_norm.weight to cuda
loading blk.38.attn_q_norm.weight to cuda
loading blk.38.attn_k_norm.weight to cuda
loading blk.38.attn_norm.weight to cuda
loading blk.38.ffn_norm.weight to cuda
loading blk.39.attn_q_norm.weight to cuda
loading blk.39.attn_k_norm.weight to cuda
loading blk.39.attn_norm.weight to cuda
loading blk.39.ffn_norm.weight to cuda
loading blk.40.attn_q_norm.weight to cuda
loading blk.40.attn_k_norm.weight to cuda
loading blk.40.attn_norm.weight to cuda
loading blk.40.ffn_norm.weight to cuda
loading blk.41.attn_q_norm.weight to cuda
loading blk.41.attn_k_norm.weight to cuda
loading blk.41.attn_norm.weight to cuda
loading blk.41.ffn_norm.weight to cuda
loading blk.42.attn_q_norm.weight to cuda
loading blk.42.attn_k_norm.weight to cuda
loading blk.42.attn_norm.weight to cuda
loading blk.42.ffn_norm.weight to cuda
loading blk.43.attn_q_norm.weight to cuda
loading blk.43.attn_k_norm.weight to cuda
loading blk.43.attn_norm.weight to cuda
loading blk.43.ffn_norm.weight to cuda
loading blk.44.attn_q_norm.weight to cuda
loading blk.44.attn_k_norm.weight to cuda
loading blk.44.attn_norm.weight to cuda
loading blk.44.ffn_norm.weight to cuda
loading blk.45.attn_q_norm.weight to cuda
loading blk.45.attn_k_norm.weight to cuda
loading blk.45.attn_norm.weight to cuda
loading blk.45.ffn_norm.weight to cuda
loading blk.46.attn_q_norm.weight to cuda
loading blk.46.attn_k_norm.weight to cuda
loading blk.46.attn_norm.weight to cuda
loading blk.46.ffn_norm.weight to cuda
loading blk.47.attn_q_norm.weight to cuda
loading blk.47.attn_k_norm.weight to cuda
loading blk.47.attn_norm.weight to cuda
loading blk.47.ffn_norm.weight to cuda
loading output_norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 1011

复现步骤

python ./ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --model_name Qwen3-30B-A3B --model_path Qwen/Qwen3-30B-A3B --gguf_path /data/qwen3/Qwen3-30B-A3B-Q4_K_M.gguf --port 10002 --max_new_tokens 1024 --cache_lens 32768 --chunk_size 256 --max_batch_size 4 --backend_type balance_serve

环境信息

docker ktransformer:0.3
rtx3080
cpu i7-13700

Originally created by @venssy on GitHub (May 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1285 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 Qwen3-30B-A3B 卡在 `Getting inference context from sched_client. sched_rpc started with PID: 1011` 没有往下运行，api调用无响应 ``` 2025-05-12 14:59:25,855 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend found flashinfer found flash_attn set start method Connected to server at tcp://localhost:53335 2025-05-12 14:59:50,143 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend found flashinfer found flash_attn start method already set to spawn Connected to server at tcp://localhost:53335 Injecting model as ktransformers.operators.models . KQwen2MoeModel Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention Injecting model.layers.0.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.0.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.0.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2 Injecting model.layers.0.mlp.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2 Injecting model.layers.0.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.0.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm ...... Injecting model.layers.47 as default Injecting model.layers.47.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention Injecting model.layers.47.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.47.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.47.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding Injecting model.layers.47.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2 Injecting model.layers.47.mlp.gate as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.47.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2 Injecting model.layers.47.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.layers.47.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm Injecting model.rotary_emb as default Injecting cache as default Injecting lm_head as ktransformers.operators.linear . KTransformersLinear loading token_embd.weight to cpu loading token_embd.weight with CPU loading blk.0.attn_q_norm.weight to cuda loading blk.0.attn_k_norm.weight to cuda loading blk.0.attn_norm.weight to cuda loading blk.0.ffn_norm.weight to cuda loading blk.1.attn_q_norm.weight to cuda loading blk.1.attn_k_norm.weight to cuda loading blk.1.attn_norm.weight to cuda loading blk.1.ffn_norm.weight to cuda loading blk.2.attn_q_norm.weight to cuda loading blk.2.attn_k_norm.weight to cuda loading blk.2.attn_norm.weight to cuda loading blk.2.ffn_norm.weight to cuda loading blk.3.attn_q_norm.weight to cuda loading blk.3.attn_k_norm.weight to cuda loading blk.3.attn_norm.weight to cuda loading blk.3.ffn_norm.weight to cuda loading blk.4.attn_q_norm.weight to cuda loading blk.4.attn_k_norm.weight to cuda loading blk.4.attn_norm.weight to cuda loading blk.4.ffn_norm.weight to cuda loading blk.5.attn_q_norm.weight to cuda loading blk.5.attn_k_norm.weight to cuda loading blk.5.attn_norm.weight to cuda loading blk.5.ffn_norm.weight to cuda loading blk.6.attn_q_norm.weight to cuda loading blk.6.attn_k_norm.weight to cuda loading blk.6.attn_norm.weight to cuda loading blk.6.ffn_norm.weight to cuda loading blk.7.attn_q_norm.weight to cuda loading blk.7.attn_k_norm.weight to cuda loading blk.7.attn_norm.weight to cuda loading blk.7.ffn_norm.weight to cuda loading blk.8.attn_q_norm.weight to cuda loading blk.8.attn_k_norm.weight to cuda loading blk.8.attn_norm.weight to cuda loading blk.8.ffn_norm.weight to cuda loading blk.9.attn_q_norm.weight to cuda loading blk.9.attn_k_norm.weight to cuda loading blk.9.attn_norm.weight to cuda loading blk.9.ffn_norm.weight to cuda loading blk.10.attn_q_norm.weight to cuda loading blk.10.attn_k_norm.weight to cuda loading blk.10.attn_norm.weight to cuda loading blk.10.ffn_norm.weight to cuda loading blk.11.attn_q_norm.weight to cuda loading blk.11.attn_k_norm.weight to cuda loading blk.11.attn_norm.weight to cuda loading blk.11.ffn_norm.weight to cuda loading blk.12.attn_q_norm.weight to cuda loading blk.12.attn_k_norm.weight to cuda loading blk.12.attn_norm.weight to cuda loading blk.12.ffn_norm.weight to cuda loading blk.13.attn_q_norm.weight to cuda loading blk.13.attn_k_norm.weight to cuda loading blk.13.attn_norm.weight to cuda loading blk.13.ffn_norm.weight to cuda loading blk.14.attn_q_norm.weight to cuda loading blk.14.attn_k_norm.weight to cuda loading blk.14.attn_norm.weight to cuda loading blk.14.ffn_norm.weight to cuda loading blk.15.attn_q_norm.weight to cuda loading blk.15.attn_k_norm.weight to cuda loading blk.15.attn_norm.weight to cuda loading blk.15.ffn_norm.weight to cuda loading blk.16.attn_q_norm.weight to cuda loading blk.16.attn_k_norm.weight to cuda loading blk.16.attn_norm.weight to cuda loading blk.16.ffn_norm.weight to cuda loading blk.17.attn_q_norm.weight to cuda loading blk.17.attn_k_norm.weight to cuda loading blk.17.attn_norm.weight to cuda loading blk.17.ffn_norm.weight to cuda loading blk.18.attn_q_norm.weight to cuda loading blk.18.attn_k_norm.weight to cuda loading blk.18.attn_norm.weight to cuda loading blk.18.ffn_norm.weight to cuda loading blk.19.attn_q_norm.weight to cuda loading blk.19.attn_k_norm.weight to cuda loading blk.19.attn_norm.weight to cuda loading blk.19.ffn_norm.weight to cuda loading blk.20.attn_q_norm.weight to cuda loading blk.20.attn_k_norm.weight to cuda loading blk.20.attn_norm.weight to cuda loading blk.20.ffn_norm.weight to cuda loading blk.21.attn_q_norm.weight to cuda loading blk.21.attn_k_norm.weight to cuda loading blk.21.attn_norm.weight to cuda loading blk.21.ffn_norm.weight to cuda loading blk.22.attn_q_norm.weight to cuda loading blk.22.attn_k_norm.weight to cuda loading blk.22.attn_norm.weight to cuda loading blk.22.ffn_norm.weight to cuda loading blk.23.attn_q_norm.weight to cuda loading blk.23.attn_k_norm.weight to cuda loading blk.23.attn_norm.weight to cuda loading blk.23.ffn_norm.weight to cuda loading blk.24.attn_q_norm.weight to cuda loading blk.24.attn_k_norm.weight to cuda loading blk.24.attn_norm.weight to cuda loading blk.24.ffn_norm.weight to cuda loading blk.25.attn_q_norm.weight to cuda loading blk.25.attn_k_norm.weight to cuda loading blk.25.attn_norm.weight to cuda loading blk.25.ffn_norm.weight to cuda loading blk.26.attn_q_norm.weight to cuda loading blk.26.attn_k_norm.weight to cuda loading blk.26.attn_norm.weight to cuda loading blk.26.ffn_norm.weight to cuda loading blk.27.attn_q_norm.weight to cuda loading blk.27.attn_k_norm.weight to cuda loading blk.27.attn_norm.weight to cuda loading blk.27.ffn_norm.weight to cuda loading blk.28.attn_q_norm.weight to cuda loading blk.28.attn_k_norm.weight to cuda loading blk.28.attn_norm.weight to cuda loading blk.28.ffn_norm.weight to cuda loading blk.29.attn_q_norm.weight to cuda loading blk.29.attn_k_norm.weight to cuda loading blk.29.attn_norm.weight to cuda loading blk.29.ffn_norm.weight to cuda loading blk.30.attn_q_norm.weight to cuda loading blk.30.attn_k_norm.weight to cuda loading blk.30.attn_norm.weight to cuda loading blk.30.ffn_norm.weight to cuda loading blk.31.attn_q_norm.weight to cuda loading blk.31.attn_k_norm.weight to cuda loading blk.31.attn_norm.weight to cuda loading blk.31.ffn_norm.weight to cuda loading blk.32.attn_q_norm.weight to cuda loading blk.32.attn_k_norm.weight to cuda loading blk.32.attn_norm.weight to cuda loading blk.32.ffn_norm.weight to cuda loading blk.33.attn_q_norm.weight to cuda loading blk.33.attn_k_norm.weight to cuda loading blk.33.attn_norm.weight to cuda loading blk.33.ffn_norm.weight to cuda loading blk.34.attn_q_norm.weight to cuda loading blk.34.attn_k_norm.weight to cuda loading blk.34.attn_norm.weight to cuda loading blk.34.ffn_norm.weight to cuda loading blk.35.attn_q_norm.weight to cuda loading blk.35.attn_k_norm.weight to cuda loading blk.35.attn_norm.weight to cuda loading blk.35.ffn_norm.weight to cuda loading blk.36.attn_q_norm.weight to cuda loading blk.36.attn_k_norm.weight to cuda loading blk.36.attn_norm.weight to cuda loading blk.36.ffn_norm.weight to cuda loading blk.37.attn_q_norm.weight to cuda loading blk.37.attn_k_norm.weight to cuda loading blk.37.attn_norm.weight to cuda loading blk.37.ffn_norm.weight to cuda loading blk.38.attn_q_norm.weight to cuda loading blk.38.attn_k_norm.weight to cuda loading blk.38.attn_norm.weight to cuda loading blk.38.ffn_norm.weight to cuda loading blk.39.attn_q_norm.weight to cuda loading blk.39.attn_k_norm.weight to cuda loading blk.39.attn_norm.weight to cuda loading blk.39.ffn_norm.weight to cuda loading blk.40.attn_q_norm.weight to cuda loading blk.40.attn_k_norm.weight to cuda loading blk.40.attn_norm.weight to cuda loading blk.40.ffn_norm.weight to cuda loading blk.41.attn_q_norm.weight to cuda loading blk.41.attn_k_norm.weight to cuda loading blk.41.attn_norm.weight to cuda loading blk.41.ffn_norm.weight to cuda loading blk.42.attn_q_norm.weight to cuda loading blk.42.attn_k_norm.weight to cuda loading blk.42.attn_norm.weight to cuda loading blk.42.ffn_norm.weight to cuda loading blk.43.attn_q_norm.weight to cuda loading blk.43.attn_k_norm.weight to cuda loading blk.43.attn_norm.weight to cuda loading blk.43.ffn_norm.weight to cuda loading blk.44.attn_q_norm.weight to cuda loading blk.44.attn_k_norm.weight to cuda loading blk.44.attn_norm.weight to cuda loading blk.44.ffn_norm.weight to cuda loading blk.45.attn_q_norm.weight to cuda loading blk.45.attn_k_norm.weight to cuda loading blk.45.attn_norm.weight to cuda loading blk.45.ffn_norm.weight to cuda loading blk.46.attn_q_norm.weight to cuda loading blk.46.attn_k_norm.weight to cuda loading blk.46.attn_norm.weight to cuda loading blk.46.ffn_norm.weight to cuda loading blk.47.attn_q_norm.weight to cuda loading blk.47.attn_k_norm.weight to cuda loading blk.47.attn_norm.weight to cuda loading blk.47.ffn_norm.weight to cuda loading output_norm.weight to cuda Getting inference context from sched_client. sched_rpc started with PID: 1011 ``` ### 复现步骤 `python ./ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --model_name Qwen3-30B-A3B --model_path Qwen/Qwen3-30B-A3B --gguf_path /data/qwen3/Qwen3-30B-A3B-Q4_K_M.gguf --port 10002 --max_new_tokens 1024 --cache_lens 32768 --chunk_size 256 --max_batch_size 4 --backend_type balance_serve` ### 环境信息 docker ktransformer:0.3 rtx3080 cpu i7-13700

bsakaguchi closed this issue 2026-10-29 14:45:37 +00:00

bsakaguchi commented 2026-10-29 14:45:38 +00:00

Author

Owner

Copy link

@venssy commented on GitHub (May 12, 2025):

参考 #1254

<!-- gh-comment-id:2872978748 --> @venssy commented on GitHub (May 12, 2025): 参考 #1254

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#872

No description provided.