kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1336] [Bug] ktransformers Docker container hangs and never starts HTTP server on port 10002 #900

New issue
Open
opened 2026-10-29 14:45:53 +00:00 by bsakaguchi · 10 comments
bsakaguchi commented 2026-10-29 14:45:53 +00:00
Owner
Copy link

Originally created by @mtcl on GitHub (May 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1336

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Title
ktransformers Docker container hangs and never starts HTTP server on port 10002

It just says below two lines and then stays there, i cannot do any request response at 10002

Getting inference context from sched_client.
sched_rpc started with PID: 381

Reproduction

📋 Steps to reproduce

  1. Stop any existing container:
   docker stop ktransformers

  1. Launch fresh container:

    docker run --rm --gpus all \
  -v /media/mukul/extra/models:/models \
  -p 10002:10002 \
  --name ktransformers \
  -itd approachingai/ktransformers:v0.3.1-AVX512

  2. Verify container is running:

    docker ps

  3. attach a shell via bash :

    docker exec -it ktransformers /bin/bash

  4. Launch server manually:

    PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  python3 ktransformers/server/main.py \
  --gguf_path /models/KVCache-ai/DeepSeek-V3-3024-GGML-FP8-Hybrid/Q4_K_M_FP8 \
  --model_path deepseek-ai/DeepSeek-V3-0324 \
  --model_name KVCache-ai/DeepSeek-V3-0324-q4km-fp8 \
  --cpu_infer 57 \
  --max_new_tokens 16384 \
  --cache_lens 32768 \
  --cache_q4 true \
  --temperature 0.6 \
  --top_p 0.95 \
  --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts-serve.yaml \
  --use_cuda_graph \
  --backend_type balance_serve \
  --host 0.0.0.0 \
  --port 10002

  5. Observe that it logs all the “Injecting model…” messages, but never prints anything like INFO: Uvicorn running on ...:10002, and curl localhost:10002 never responds.

🛑 Observed behavior

docker stop ktransformers
docker run --rm --gpus all
-v /media/mukul/extra/models:/models
-p 10002:10002
--name ktransformers
-itd approachingai/ktransformers:v0.3.1-AVX512

docker exec -it ktransformers /bin/bash
Error response from daemon: No such container: ktransformers
bda92b2265cd3a33a72f96cb95adeafbf009729f9ae4d1fdb19b7a0d787e21ec
root@bda92b2265cd:/workspace/ktransformers# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py
--gguf_path /models/KVCache-ai/DeepSeek-V3-3024-GGML-FP8-Hybrid/Q4_K_M_FP8
--model_path deepseek-ai/DeepSeek-V3-0324
--model_name KVCache-ai/DeepSeek-V3-0324-q4km-fp8
--cpu_infer 57
--max_new_tokens 16384
--cache_lens 32768
--cache_q4 true
--temperature 0.6
--top_p 0.95
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts-serve.yaml
--use_cuda_graph
--backend_type balance_serve
--host 0.0.0.0
--port 10002
2025-05-25 02:05:25,657 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 15.8MB/s]
configuration_deepseek.py: 100%|████████████████████████████████████████████████████████████████████████████| 9.90k/9.90k [00:00<00:00, 21.6MB/s]
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V3-0324:

  • configuration_deepseek.py
    . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
    found flash_attn
    start method already set to fork
    tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████| 3.88k/3.88k [00:00<00:00, 21.0MB/s]
    tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 7.85M/7.85M [00:00<00:00, 19.0MB/s]
    Connected to server at tcp://localhost:54055
    2025-05-25 02:05:33,896 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
    found flashinfer
    found flash_attn
    start method already set to fork
    Connected to server at tcp://localhost:54055
    Injecting model as ktransformers.operators.models . KDeepseekV2Model
    Injecting model.embed_tokens as default
    Injecting model.layers as default
    Injecting model.layers.0 as default
    Injecting model.layers.0.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.0.self_attn.kv_b_proj as default
    Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.0.mlp as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.0.mlp.act_fn as default
    Injecting model.layers.0.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.0.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.1 as default
    Injecting model.layers.1.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.1.self_attn.kv_b_proj as default
    Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.1.mlp as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.1.mlp.act_fn as default
    Injecting model.layers.1.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.1.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.2 as default
    Injecting model.layers.2.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.2.self_attn.kv_b_proj as default
    Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.2.mlp as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.2.mlp.act_fn as default
    Injecting model.layers.2.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.2.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.3 as default
    Injecting model.layers.3.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.3.self_attn.kv_b_proj as default
    Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2
    Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
    Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
    Injecting model.layers.3.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.3.mlp.shared_experts.act_fn as default
    Injecting model.layers.3.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.3.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.4 as default
    Injecting model.layers.4.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.4.self_attn.kv_b_proj as default
    Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2
    Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
    Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
    Injecting model.layers.4.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.4.mlp.shared_experts.act_fn as default
    Injecting model.layers.4.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.4.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    .
    .
    .
    .
    Injecting model.layers.60 as default
    Injecting model.layers.60.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn
    Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.60.self_attn.kv_b_proj as default
    Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
    Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2
    Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
    Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
    Injecting model.layers.60.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP
    Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
    Injecting model.layers.60.mlp.shared_experts.act_fn as default
    Injecting model.layers.60.input_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.layers.60.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm
    Injecting model.norm as ktransformers.operators.layernorm . RMSNorm
    Injecting cache as default
    Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
    loading model.embed_tokens.weight to cpu
    loading model.layers.0.self_attn.q_a_layernorm.weight to cuda
    loading model.layers.0.self_attn.kv_a_layernorm.weight to cuda
    loading model.layers.0.self_attn.kv_b_proj.weight to cuda:0
    loading model.layers.0.input_layernorm.weight to cuda
    loading model.layers.0.post_attention_layernorm.weight to cuda
    loading model.layers.1.self_attn.q_a_layernorm.weight to cuda
    loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda
    loading model.layers.1.self_attn.kv_b_proj.weight to cuda:0
    .
    .
    .
    .
    .

loading model.layers.54.post_attention_layernorm.weight to cuda
loading model.layers.55.self_attn.q_a_layernorm.weight to cuda
loading model.layers.55.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.55.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.55.input_layernorm.weight to cuda
loading model.layers.55.post_attention_layernorm.weight to cuda
loading model.layers.56.self_attn.q_a_layernorm.weight to cuda
loading model.layers.56.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.56.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.56.input_layernorm.weight to cuda
loading model.layers.56.post_attention_layernorm.weight to cuda
loading model.layers.57.self_attn.q_a_layernorm.weight to cuda
loading model.layers.57.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.57.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.57.input_layernorm.weight to cuda
loading model.layers.57.post_attention_layernorm.weight to cuda
loading model.layers.58.self_attn.q_a_layernorm.weight to cuda
loading model.layers.58.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.58.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.58.input_layernorm.weight to cuda
loading model.layers.58.post_attention_layernorm.weight to cuda
loading model.layers.59.self_attn.q_a_layernorm.weight to cuda
loading model.layers.59.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.59.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.59.input_layernorm.weight to cuda
loading model.layers.59.post_attention_layernorm.weight to cuda
loading model.layers.60.self_attn.q_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.60.input_layernorm.weight to cuda
loading model.layers.60.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 381

Environment

##Environment

  • Host OS: Ubuntu 22.04 LTS
  • Docker Engine: e.g. 24.0.2
  • ktransformers image: approachingai/ktransformers:v0.3.1-AVX512
  • GPU: NVIDIA RTX 4090
  • CUDA version: from nvidia-smi
    nvcc --version
    Cuda compilation tools, release 12.9, V12.9.41
    Build cuda_12.9.r12.9/compiler.35813241_0
    nvidia-smi
    Driver Version: 575.51.03 CUDA Version: 12.9
  • Python: 3.11.x inside container
Originally created by @mtcl on GitHub (May 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1336 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug **Title** ktransformers Docker container hangs and never starts HTTP server on port 10002 It just says below two lines and then stays there, i cannot do any request response at 10002 Getting inference context from sched_client. sched_rpc started with PID: 381 ### Reproduction ## 📋 Steps to reproduce 1. Stop any existing container: ```bash docker stop ktransformers ``` 2. Launch fresh container: ```bash docker run --rm --gpus all \ -v /media/mukul/extra/models:/models \ -p 10002:10002 \ --name ktransformers \ -itd approachingai/ktransformers:v0.3.1-AVX512 ``` 3. Verify container is running: ```bash docker ps ``` 4. attach a shell via bash : ```bash docker exec -it ktransformers /bin/bash ``` 5. Launch server manually: ```bash PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ python3 ktransformers/server/main.py \ --gguf_path /models/KVCache-ai/DeepSeek-V3-3024-GGML-FP8-Hybrid/Q4_K_M_FP8 \ --model_path deepseek-ai/DeepSeek-V3-0324 \ --model_name KVCache-ai/DeepSeek-V3-0324-q4km-fp8 \ --cpu_infer 57 \ --max_new_tokens 16384 \ --cache_lens 32768 \ --cache_q4 true \ --temperature 0.6 \ --top_p 0.95 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts-serve.yaml \ --use_cuda_graph \ --backend_type balance_serve \ --host 0.0.0.0 \ --port 10002 ``` 7. Observe that it logs all the “Injecting model…” messages, but never prints anything like `INFO: Uvicorn running on ...:10002`, and `curl localhost:10002` never responds. --- ## 🛑 Observed behavior docker stop ktransformers docker run --rm --gpus all \ -v /media/mukul/extra/models:/models \ -p 10002:10002 \ --name ktransformers \ -itd approachingai/ktransformers:v0.3.1-AVX512 docker exec -it ktransformers /bin/bash Error response from daemon: No such container: ktransformers bda92b2265cd3a33a72f96cb95adeafbf009729f9ae4d1fdb19b7a0d787e21ec root@bda92b2265cd:/workspace/ktransformers# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py \ --gguf_path /models/KVCache-ai/DeepSeek-V3-3024-GGML-FP8-Hybrid/Q4_K_M_FP8 \ --model_path deepseek-ai/DeepSeek-V3-0324 \ --model_name KVCache-ai/DeepSeek-V3-0324-q4km-fp8 \ --cpu_infer 57 \ --max_new_tokens 16384 \ --cache_lens 32768 \ --cache_q4 true \ --temperature 0.6 \ --top_p 0.95 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts-serve.yaml \ --use_cuda_graph \ --backend_type balance_serve \ --host 0.0.0.0 \ --port 10002 2025-05-25 02:05:25,657 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend found flashinfer config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 15.8MB/s] configuration_deepseek.py: 100%|████████████████████████████████████████████████████████████████████████████| 9.90k/9.90k [00:00<00:00, 21.6MB/s] A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V3-0324: - configuration_deepseek.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. found flash_attn start method already set to fork tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████| 3.88k/3.88k [00:00<00:00, 21.0MB/s] tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 7.85M/7.85M [00:00<00:00, 19.0MB/s] Connected to server at tcp://localhost:54055 2025-05-25 02:05:33,896 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend found flashinfer found flash_attn start method already set to fork Connected to server at tcp://localhost:54055 Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.0.self_attn.kv_b_proj as default Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.0.mlp as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.mlp.act_fn as default Injecting model.layers.0.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.0.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.1 as default Injecting model.layers.1.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.1.self_attn.kv_b_proj as default Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.1.mlp as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.1.mlp.act_fn as default Injecting model.layers.1.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.1.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.2 as default Injecting model.layers.2.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.2.self_attn.kv_b_proj as default Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.2.mlp as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.2.mlp.act_fn as default Injecting model.layers.2.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.2.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.3 as default Injecting model.layers.3.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.3.self_attn.kv_b_proj as default Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2 Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2 Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.3.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.3.mlp.shared_experts.act_fn as default Injecting model.layers.3.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.3.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.4 as default Injecting model.layers.4.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.4.self_attn.kv_b_proj as default Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2 Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2 Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.4.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.4.mlp.shared_experts.act_fn as default Injecting model.layers.4.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.4.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm . . . . Injecting model.layers.60 as default Injecting model.layers.60.self_attn as ktransformers.operators.balance_serve_attention . flashinfer_attn Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.q_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.60.self_attn.kv_b_proj as default Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoEV2 Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2 Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.60.mlp.shared_experts as ktransformers.operators.mlp . kDeepseekV3MLP Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.act_fn as default Injecting model.layers.60.input_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.layers.60.post_attention_layernorm as ktransformers.operators.layernorm . RMSNorm Injecting model.norm as ktransformers.operators.layernorm . RMSNorm Injecting cache as default Injecting lm_head as ktransformers.operators.linear . KTransformersLinear loading model.embed_tokens.weight to cpu loading model.layers.0.self_attn.q_a_layernorm.weight to cuda loading model.layers.0.self_attn.kv_a_layernorm.weight to cuda loading model.layers.0.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.0.input_layernorm.weight to cuda loading model.layers.0.post_attention_layernorm.weight to cuda loading model.layers.1.self_attn.q_a_layernorm.weight to cuda loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda loading model.layers.1.self_attn.kv_b_proj.weight to cuda:0 . . . . . loading model.layers.54.post_attention_layernorm.weight to cuda loading model.layers.55.self_attn.q_a_layernorm.weight to cuda loading model.layers.55.self_attn.kv_a_layernorm.weight to cuda loading model.layers.55.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.55.input_layernorm.weight to cuda loading model.layers.55.post_attention_layernorm.weight to cuda loading model.layers.56.self_attn.q_a_layernorm.weight to cuda loading model.layers.56.self_attn.kv_a_layernorm.weight to cuda loading model.layers.56.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.56.input_layernorm.weight to cuda loading model.layers.56.post_attention_layernorm.weight to cuda loading model.layers.57.self_attn.q_a_layernorm.weight to cuda loading model.layers.57.self_attn.kv_a_layernorm.weight to cuda loading model.layers.57.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.57.input_layernorm.weight to cuda loading model.layers.57.post_attention_layernorm.weight to cuda loading model.layers.58.self_attn.q_a_layernorm.weight to cuda loading model.layers.58.self_attn.kv_a_layernorm.weight to cuda loading model.layers.58.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.58.input_layernorm.weight to cuda loading model.layers.58.post_attention_layernorm.weight to cuda loading model.layers.59.self_attn.q_a_layernorm.weight to cuda loading model.layers.59.self_attn.kv_a_layernorm.weight to cuda loading model.layers.59.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.59.input_layernorm.weight to cuda loading model.layers.59.post_attention_layernorm.weight to cuda loading model.layers.60.self_attn.q_a_layernorm.weight to cuda loading model.layers.60.self_attn.kv_a_layernorm.weight to cuda loading model.layers.60.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.60.input_layernorm.weight to cuda loading model.layers.60.post_attention_layernorm.weight to cuda loading model.norm.weight to cuda Getting inference context from sched_client. sched_rpc started with PID: 381 ### Environment ##Environment - **Host OS**: Ubuntu 22.04 LTS - **Docker Engine**: e.g. 24.0.2 - **ktransformers image**: `approachingai/ktransformers:v0.3.1-AVX512` - **GPU**: NVIDIA RTX 4090 - **CUDA version**: from `nvidia-smi` nvcc --version Cuda compilation tools, release 12.9, V12.9.41 Build cuda_12.9.r12.9/compiler.35813241_0 nvidia-smi Driver Version: 575.51.03 CUDA Version: 12.9 - **Python**: 3.11.x inside container
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@113514089 commented on GitHub (May 27, 2025):

遇到 一样的问题。

<!-- gh-comment-id:2912532137 --> @113514089 commented on GitHub (May 27, 2025): 遇到 一样的问题。
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@Ice-YY commented on GitHub (May 29, 2025):

一样的问题， 完全按照官方说明编译的，用ktransformer 架构可以正常运行

<!-- gh-comment-id:2919001054 --> @Ice-YY commented on GitHub (May 29, 2025): 一样的问题， 完全按照官方说明编译的，用ktransformer 架构可以正常运行
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@apalesnail commented on GitHub (Jun 2, 2025):

Same problem here. Appears model is loaded and ready however there is no way to interact via port 10002

<!-- gh-comment-id:2928555557 --> @apalesnail commented on GitHub (Jun 2, 2025): Same problem here. Appears model is loaded and ready however there is no way to interact via port 10002
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@EthanFeng-coder commented on GitHub (Jun 3, 2025):

same on Qwen_Qwen3-30B-A3B-GGUF no server start DeepSeek-V2-Lite-Chat-GGUF working

<!-- gh-comment-id:2933745992 --> @EthanFeng-coder commented on GitHub (Jun 3, 2025): same on Qwen_Qwen3-30B-A3B-GGUF no server start DeepSeek-V2-Lite-Chat-GGUF working
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@JamesDConley commented on GitHub (Jun 12, 2025):

I've had the same issue on an Epyc 7502 system with a couple A6000's.
Built the docker image locally and got this when running like

python ktransformers/server/main.py \
  --model_path deepseek-ai/DeepSeek-V3 \
  --gguf_path /models/DeepSeek-R1-0528/Q4_K_M \
  --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml \
  --cpu_infer 62 \
  --max_new_tokens 1024 \
  --max_batch_size 4 \
  --chunk_size 256 \
  --cache_lens 32768 \
  --backend_type balance_serve \
  --port 10002

and some minor variations therein

Also tried running via

python ktransformers/local_chat.py \
  --model_path deepseek-ai/DeepSeek-V3 \
  --gguf_path /models/DeepSeek-R1-0528/Q4_K_M \
  --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
  --cpu_infer 62 \
  --max_new_tokens 1024 \
  --max_batch_size 4 \
  --chunk_size 256 \
  --cache_lens 32768 \
  --backend_type balance_serve \
  --port 10002

and ended up getting

Chat: Hi, how are you?
Traceback (most recent call last):
  File "/workspace/ktransformers/ktransformers/local_chat.py", line 196, in <module>
    fire.Fire(local_chat)
  File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/local_chat.py", line 185, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/util/utils.py", line 334, in prefill_and_generate
    logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/util/utils.py", line 286, in chunk_prefill
    logits = model(
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/operators/models.py", line 751, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'
root@09232ce5dc6b:/workspace/ktransformers#
<!-- gh-comment-id:2965030999 --> @JamesDConley commented on GitHub (Jun 12, 2025): I've had the same issue on an Epyc 7502 system with a couple A6000's. Built the docker image locally and got this when running like ``` python ktransformers/server/main.py \ --model_path deepseek-ai/DeepSeek-V3 \ --gguf_path /models/DeepSeek-R1-0528/Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml \ --cpu_infer 62 \ --max_new_tokens 1024 \ --max_batch_size 4 \ --chunk_size 256 \ --cache_lens 32768 \ --backend_type balance_serve \ --port 10002 ``` and some minor variations therein Also tried running via ``` python ktransformers/local_chat.py \ --model_path deepseek-ai/DeepSeek-V3 \ --gguf_path /models/DeepSeek-R1-0528/Q4_K_M \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --cpu_infer 62 \ --max_new_tokens 1024 \ --max_batch_size 4 \ --chunk_size 256 \ --cache_lens 32768 \ --backend_type balance_serve \ --port 10002 ``` and ended up getting ``` Chat: Hi, how are you? Traceback (most recent call last): File "/workspace/ktransformers/ktransformers/local_chat.py", line 196, in <module> fire.Fire(local_chat) File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/local_chat.py", line 185, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/util/utils.py", line 334, in prefill_and_generate logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/util/utils.py", line 286, in chunk_prefill logits = model( ^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward outputs = self.model( ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/operators/models.py", line 751, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask' root@09232ce5dc6b:/workspace/ktransformers# ```
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@apalesnail commented on GitHub (Jun 12, 2025):

Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).

<!-- gh-comment-id:2966773629 --> @apalesnail commented on GitHub (Jun 12, 2025): Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).
bsakaguchi commented 2026-10-29 14:45:54 +00:00
Author
Owner
Copy link

@3dlabsio commented on GitHub (Jul 4, 2025):

Same problem and copying the small files into model_path still didn't work.

<!-- gh-comment-id:3034240837 --> @3dlabsio commented on GitHub (Jul 4, 2025): Same problem and copying the small files into model_path still didn't work.
bsakaguchi commented 2026-10-29 14:45:55 +00:00
Author
Owner
Copy link

@JamesDConley commented on GitHub (Jul 7, 2025):

Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).

Yeah this solved it for me, thanks!

<!-- gh-comment-id:3043303899 --> @JamesDConley commented on GitHub (Jul 7, 2025): > Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face). Yeah this solved it for me, thanks!
bsakaguchi commented 2026-10-29 14:45:55 +00:00
Author
Owner
Copy link

@TheLegendOfKitty commented on GitHub (Jul 9, 2025):

Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).

What version are you using where this works?

<!-- gh-comment-id:3054348781 --> @TheLegendOfKitty commented on GitHub (Jul 9, 2025): > Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face). What version are you using where this works?
bsakaguchi commented 2026-10-29 14:45:55 +00:00
Author
Owner
Copy link

@apalesnail commented on GitHub (Jul 9, 2025):

This works for 3.1 & 3.2. I have not tested earlier versions

<!-- gh-comment-id:3054440826 --> @apalesnail commented on GitHub (Jul 9, 2025): This works for 3.1 & 3.2. I have not tested earlier versions
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#900
No description provided.