[GH-ISSUE #461] DeepSeek-V3-Q4_K_M docker 环境运行错误： Illegal instruction 。命令显示到 loading blk.3.attn_kv_b.weight to cuda:0 就停止了 #321

New issue

Closed

opened 2026-10-29 14:37:49 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:37:49 +00:00

Owner

Originally created by @xxzjyq on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/461

硬件：
- CPU: Intel(R) Xeon(R) Gold 6338 32 核心 2颗
- MEMORY： DDR4 64GB 8张
- GPU：4090 4 张，分配了一张给 docker 使用
环境：
- docker image: approachingai/ktransformers:0.2.1
- docker-compose

services:
  ktransformers:
    image: approachingai/ktransformers:0.2.1
    container_name: ktransformers
    shm_size: '450g'
    ports:
      - 9112:9112
    volumes:
      - /root/.cache/huggingface:/root/.cache/huggingface
      - /root/.cache/modelscope:/root/.cache/modelscope
      - ./data:/workspace/data
    environment:
      - HF_ENDPOINT=https://hf-mirror.com
    restart: always
    deploy:
      resources:
          devices:
            - driver: nvidia
              device_ids: ["2"]
              capabilities: [gpu]

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V3 --gguf_path /root/.cache/modelscope/hub/unsloth/DeepSeek-V3-GGUF/DeepSeek-V3-Q4_K_M --cpu_infer 33 --max_new_tokens 1000
using custom modeling_xxx.py.
...etc
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
/opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:482: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.)
  data = torch.from_numpy(data)
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_q_a_norm.weight to cuda:0
loading blk.1.attn_kv_a_norm.weight to cuda:0
loading blk.1.attn_kv_b.weight to cuda:0
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_q_a_norm.weight to cuda:0
loading blk.2.attn_kv_a_norm.weight to cuda:0
loading blk.2.attn_kv_b.weight to cuda:0
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_q_a_norm.weight to cuda:0
loading blk.3.attn_kv_a_norm.weight to cuda:0
loading blk.3.attn_kv_b.weight to cuda:0
Illegal instruction

Originally created by @xxzjyq on GitHub (Feb 18, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/461 + 硬件： - CPU: Intel(R) Xeon(R) Gold 6338 32 核心 2颗 - MEMORY： DDR4 64GB 8张 - GPU：4090 4 张，分配了一张给 docker 使用 + 环境： - docker image: approachingai/ktransformers:0.2.1 - docker-compose ```yaml services: ktransformers: image: approachingai/ktransformers:0.2.1 container_name: ktransformers shm_size: '450g' ports: - 9112:9112 volumes: - /root/.cache/huggingface:/root/.cache/huggingface - /root/.cache/modelscope:/root/.cache/modelscope - ./data:/workspace/data environment: - HF_ENDPOINT=https://hf-mirror.com restart: always deploy: resources: devices: - driver: nvidia device_ids: ["2"] capabilities: [gpu] ``` ```cmd python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V3 --gguf_path /root/.cache/modelscope/hub/unsloth/DeepSeek-V3-GGUF/DeepSeek-V3-Q4_K_M --cpu_infer 33 --max_new_tokens 1000 using custom modeling_xxx.py. ...etc Injecting model.layers.60.mlp.shared_experts.act_fn as default Injecting model.layers.60.input_layernorm as default Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as default loading token_embd.weight to cpu /opt/conda/lib/python3.10/site-packages/ktransformers/util/custom_gguf.py:482: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905979055/work/torch/csrc/utils/tensor_numpy.cpp:206.) data = torch.from_numpy(data) loading blk.0.attn_q_a_norm.weight to cuda:0 loading blk.0.attn_kv_a_norm.weight to cuda:0 loading blk.0.attn_kv_b.weight to cuda:0 loading blk.0.attn_norm.weight to cuda:0 loading blk.0.ffn_norm.weight to cuda:0 loading blk.1.attn_q_a_norm.weight to cuda:0 loading blk.1.attn_kv_a_norm.weight to cuda:0 loading blk.1.attn_kv_b.weight to cuda:0 loading blk.1.attn_norm.weight to cuda:0 loading blk.1.ffn_norm.weight to cuda:0 loading blk.2.attn_q_a_norm.weight to cuda:0 loading blk.2.attn_kv_a_norm.weight to cuda:0 loading blk.2.attn_kv_b.weight to cuda:0 loading blk.2.attn_norm.weight to cuda:0 loading blk.2.ffn_norm.weight to cuda:0 loading blk.3.attn_q_a_norm.weight to cuda:0 loading blk.3.attn_kv_a_norm.weight to cuda:0 loading blk.3.attn_kv_b.weight to cuda:0 Illegal instruction ```

bsakaguchi closed this issue

2026-10-29 14:37:50 +00:00

bsakaguchi commented

2026-10-29 14:37:50 +00:00

Author

Owner

@Atream commented on GitHub (Feb 18, 2025):

源码编译或者下载正确版本的whl，检查下你的CPU支持的指令集，不支持AVX512就下载AVX2的

Compile the source code or download the correct version of the .whl file. Check the instruction sets supported by your CPU. If AVX512 is not supported, download the version that supports AVX2.

@Atream commented on GitHub (Feb 18, 2025): 源码编译或者下载正确版本的whl，检查下你的CPU支持的指令集，不支持AVX512就下载AVX2的 Compile the source code or download the correct version of the .whl file. Check the instruction sets supported by your CPU. If AVX512 is not supported, download the version that supports AVX2.