deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #487] AttributeError: Layer 'ColumnParallelLinear(in_features=512, output_features=2048, bias=False, tp_size=16, gather_output=False)' has neither weight nor qweight #389

New issue
Closed
opened 2026-10-26 21:20:31 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:20:31 +00:00
Owner
Copy link

Originally created by @Kha-Zix-1 on GitHub (Feb 23, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/487

I want to get a W8A16 deepseekR1-671B. The step is:

I make a little change to the script, here is my script

from transformers import AutoTokenizer
from modeling_deepseek import DeepseekV3ForCausalLM

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot

MODEL_ID = "/data/models/DeepSeek-R1-bf16/DeepSeek-R1-bf16/"
OUTPUT_DIR = "/data/models/DeepSeek-R1-w8a16"

# Load model
# Note: device_map="auto" will offload to CPU if not enough space on GPU.
model = DeepseekV3ForCausalLM.from_pretrained(
    MODEL_ID, device_map="auto", torch_dtype="auto", trust_remote_code=True
)

# Configure the quantization scheme and algorithm (PTQ + FP8_DYNAMIC).
recipe = QuantizationModifier(
    targets="Linear", scheme="W8A16", ignore=["lm_head", "re:.*mlp.gate$"]
)

# Apply quantization and save in `compressed-tensors` format.
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=AutoTokenizer.from_pretrained(MODEL_ID),
    output_dir=OUTPUT_DIR,
)

and here is part of the model.safetensors.index.json of the quantized model:

    "model.layers.25.self_attn.kv_a_layernorm.weight": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_packed": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_scale": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_shape": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_b_proj.weight_packed": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_b_proj.weight_scale": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.kv_b_proj.weight_shape": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.o_proj.weight_packed": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.o_proj.weight_scale": "model-00052-of-00135.safetensors",
    "model.layers.25.self_attn.o_proj.weight_shape": "model-00052-of-00135.safetensors",

When I use vllm to load the model, the error is:

Traceback (most recent call last):
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
ERROR 02-23 12:04:38 engine.py:400]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
ERROR 02-23 12:04:38 engine.py:400]     return cls(ipc_path=ipc_path,
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__
ERROR 02-23 12:04:38 engine.py:400]     self.engine = LLMEngine(*args, **kwargs)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 273, in __init__
ERROR 02-23 12:04:38 engine.py:400]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 271, in __init__
ERROR 02-23 12:04:38 engine.py:400]     super().__init__(*args, **kwargs)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 02-23 12:04:38 engine.py:400]     self._init_executor()
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 90, in _init_executor
ERROR 02-23 12:04:38 engine.py:400]     self._init_workers_ray(placement_group)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 360, in _init_workers_ray
ERROR 02-23 12:04:38 engine.py:400]     self._run_workers("load_model",
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 480, in _run_workers
ERROR 02-23 12:04:38 engine.py:400]     self.driver_worker.execute_method(sent_method, *args, **kwargs)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 582, in execute_method
ERROR 02-23 12:04:38 engine.py:400]     raise e
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 573, in execute_method
ERROR 02-23 12:04:38 engine.py:400]     return run_method(target, method, args, kwargs)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
ERROR 02-23 12:04:38 engine.py:400]     return func(*args, **kwargs)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker.py", line 183, in load_model
ERROR 02-23 12:04:38 engine.py:400]     self.model_runner.load_model()
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
ERROR 02-23 12:04:38 engine.py:400]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
ERROR 02-23 12:04:38 engine.py:400]     return loader.load_model(vllm_config=vllm_config)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 420, in load_model
ERROR 02-23 12:04:38 engine.py:400]     _process_weights_after_loading(model, model_config, target_device)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 177, in _process_weights_after_loading
ERROR 02-23 12:04:38 engine.py:400]     module.process_weights_after_loading(model_config.dtype)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/layer.py", line 222, in process_weights_after_loading
ERROR 02-23 12:04:38 engine.py:400]     self.impl.process_weights_after_loading(act_dtype)
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py", line 293, in process_weights_after_loading
ERROR 02-23 12:04:38 engine.py:400]     weight_dtype = get_layer_weight(self.kv_b_proj).dtype
ERROR 02-23 12:04:38 engine.py:400]   File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py", line 276, in get_layer_weight
ERROR 02-23 12:04:38 engine.py:400]     raise AttributeError(
ERROR 02-23 12:04:38 engine.py:400] AttributeError: Layer 'ColumnParallelLinear(in_features=512, output_features=2048, bias=False, tp_size=16, gather_output=False)' has neither weight nor qweight
Originally created by @Kha-Zix-1 on GitHub (Feb 23, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/487 I want to get a W8A16 deepseekR1-671B. The step is: + I download the fp8 version https://huggingface.co/deepseek-ai/DeepSeek-R1 + get the bf16 version using https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py + run the script https://github.com/vllm-project/llm-compressor/blob/main/examples/quantizing_moe/deepseek_moe_w8a8_int8.py I make a little change to the script, here is my script ``` python from transformers import AutoTokenizer from modeling_deepseek import DeepseekV3ForCausalLM from llmcompressor.modifiers.quantization import QuantizationModifier from llmcompressor.transformers import oneshot MODEL_ID = "/data/models/DeepSeek-R1-bf16/DeepSeek-R1-bf16/" OUTPUT_DIR = "/data/models/DeepSeek-R1-w8a16" # Load model # Note: device_map="auto" will offload to CPU if not enough space on GPU. model = DeepseekV3ForCausalLM.from_pretrained( MODEL_ID, device_map="auto", torch_dtype="auto", trust_remote_code=True ) # Configure the quantization scheme and algorithm (PTQ + FP8_DYNAMIC). recipe = QuantizationModifier( targets="Linear", scheme="W8A16", ignore=["lm_head", "re:.*mlp.gate$"] ) # Apply quantization and save in `compressed-tensors` format. oneshot( model=model, recipe=recipe, tokenizer=AutoTokenizer.from_pretrained(MODEL_ID), output_dir=OUTPUT_DIR, ) ``` and here is part of the `model.safetensors.index.json` of the quantized model: ``` "model.layers.25.self_attn.kv_a_layernorm.weight": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_packed": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_scale": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_a_proj_with_mqa.weight_shape": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_b_proj.weight_packed": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_b_proj.weight_scale": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.kv_b_proj.weight_shape": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.o_proj.weight_packed": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.o_proj.weight_scale": "model-00052-of-00135.safetensors", "model.layers.25.self_attn.o_proj.weight_shape": "model-00052-of-00135.safetensors", ``` When I use vllm to load the model, the error is: ``` Traceback (most recent call last): ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine ERROR 02-23 12:04:38 engine.py:400] engine = MQLLMEngine.from_engine_args(engine_args=engine_args, ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args ERROR 02-23 12:04:38 engine.py:400] return cls(ipc_path=ipc_path, ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__ ERROR 02-23 12:04:38 engine.py:400] self.engine = LLMEngine(*args, **kwargs) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 273, in __init__ ERROR 02-23 12:04:38 engine.py:400] self.model_executor = executor_class(vllm_config=vllm_config, ) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 271, in __init__ ERROR 02-23 12:04:38 engine.py:400] super().__init__(*args, **kwargs) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in __init__ ERROR 02-23 12:04:38 engine.py:400] self._init_executor() ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 90, in _init_executor ERROR 02-23 12:04:38 engine.py:400] self._init_workers_ray(placement_group) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 360, in _init_workers_ray ERROR 02-23 12:04:38 engine.py:400] self._run_workers("load_model", ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 480, in _run_workers ERROR 02-23 12:04:38 engine.py:400] self.driver_worker.execute_method(sent_method, *args, **kwargs) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 582, in execute_method ERROR 02-23 12:04:38 engine.py:400] raise e ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 573, in execute_method ERROR 02-23 12:04:38 engine.py:400] return run_method(target, method, args, kwargs) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method ERROR 02-23 12:04:38 engine.py:400] return func(*args, **kwargs) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/worker.py", line 183, in load_model ERROR 02-23 12:04:38 engine.py:400] self.model_runner.load_model() ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1112, in load_model ERROR 02-23 12:04:38 engine.py:400] self.model = get_model(vllm_config=self.vllm_config) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model ERROR 02-23 12:04:38 engine.py:400] return loader.load_model(vllm_config=vllm_config) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 420, in load_model ERROR 02-23 12:04:38 engine.py:400] _process_weights_after_loading(model, model_config, target_device) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 177, in _process_weights_after_loading ERROR 02-23 12:04:38 engine.py:400] module.process_weights_after_loading(model_config.dtype) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/layer.py", line 222, in process_weights_after_loading ERROR 02-23 12:04:38 engine.py:400] self.impl.process_weights_after_loading(act_dtype) ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py", line 293, in process_weights_after_loading ERROR 02-23 12:04:38 engine.py:400] weight_dtype = get_layer_weight(self.kv_b_proj).dtype ERROR 02-23 12:04:38 engine.py:400] File "/home/hcece/miniconda3/envs/vllm_073/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py", line 276, in get_layer_weight ERROR 02-23 12:04:38 engine.py:400] raise AttributeError( ERROR 02-23 12:04:38 engine.py:400] AttributeError: Layer 'ColumnParallelLinear(in_features=512, output_features=2048, bias=False, tp_size=16, gather_output=False)' has neither weight nor qweight ```
bsakaguchi commented 2026-10-26 21:20:32 +00:00
Author
Owner
Copy link

@Kha-Zix-1 commented on GitHub (Feb 23, 2025):

I have fixed this bug already! Let's turn to https://github.com/vllm-project/vllm/blob/v0.7.3/vllm/attention/backends/mla/utils.py

def get_layer_weight(layer):
    if hasattr(layer, "weight"):
        return layer.weight
    elif hasattr(layer, "qweight"):
        return layer.qweight
    else:
        raise AttributeError(
            f"Layer '{layer}' has neither weight nor qweight")

because in model.safetensors.index.json, the attention weight name is

weight_packed

Therefore the vllm can't know what the weight_packed is. I change it to

def get_layer_weight(layer):
      if hasattr(layer, "weight"):
          return layer.weight
      elif hasattr(layer, "qweight"):
          return layer.qweight
      elif hasattr(layer, "weight_packed"):
          return layer.weight_packed
      else:
          raise AttributeError(
            f"Layer '{layer}' has neither weight nor qweight")

and the bug is fixed!

<!-- gh-comment-id:2676847247 --> @Kha-Zix-1 commented on GitHub (Feb 23, 2025): I have fixed this bug already! Let's turn to https://github.com/vllm-project/vllm/blob/v0.7.3/vllm/attention/backends/mla/utils.py ``` python def get_layer_weight(layer): if hasattr(layer, "weight"): return layer.weight elif hasattr(layer, "qweight"): return layer.qweight else: raise AttributeError( f"Layer '{layer}' has neither weight nor qweight") ``` because in `model.safetensors.index.json`, the attention weight name is ``` weight_packed ``` Therefore the vllm can't know what the `weight_packed` is. I change it to ``` python def get_layer_weight(layer): if hasattr(layer, "weight"): return layer.weight elif hasattr(layer, "qweight"): return layer.qweight elif hasattr(layer, "weight_packed"): return layer.weight_packed else: raise AttributeError( f"Layer '{layer}' has neither weight nor qweight") ``` and the bug is fixed!
bsakaguchi commented 2026-10-26 21:20:33 +00:00
Author
Owner
Copy link

@Kha-Zix-1 commented on GitHub (Feb 23, 2025):

Here is how I execute the serve:

First I get 2 * 8A100 GPUs, and use Ray to set multi-node execution:

# Main-node
NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME="ens81f0" GLOO_SOCKET_IFNAME="ens81f0" ray start --head --dashboard-host 0.0.0.0
# second node
NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME="ens81f0" GLOO_SOCKET_IFNAME="ens81f0" ray start --address='<your-ip>:<port>'

Then set the global vllm parameter

VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
export VLLM_HOST_IP

Then execute the vllm serve on the main node:

NCCL_IB_DISABLE=1 NCCL_DEBUG=INFO python -m vllm.entrypoints.openai.api_server --model /data/models/DeepSeek-R1-w8a16 --trust-remote-code --served-model-name deepseek-r1-w8a16 --tensor-parallel-size 8 --pipeline-parallel-size 2 --gpu-memory-utilization 0.85 --uvicorn-log-level debug --max-model-len 16000 --host 0.0.0.0 --port 11000 --dtype float16
<!-- gh-comment-id:2676847547 --> @Kha-Zix-1 commented on GitHub (Feb 23, 2025): Here is how I execute the serve: First I get 2 * 8A100 GPUs, and use Ray to set multi-node execution: ``` # Main-node NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME="ens81f0" GLOO_SOCKET_IFNAME="ens81f0" ray start --head --dashboard-host 0.0.0.0 # second node NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME="ens81f0" GLOO_SOCKET_IFNAME="ens81f0" ray start --address='<your-ip>:<port>' ``` Then set the global vllm parameter ``` VLLM_HOST_IP=$(hostname -I | awk '{print $1}') export VLLM_HOST_IP ``` Then execute the vllm serve on the main node: ``` NCCL_IB_DISABLE=1 NCCL_DEBUG=INFO python -m vllm.entrypoints.openai.api_server --model /data/models/DeepSeek-R1-w8a16 --trust-remote-code --served-model-name deepseek-r1-w8a16 --tensor-parallel-size 8 --pipeline-parallel-size 2 --gpu-memory-utilization 0.85 --uvicorn-log-level debug --max-model-len 16000 --host 0.0.0.0 --port 11000 --dtype float16 ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#389
No description provided.