deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #554] 使用auto-gptq 量化DeepSeek-R1-Distill-Llama-8B模型，遇到了错误Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm) #442

New issue
Closed
opened 2026-10-26 21:21:23 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:21:23 +00:00
Owner
Copy link

Originally created by @davidray222 on GitHub (Mar 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/554

当前行为 | Current Behavior
我使用auto-gptq量化方式，量化了Qwen/QwQ-32B，但遇到了

File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 335, in forward
freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)
我是使用AutoGPTQ/examples/quantization
/quant_with_alpaca.py 的程序来量化。
指令为: CUDA_VISIBLE_DEVICES=0 python quant_with_alpaca.py --pretrained_model_dir deepseek-ai/DeepSeek-R1-Distill-Llama-8B --quantized_model_dir deepseek-llama7b-quant4bit-g32 --bits 4 --group_size 32 --save_and_reload

import json
import random
import time
from argparse import ArgumentParser

import torch
from datasets import Dataset
from transformers import AutoTokenizer, TextGenerationPipeline

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig


def load_data(data_path, tokenizer, n_samples):
    with open(data_path, "r", encoding="utf-8") as f:
        raw_data = json.load(f)

    raw_data = random.sample(raw_data, k=min(n_samples, len(raw_data)))

    def dummy_gen():
        return raw_data

    def tokenize(examples):
        instructions = examples["instruction"]
        inputs = examples["input"]
        outputs = examples["output"]

        prompts = []
        texts = []
        input_ids = []
        attention_mask = []
        for istr, inp, opt in zip(instructions, inputs, outputs):
            if inp:
                prompt = f"Instruction:\n{istr}\nInput:\n{inp}\nOutput:\n"
                text = prompt + opt
            else:
                prompt = f"Instruction:\n{istr}\nOutput:\n"
                text = prompt + opt
            if len(tokenizer(prompt)["input_ids"]) >= tokenizer.model_max_length:
                continue

            tokenized_data = tokenizer(text)

            input_ids.append(tokenized_data["input_ids"][: tokenizer.model_max_length])
            attention_mask.append(tokenized_data["attention_mask"][: tokenizer.model_max_length])
            prompts.append(prompt)
            texts.append(text)

        return {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "prompt": prompts,
        }

    dataset = Dataset.from_generator(dummy_gen)

    dataset = dataset.map(
        tokenize,
        batched=True,
        batch_size=len(dataset),
        num_proc=1,
        keep_in_memory=True,
        load_from_cache_file=False,
        remove_columns=["instruction", "input"],
    )

    dataset = dataset.to_list()

    for sample in dataset:
        sample["input_ids"] = torch.LongTensor(sample["input_ids"])
        sample["attention_mask"] = torch.LongTensor(sample["attention_mask"])

    return dataset


def main():
    parser = ArgumentParser()
    parser.add_argument("--pretrained_model_dir", type=str)
    parser.add_argument("--quantized_model_dir", type=str, default=None)
    parser.add_argument("--bits", type=int, default=4, choices=[2, 3, 4, 8])
    parser.add_argument(
        "--group_size",
        type=int,
        default=128,
        help="group size, -1 means no grouping or full rank",
    )
    parser.add_argument("--desc_act", action="store_true", help="whether to quantize with desc_act")
    parser.add_argument(
        "--num_samples",
        type=int,
        default=128,
        help="how many samples will be used to quantize model",
    )
    parser.add_argument(
        "--save_and_reload",
        action="store_true",
        help="whether save quantized model to disk and reload back",
    )
    parser.add_argument("--fast_tokenizer", action="store_true", help="whether use fast tokenizer")
    parser.add_argument(
        "--use_triton",
        action="store_true",
        help="whether use triton to speedup at inference",
    )
    parser.add_argument(
        "--per_gpu_max_memory",
        type=int,
        default=None,
        help="max memory used to load model per gpu",
    )
    parser.add_argument(
        "--cpu_max_memory",
        type=int,
        default=None,
        help="max memory used to offload model to cpu",
    )
    parser.add_argument(
        "--quant_batch_size",
        type=int,
        default=1,
        help="examples batch size for quantization",
    )
    parser.add_argument(
        "--trust_remote_code",
        action="store_true",
        help="whether to trust remote code when loading model",
    )
    args = parser.parse_args()

    max_memory = {}
    if args.per_gpu_max_memory is not None and args.per_gpu_max_memory > 0:
        if torch.cuda.is_available():
            max_memory.update({i: f"{args.per_gpu_max_memory}GIB" for i in range(torch.cuda.device_count())})
    if args.cpu_max_memory is not None and args.cpu_max_memory > 0 and max_memory:
        max_memory["cpu"] = f"{args.cpu_max_memory}GIB"
    if not max_memory:
        max_memory = None

    tokenizer = AutoTokenizer.from_pretrained(
        args.pretrained_model_dir,
        use_fast=args.fast_tokenizer,
        trust_remote_code=args.trust_remote_code,
    )
    model = AutoGPTQForCausalLM.from_pretrained(
        args.pretrained_model_dir,
        quantize_config=BaseQuantizeConfig(bits=args.bits, group_size=args.group_size, desc_act=args.desc_act),
        max_memory=max_memory,
        trust_remote_code=args.trust_remote_code,
    )

    examples = load_data("dataset/alpaca_data_cleaned.json", tokenizer, args.num_samples)
    examples_for_quant = [
        {"input_ids": example["input_ids"], "attention_mask": example["attention_mask"]} for example in examples
    ]

    start = time.time()
    model.quantize(
        examples_for_quant,
        batch_size=args.quant_batch_size,
        use_triton=args.use_triton,
        autotune_warmup_after_quantized=args.use_triton,
    )
    end = time.time()
    print(f"quantization took: {end - start: .4f}s")

    if not args.quantized_model_dir:
        args.quantized_model_dir = args.pretrained_model_dir

    if args.save_and_reload:
        model.save_quantized(args.quantized_model_dir)
        del model
        if torch.cuda.is_available():
            torch.cuda.empty_cache()
        model = AutoGPTQForCausalLM.from_quantized(
            args.quantized_model_dir,
            device="cuda:0",
            use_triton=args.use_triton,
            max_memory=max_memory,
            inject_fused_mlp=True,
            inject_fused_attention=True,
            trust_remote_code=args.trust_remote_code,
        )

    pipeline_init_kwargs = {"model": model, "tokenizer": tokenizer}
    if not max_memory:
        pipeline_init_kwargs["device"] = "cuda:0"
    pipeline = TextGenerationPipeline(**pipeline_init_kwargs)
    for example in random.sample(examples, k=min(4, len(examples))):
        print(f"prompt: {example['prompt']}")
        print("-" * 42)
        print(f"golden: {example['output']}")
        print("-" * 42)
        start = time.time()
        generated_text = pipeline(
            example["prompt"],
            return_full_text=False,
            num_beams=1,
            max_length=len(example["input_ids"])
            + 128,  # use this instead of max_new_token to disable UserWarning when integrate with logging
        )[0]["generated_text"]
        end = time.time()
        print(f"quant: {generated_text}")
        num_new_tokens = len(tokenizer(generated_text)["input_ids"])
        print(f"generate {num_new_tokens} tokens using {end-start: .4f}s, {num_new_tokens / (end - start)} tokens/s.")
        print("=" * 42)


if __name__ == "__main__":
    import logging

    logging.basicConfig(
        format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
        level=logging.INFO,
        datefmt="%Y-%m-%d %H:%M:%S",
    )

    main()

但遇到

Traceback (most recent call last):
  File "/mnt/8tb_raid/david_model/qalora_author/AutoGPTQ/examples/quantization/quant_with_alpaca.py", line 216, in <module>
    main()
  File "/mnt/8tb_raid/david_model/qalora_author/AutoGPTQ/examples/quantization/quant_with_alpaca.py", line 156, in main
    model.quantize(
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 392, in quantize
    self.model(**example)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 842, in forward
    outputs = self.model(
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 571, in forward
    position_embeddings = self.rotary_emb(hidden_states, position_ids)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 132, in forward
    freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)

期望行为 | Expected Behavior
能成功量化模型成8bit 4bit group_size=32

复现方法 | Steps To Reproduce
No response

运行环境 | Environment

  • OS:
  • Python:3.10.16
  • Transformers:4.49.0
  • PyTorch:2.4.1
  • CUDA (python -c 'import torch; print(torch.version.cuda)'):12.6
  • auto-gptq:0.7.1
    备注 | Anything else?
    想請為各位有沒有解決方法，或是誰有成功使用auto-gptq量化模型的人能提供步驟，謝謝各位先進!
Originally created by @davidray222 on GitHub (Mar 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/554 当前行为 | Current Behavior 我使用auto-gptq量化方式，量化了Qwen/QwQ-32B，但遇到了 File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 335, in forward freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm) 我是使用AutoGPTQ/examples/quantization /quant_with_alpaca.py 的程序来量化。 指令为: `CUDA_VISIBLE_DEVICES=0 python quant_with_alpaca.py --pretrained_model_dir deepseek-ai/DeepSeek-R1-Distill-Llama-8B --quantized_model_dir deepseek-llama7b-quant4bit-g32 --bits 4 --group_size 32 --save_and_reload` ``` import json import random import time from argparse import ArgumentParser import torch from datasets import Dataset from transformers import AutoTokenizer, TextGenerationPipeline from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig def load_data(data_path, tokenizer, n_samples): with open(data_path, "r", encoding="utf-8") as f: raw_data = json.load(f) raw_data = random.sample(raw_data, k=min(n_samples, len(raw_data))) def dummy_gen(): return raw_data def tokenize(examples): instructions = examples["instruction"] inputs = examples["input"] outputs = examples["output"] prompts = [] texts = [] input_ids = [] attention_mask = [] for istr, inp, opt in zip(instructions, inputs, outputs): if inp: prompt = f"Instruction:\n{istr}\nInput:\n{inp}\nOutput:\n" text = prompt + opt else: prompt = f"Instruction:\n{istr}\nOutput:\n" text = prompt + opt if len(tokenizer(prompt)["input_ids"]) >= tokenizer.model_max_length: continue tokenized_data = tokenizer(text) input_ids.append(tokenized_data["input_ids"][: tokenizer.model_max_length]) attention_mask.append(tokenized_data["attention_mask"][: tokenizer.model_max_length]) prompts.append(prompt) texts.append(text) return { "input_ids": input_ids, "attention_mask": attention_mask, "prompt": prompts, } dataset = Dataset.from_generator(dummy_gen) dataset = dataset.map( tokenize, batched=True, batch_size=len(dataset), num_proc=1, keep_in_memory=True, load_from_cache_file=False, remove_columns=["instruction", "input"], ) dataset = dataset.to_list() for sample in dataset: sample["input_ids"] = torch.LongTensor(sample["input_ids"]) sample["attention_mask"] = torch.LongTensor(sample["attention_mask"]) return dataset def main(): parser = ArgumentParser() parser.add_argument("--pretrained_model_dir", type=str) parser.add_argument("--quantized_model_dir", type=str, default=None) parser.add_argument("--bits", type=int, default=4, choices=[2, 3, 4, 8]) parser.add_argument( "--group_size", type=int, default=128, help="group size, -1 means no grouping or full rank", ) parser.add_argument("--desc_act", action="store_true", help="whether to quantize with desc_act") parser.add_argument( "--num_samples", type=int, default=128, help="how many samples will be used to quantize model", ) parser.add_argument( "--save_and_reload", action="store_true", help="whether save quantized model to disk and reload back", ) parser.add_argument("--fast_tokenizer", action="store_true", help="whether use fast tokenizer") parser.add_argument( "--use_triton", action="store_true", help="whether use triton to speedup at inference", ) parser.add_argument( "--per_gpu_max_memory", type=int, default=None, help="max memory used to load model per gpu", ) parser.add_argument( "--cpu_max_memory", type=int, default=None, help="max memory used to offload model to cpu", ) parser.add_argument( "--quant_batch_size", type=int, default=1, help="examples batch size for quantization", ) parser.add_argument( "--trust_remote_code", action="store_true", help="whether to trust remote code when loading model", ) args = parser.parse_args() max_memory = {} if args.per_gpu_max_memory is not None and args.per_gpu_max_memory > 0: if torch.cuda.is_available(): max_memory.update({i: f"{args.per_gpu_max_memory}GIB" for i in range(torch.cuda.device_count())}) if args.cpu_max_memory is not None and args.cpu_max_memory > 0 and max_memory: max_memory["cpu"] = f"{args.cpu_max_memory}GIB" if not max_memory: max_memory = None tokenizer = AutoTokenizer.from_pretrained( args.pretrained_model_dir, use_fast=args.fast_tokenizer, trust_remote_code=args.trust_remote_code, ) model = AutoGPTQForCausalLM.from_pretrained( args.pretrained_model_dir, quantize_config=BaseQuantizeConfig(bits=args.bits, group_size=args.group_size, desc_act=args.desc_act), max_memory=max_memory, trust_remote_code=args.trust_remote_code, ) examples = load_data("dataset/alpaca_data_cleaned.json", tokenizer, args.num_samples) examples_for_quant = [ {"input_ids": example["input_ids"], "attention_mask": example["attention_mask"]} for example in examples ] start = time.time() model.quantize( examples_for_quant, batch_size=args.quant_batch_size, use_triton=args.use_triton, autotune_warmup_after_quantized=args.use_triton, ) end = time.time() print(f"quantization took: {end - start: .4f}s") if not args.quantized_model_dir: args.quantized_model_dir = args.pretrained_model_dir if args.save_and_reload: model.save_quantized(args.quantized_model_dir) del model if torch.cuda.is_available(): torch.cuda.empty_cache() model = AutoGPTQForCausalLM.from_quantized( args.quantized_model_dir, device="cuda:0", use_triton=args.use_triton, max_memory=max_memory, inject_fused_mlp=True, inject_fused_attention=True, trust_remote_code=args.trust_remote_code, ) pipeline_init_kwargs = {"model": model, "tokenizer": tokenizer} if not max_memory: pipeline_init_kwargs["device"] = "cuda:0" pipeline = TextGenerationPipeline(**pipeline_init_kwargs) for example in random.sample(examples, k=min(4, len(examples))): print(f"prompt: {example['prompt']}") print("-" * 42) print(f"golden: {example['output']}") print("-" * 42) start = time.time() generated_text = pipeline( example["prompt"], return_full_text=False, num_beams=1, max_length=len(example["input_ids"]) + 128, # use this instead of max_new_token to disable UserWarning when integrate with logging )[0]["generated_text"] end = time.time() print(f"quant: {generated_text}") num_new_tokens = len(tokenizer(generated_text)["input_ids"]) print(f"generate {num_new_tokens} tokens using {end-start: .4f}s, {num_new_tokens / (end - start)} tokens/s.") print("=" * 42) if __name__ == "__main__": import logging logging.basicConfig( format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S", ) main() ``` 但遇到 ``` Traceback (most recent call last): File "/mnt/8tb_raid/david_model/qalora_author/AutoGPTQ/examples/quantization/quant_with_alpaca.py", line 216, in <module> main() File "/mnt/8tb_raid/david_model/qalora_author/AutoGPTQ/examples/quantization/quant_with_alpaca.py", line 156, in main model.quantize( File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 392, in quantize self.model(**example) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func return func(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 842, in forward outputs = self.model( File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 571, in forward position_embeddings = self.rotary_emb(hidden_states, position_ids) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/home/david/miniconda3/envs/deepseek/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 132, in forward freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm) ``` 期望行为 | Expected Behavior 能成功量化模型成8bit 4bit group_size=32 复现方法 | Steps To Reproduce No response 运行环境 | Environment - OS: - Python:3.10.16 - Transformers:4.49.0 - PyTorch:2.4.1 - CUDA (`python -c 'import torch; print(torch.version.cuda)'`):12.6 - auto-gptq:0.7.1 备注 | Anything else? 想請為各位有沒有解決方法，或是誰有成功使用auto-gptq量化模型的人能提供步驟，謝謝各位先進!
bsakaguchi 2026-10-26 21:21:23 +00:00
bsakaguchi commented 2026-10-26 21:21:25 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Apr 15, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:2803375120 --> @github-actions[bot] commented on GitHub (Apr 15, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:21:25 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (May 18, 2025):

false

<!-- gh-comment-id:2888651431 --> @github-actions[bot] commented on GitHub (May 18, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#442
No description provided.