[GH-ISSUE #41] Does not work on CPU

bsakaguchi commented

2026-10-26 21:10:33 +00:00

Owner

Originally created by @ghost on GitHub (Jan 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/41

I tried running DeepSeek-vl2-tiny on the CPU. But it throws an error.

Does it support CPU inference?

I tried backtracking, and an error is being thrown from the memory_efficient_attention of the "formers" package. When I checked the operator bindings, I saw that the operator class IMPL is only registered for the CUDA dispatch key.

So, how can I run this model on a CPU?

Below is the output:

NotImplementedError: No operator found for memory_efficient_attention_forward with inputs:
query : shape=(19, 729, 16, 72) (torch.bfloat16)
key : shape=(19, 729, 16, 72) (torch.bfloat16)
value : shape=(19, 729, 16, 72) (torch.bfloat16)
attn_bias : <class 'NoneType'>
p : 0.0
ckF is not supported because:
device=cpu (supported: {'cuda'})
bf16 is only supported on A100+ GPUs

Originally created by @ghost on GitHub (Jan 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/41 I tried running DeepSeek-vl2-tiny on the CPU. But it throws an error. Does it support CPU inference? I tried backtracking, and an error is being thrown from the memory_efficient_attention of the "formers" package. When I checked the operator bindings, I saw that the operator class IMPL is only registered for the CUDA dispatch key. So, how can I run this model on a CPU? Below is the output: NotImplementedError: No operator found for `memory_efficient_attention_forward` with inputs: query : shape=(19, 729, 16, 72) (torch.bfloat16) key : shape=(19, 729, 16, 72) (torch.bfloat16) value : shape=(19, 729, 16, 72) (torch.bfloat16) attn_bias : <class 'NoneType'> p : 0.0 `ckF` is not supported because: device=cpu (supported: {'cuda'}) bf16 is only supported on A100+ GPUs

bsakaguchi commented

2026-10-26 21:10:35 +00:00

Author

Owner

@JosTheBossX commented on GitHub (Jan 27, 2025):

Did you solve this?

@JosTheBossX commented on GitHub (Jan 27, 2025): Did you solve this?

bsakaguchi commented

2026-10-26 21:10:35 +00:00

Author

Owner

@pappukrs commented on GitHub (Jan 30, 2025):

Had you solved?

@pappukrs commented on GitHub (Jan 30, 2025): Had you solved?

bsakaguchi commented

2026-10-26 21:10:35 +00:00

Author

Owner

@saart commented on GitHub (Feb 1, 2025):

I opened a PR to solve it: https://github.com/deepseek-ai/DeepSeek-VL2/pull/48

@saart commented on GitHub (Feb 1, 2025): I opened a PR to solve it: https://github.com/deepseek-ai/DeepSeek-VL2/pull/48

bsakaguchi commented

2026-10-26 21:10:35 +00:00

Author

Owner

@research-and-develop commented on GitHub (Mar 17, 2025):

I was struggling to get the model to work with CPU as well. I needed to run the model on CPU just for testing purpose. With the help of other AI tools I came to the following script which is working with deepseek-ai/deepseek-vl2-tiny on a laptop with 24GB RAM, but when tried to run deepseek-ai/deepseek-vl2-small, I got out of memory exception. The inference process is quite slow. The OS I am using is Windows 11/10

Here is the code:

import torch
import torch.nn.functional as F
from deepseek_vl2.utils.io import load_pil_images
from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.models.siglip_vit import Attention  # Import the Attention class

# Patch the Attention.forward method to use PyTorch's scaled_dot_product_attention
original_forward = Attention.forward

def patched_attention_forward(self, x):
    B, N, C = x.shape
    qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
    q, k, v = qkv[0], qkv[1], qkv[2] 
    attn_output = F.scaled_dot_product_attention(
        q, k, v, dropout_p=self.attn_drop.p if self.training else 0.0
    )
    x = attn_output.transpose(1, 2).reshape(B, N, C)
    x = self.proj(x)
    x = self.proj_drop(x)
    return x

Attention.forward = patched_attention_forward

model_path = "deepseek-ai/deepseek-vl2-tiny"

vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_path)

tokenizer = vl_chat_processor.tokenizer

# Load model with float32 and CPU-only inference
vl_gpt = DeepseekVLV2ForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float32,  # Use float32 for CPU compatibility
    device_map="cpu",           # Explicitly use CPU
).eval()

# Single image conversation example
conversation = [
    {
        "role": "<|User|>",
        "content": "<image>\n<|ref|>Describe what is on the image<|/ref|>.",
        "images": ["path/to/image.jpg"],
    },
    {"role": "<|Assistant|>", "content": ""},
]

pil_images = load_pil_images(conversation)

prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
)

# Cast inputs to float32 to match model dtype
prepare_inputs = prepare_inputs.to(dtype=torch.float32, device=vl_gpt.device)

inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

outputs = vl_gpt.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]} {answer}")

Details of the virtual environment where I am running the script:

$ python --version
Python 3.10.5

$ pip list --format=columns
Package            Version 
------------------ ----------- 
accelerate         1.5.1
attrdict           2.0.1
certifi            2025.1.31
charset-normalizer 3.4.1
colorama           0.4.6
deepseek_vl2       1.0.0       
einops             0.8.1
filelock           3.18.0
fsspec             2025.3.0
huggingface-hub    0.29.3
idna               3.10
intel-openmp       2021.4.0
Jinja2             3.1.6
MarkupSafe         3.0.2
mkl                2021.4.0
mpmath             1.3.0
networkx           3.4.2
numpy              1.26.4
packaging          24.2
pillow             11.1.0
pip                22.0.4
psutil             7.0.0
regex              2024.11.6
requests           2.32.3
safetensors        0.5.3
sentencepiece      0.2.0
setuptools         58.1.0
six                1.17.0
sympy              1.13.1
tbb                2021.13.1
timm               1.0.15
tokenizers         0.15.2
torch              2.0.1+cu118
torchvision        0.15.2
tqdm               4.67.1
transformers       4.38.2
typing_extensions  4.12.2
urllib3            2.3.0
xformers           0.0.22

@research-and-develop commented on GitHub (Mar 17, 2025): I was struggling to get the model to work with CPU as well. I needed to run the model on CPU just for testing purpose. With the help of other AI tools I came to the following script which is working with `deepseek-ai/deepseek-vl2-tiny` on a laptop with 24GB RAM, but when tried to run `deepseek-ai/deepseek-vl2-small`, I got out of memory exception. The inference process is quite slow. The OS I am using is Windows 11/10 Here is the code: ``` import torch import torch.nn.functional as F from deepseek_vl2.utils.io import load_pil_images from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM from deepseek_vl2.models.siglip_vit import Attention # Import the Attention class # Patch the Attention.forward method to use PyTorch's scaled_dot_product_attention original_forward = Attention.forward def patched_attention_forward(self, x): B, N, C = x.shape qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4) q, k, v = qkv[0], qkv[1], qkv[2] attn_output = F.scaled_dot_product_attention( q, k, v, dropout_p=self.attn_drop.p if self.training else 0.0 ) x = attn_output.transpose(1, 2).reshape(B, N, C) x = self.proj(x) x = self.proj_drop(x) return x Attention.forward = patched_attention_forward model_path = "deepseek-ai/deepseek-vl2-tiny" vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer # Load model with float32 and CPU-only inference vl_gpt = DeepseekVLV2ForCausalLM.from_pretrained( model_path, torch_dtype=torch.float32, # Use float32 for CPU compatibility device_map="cpu", # Explicitly use CPU ).eval() # Single image conversation example conversation = [ { "role": "<|User|>", "content": "<image>\n<|ref|>Describe what is on the image<|/ref|>.", "images": ["path/to/image.jpg"], }, {"role": "<|Assistant|>", "content": ""}, ] pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ) # Cast inputs to float32 to match model dtype prepare_inputs = prepare_inputs.to(dtype=torch.float32, device=vl_gpt.device) inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) outputs = vl_gpt.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True, ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True) print(f"{prepare_inputs['sft_format'][0]} {answer}") ``` Details of the virtual environment where I am running the script: ``` $ python --version Python 3.10.5 $ pip list --format=columns Package Version ------------------ ----------- accelerate 1.5.1 attrdict 2.0.1 certifi 2025.1.31 charset-normalizer 3.4.1 colorama 0.4.6 deepseek_vl2 1.0.0 einops 0.8.1 filelock 3.18.0 fsspec 2025.3.0 huggingface-hub 0.29.3 idna 3.10 intel-openmp 2021.4.0 Jinja2 3.1.6 MarkupSafe 3.0.2 mkl 2021.4.0 mpmath 1.3.0 networkx 3.4.2 numpy 1.26.4 packaging 24.2 pillow 11.1.0 pip 22.0.4 psutil 7.0.0 regex 2024.11.6 requests 2.32.3 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 58.1.0 six 1.17.0 sympy 1.13.1 tbb 2021.13.1 timm 1.0.15 tokenizers 0.15.2 torch 2.0.1+cu118 torchvision 0.15.2 tqdm 4.67.1 transformers 4.38.2 typing_extensions 4.12.2 urllib3 2.3.0 xformers 0.0.22 ```

bsakaguchi referenced this issue

2026-10-26 21:12:31 +00:00

[PR #32] [CLOSED] Added an installation bash script, fixed 2 minor errors preventing the web_demo.py from running #132

Rows
Columns

[GH-ISSUE #41] Does not work on CPU #32