deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #82] Problem with def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1): (index out of bounds error) #69

New issue
Open
opened 2026-10-26 21:11:19 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:11:19 +00:00
Owner
Copy link

Originally created by @ukaprch on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/82

Running model for an image. Note: I am using a quantized (qint8) version of the model.

> deepseek_vl2.models.modeling_deepseek.apply_rotary_pos_emb : 360 Python deepseek_vl2.models.modeling_deepseek.forward : 886 (Current frame) Python torch.nn.modules.module._call_impl : 1750 Python torch.nn.modules.module._wrapped_call_impl : 1739 Python deepseek_vl2.models.modeling_deepseek.forward : 1298 Python torch.nn.modules.module._call_impl : 1750 Python torch.nn.modules.module._wrapped_call_impl : 1739 Python deepseek_vl2.models.modeling_deepseek.forward : 1585 Python torch.nn.modules.module._call_impl : 1750 Python
When calling def apply_rotary_pos_emb the COS and SIN tensors are empty resulting in an error:

Image

So deepseek changed the LLama original code. These (buffers) apparently are not initialized in this model. How do we get around this error?

After using the chunk_size=512 # prefilling size the error above went away producing a new error:
Then I got a new error message trying to update the version on a torch tensor. 1st problem: Why in inference mode would the model be updating a torch tensor version? The 2nd problem has to do with Quanto not handling the problem internally as the tensor in question was quantized.

Here's the relevant code in def forward (modeling_deepseek.py; line number 898) where I had to make a change.

    qclone = self.kv_b_proj.weight.detach().clone()     << insert this new code to bypass version update error on quantized tensor
    qclone = qclone.to(dtype=self.kv_b_proj.weight.dtype,device=self.kv_b_proj.weight.device) << insert as above
    #kv_b_proj = self.kv_b_proj.weight.view(self.num_heads, -1, self.kv_lora_rank)   << produces version error on q tensor
    kv_b_proj = qclone.view(self.num_heads, -1, self.kv_lora_rank)            << refer to new cloned tensor
Originally created by @ukaprch on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/82 Running model for an image. Note: I am using a quantized (qint8) version of the model. `> deepseek_vl2.models.modeling_deepseek.apply_rotary_pos_emb : 360 Python deepseek_vl2.models.modeling_deepseek.forward : 886 (Current frame) Python torch.nn.modules.module._call_impl : 1750 Python torch.nn.modules.module._wrapped_call_impl : 1739 Python deepseek_vl2.models.modeling_deepseek.forward : 1298 Python torch.nn.modules.module._call_impl : 1750 Python torch.nn.modules.module._wrapped_call_impl : 1739 Python deepseek_vl2.models.modeling_deepseek.forward : 1585 Python torch.nn.modules.module._call_impl : 1750 Python ` When calling def apply_rotary_pos_emb the COS and SIN tensors are empty resulting in an error: ![Image](https://github.com/user-attachments/assets/4ba90070-1978-4489-806c-75784341c7c5) So deepseek changed the LLama original code. These (buffers) apparently are not initialized in this model. How do we get around this error? After using the chunk_size=512 # prefilling size the error above went away producing a new error: Then I got a new error message trying to update the version on a torch tensor. 1st problem: Why in inference mode would the model be updating a torch tensor version? The 2nd problem has to do with Quanto not handling the problem internally as the tensor in question was quantized. Here's the relevant code in def forward (modeling_deepseek.py; line number 898) where I had to make a change. qclone = self.kv_b_proj.weight.detach().clone() << insert this new code to bypass version update error on quantized tensor qclone = qclone.to(dtype=self.kv_b_proj.weight.dtype,device=self.kv_b_proj.weight.device) << insert as above #kv_b_proj = self.kv_b_proj.weight.view(self.num_heads, -1, self.kv_lora_rank) << produces version error on q tensor kv_b_proj = qclone.view(self.num_heads, -1, self.kv_lora_rank) << refer to new cloned tensor
bsakaguchi commented 2026-10-26 21:11:21 +00:00
Author
Owner
Copy link

@MirkoDeVita98 commented on GitHub (Feb 25, 2025):

I have the same error, were you able to solve it?

EDIT: I found a way around it. I am now running the model using incremental prefilling that can be found in inference.py + the split model found here + the fix for multi parallelism

<!-- gh-comment-id:2682835117 --> @MirkoDeVita98 commented on GitHub (Feb 25, 2025): I have the same error, were you able to solve it? EDIT: I found a way around it. I am now running the model using incremental prefilling that can be found in inference.py + the split model found [here](https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2558914207) + [the fix for multi parallelism](https://github.com/deepseek-ai/DeepSeek-VL2/pull/91/commits/8ed2d7ff8d6bffc25342f578604c1ae358b90e23)
bsakaguchi commented 2026-10-26 21:11:21 +00:00
Author
Owner
Copy link

@ukaprch commented on GitHub (Feb 26, 2025):

Initially is used the prefill method they proposed:

# incremental_prefilling when using 40G GPU for vl2-small inputs_embeds, past_key_values = vl_gpt.incremental_prefilling( input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, attention_mask=prepare_inputs.attention_mask, chunk_size=512 # prefilling size )

Then I think instead of the above you can set this parameter:
past_key_values=None,

before you generate and thus, you may not need the prefill step.

In any event, I kept running into problems with vl2-small and opted to
just use the vl2-tiny which interestingly enough worked w/o a hitch.

<!-- gh-comment-id:2684982873 --> @ukaprch commented on GitHub (Feb 26, 2025): Initially is used the prefill method they proposed: ` # incremental_prefilling when using 40G GPU for vl2-small inputs_embeds, past_key_values = vl_gpt.incremental_prefilling( input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, attention_mask=prepare_inputs.attention_mask, chunk_size=512 # prefilling size )` Then I think instead of the above you can set this parameter: past_key_values=None, before you generate and thus, you may not need the prefill step. In any event, I kept running into problems with vl2-small and opted to just use the vl2-tiny which interestingly enough worked w/o a hitch.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#69
No description provided.