[GH-ISSUE #4 ] Cannot inference model #1

Author

Owner

@ApolloRay commented on GitHub (Dec 16, 2024):

cos = cos[position_ids].unsqueeze(unsqueeze_dim)

IndexError: index is out of bounds for dimension with size 0

same problem

@ApolloRay commented on GitHub (Dec 16, 2024): cos = cos[position_ids].unsqueeze(unsqueeze_dim) IndexError: index is out of bounds for dimension with size 0 same problem

bsakaguchi commented

Author

Owner

@williamium3000 commented on GitHub (Dec 16, 2024):

@learningpro I use the following env

I cannot use the 12.4 cuda since my cuda is only 12.3 (using 12.4 cuda torch gives me

undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12

problem.
However, I believe the cuda is not a major issue here.

@williamium3000 commented on GitHub (Dec 16, 2024): @learningpro I use the following env ![image](https://github.com/user-attachments/assets/a2523dd9-8d68-4b43-9d86-eaacda42f84a) I cannot use the 12.4 cuda since my cuda is only 12.3 (using 12.4 cuda torch gives me ``` undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12 ``` problem. However, I believe the cuda is not a major issue here.

bsakaguchi commented

Author

Owner

@ApolloRay commented on GitHub (Dec 16, 2024):

fix it.
you can add input_ids = prepare_inputs["input_ids"], at line 106 in inference.py
截屏2024-12-16 20 00 35

@ApolloRay commented on GitHub (Dec 16, 2024): fix it. you can add input_ids = prepare_inputs["input_ids"], at line 106 in inference.py <img width="415" alt="截屏2024-12-16 20 00 35" src="https://github.com/user-attachments/assets/5dba04fb-fb17-4909-b122-3cfa79f0cd2a" />

bsakaguchi commented

Author

Owner

@robinren03 commented on GitHub (Dec 16, 2024):

fix it. you can add input_ids = prepare_inputs["input_ids"], at line 106 in inference.py

Thank you! It works for me.

@robinren03 commented on GitHub (Dec 16, 2024): > fix it. you can add input_ids = prepare_inputs["input_ids"], at line 106 in inference.py <img alt="截屏2024-12-16 20 00 35" width="415" src="https://private-user-images.githubusercontent.com/36093263/396086643-5dba04fb-fb17-4909-b122-3cfa79f0cd2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQzNTkyNjYsIm5iZiI6MTczNDM1ODk2NiwicGF0aCI6Ii8zNjA5MzI2My8zOTYwODY2NDMtNWRiYTA0ZmItZmIxNy00OTA5LWIxMjItM2NmYTc5ZjBjZDJhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjE2VDE0MjI0NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQwZDQ3Y2I1NDc3ZjE4ZTQ4MDk1NWMyZTFlYTNkZDllNzAxYTRkYWYzMTZlZDY2MTZiNmIxMWQ4YTMxNGZjNzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.xCahjC3u9Pv28TBVe4gfh--LSTZ3UEEz_5nlyRDYL1A"> Thank you! It works for me.

bsakaguchi commented

Author

Owner

@Adenialzz commented on GitHub (Dec 17, 2024):

What is the expected output of the demo about two giraffes images ?

@Adenialzz commented on GitHub (Dec 17, 2024): What is the expected output of the demo about two giraffes images ?

bsakaguchi commented

Author

Owner

@HubHop commented on GitHub (Dec 18, 2024):

Hi guys, if you're encountering the issue mentioned, please try running pip install transformers==4.38.2.

File "/william/repo/DeepSeek-VL2/deepseek_vl/models/modeling_deepseek.py", line 366, in apply_rotary_pos_emb
cos = cos[position_ids].unsqueeze(unsqueeze_dim)
IndexError: index is out of bounds for dimension with size 0

@HubHop commented on GitHub (Dec 18, 2024): Hi guys, if you're encountering the issue mentioned, please try running `pip install transformers==4.38.2`. > File "/william/repo/DeepSeek-VL2/deepseek_vl/models/modeling_deepseek.py", line 366, in apply_rotary_pos_emb > cos = cos[position_ids].unsqueeze(unsqueeze_dim) > IndexError: index is out of bounds for dimension with size 0

bsakaguchi commented

Author

Owner

@HubHop commented on GitHub (Dec 18, 2024):

@Adenialzz Here is one example from DeepSeek-VL2-tiny:

Input:
<|User|>:
<|ref|>The giraffe at the back.<|/ref|>.

Expected output:
<|Assistant|>: <|ref|>The giraffe at the back.<|/ref|><|det|>582, 271, 942, 888<|/det|><｜end▁of▁sentence｜>

Please make sure you have set skip_special_tokens=False.

@HubHop commented on GitHub (Dec 18, 2024): @Adenialzz Here is one example from DeepSeek-VL2-tiny: **Input:** <|User|>: <image> <|ref|>The giraffe at the back.<|/ref|>. **Expected output:** <|Assistant|>: <|ref|>The giraffe at the back.<|/ref|><|det|>[[582, 271, 942, 888]]<|/det|><｜end▁of▁sentence｜> Please make sure you have set `skip_special_tokens=False`.

bsakaguchi commented

Author

Owner

@williamium3000 commented on GitHub (Dec 18, 2024):

@HubHop Can you give an example of interleaved image text? The case in the hugging face won't work. Thanks!

@williamium3000 commented on GitHub (Dec 18, 2024): @HubHop Can you give an example of interleaved image text? The case in the hugging face won't work. Thanks!

bsakaguchi commented

Author

Owner

@HubHop commented on GitHub (Dec 18, 2024):

@williamium3000 Sure! Below is one example of interleaved image-texts. No other changes required in the demo from the readme.

conversation = [
    {
        "role": "<|User|>",
        "content": "This is image_1: <image>\n"
                   "This is image_2: <image>\n"
                   "This is image_3: <image>\n If I am a vegetarian, what can I cook with these ingredients?",
        "images": [
            "images/multi_image_1.png",
            "images/multi_image_2.jpg",
            "images/multi_image_3.jpg",
        ],
    },
    {"role": "<|Assistant|>", "content": ""}
]

Expected output:

<|User|>: This is image_1: <image>
This is image_2: <image>
This is image_3: <image>
 If I am a vegetarian, what can I cook with these ingredients?

<|Assistant|>: Vegetarian dishes such as stir-fries, soups, stews, or roasted vegetables could be made using carrots and corn.<｜end▁of▁sentence｜>

Example images:

@HubHop commented on GitHub (Dec 18, 2024): @williamium3000 Sure! Below is one example of interleaved image-texts. No other changes required in the demo from the readme. ```python conversation = [ { "role": "<|User|>", "content": "This is image_1: <image>\n" "This is image_2: <image>\n" "This is image_3: <image>\n If I am a vegetarian, what can I cook with these ingredients?", "images": [ "images/multi_image_1.png", "images/multi_image_2.jpg", "images/multi_image_3.jpg", ], }, {"role": "<|Assistant|>", "content": ""} ] ``` **Expected output:** ``` <|User|>: This is image_1: <image> This is image_2: <image> This is image_3: <image> If I am a vegetarian, what can I cook with these ingredients? <|Assistant|>: Vegetarian dishes such as stir-fries, soups, stews, or roasted vegetables could be made using carrots and corn.<｜end▁of▁sentence｜> ``` **Example images:** <img src="https://github.com/user-attachments/assets/b4e7f85e-11f4-4b32-9dec-0da7342f07c3" alt="multi_image_1" width="200" /> <img src="https://github.com/user-attachments/assets/696af725-1884-4c36-88d8-900ed2831f38" alt="multi_image_2" width="200" /> <img src="https://github.com/user-attachments/assets/29657e83-5571-4470-bc54-f7f04219653e" alt="multi_image_3" width="200" />

bsakaguchi commented

Author

Owner

@williamium3000 commented on GitHub (Dec 18, 2024):

thanks

@williamium3000 commented on GitHub (Dec 18, 2024): thanks

bsakaguchi commented

Author

Owner

@williamium3000 commented on GitHub (Dec 20, 2024):

Hi, can you please provide a multi-GPU inference (splitting the model to multiple GPUs) code as the standard deepseek-vl2 is too large to be handled on one single GPU?

@williamium3000 commented on GitHub (Dec 20, 2024): Hi, can you please provide a multi-GPU inference (splitting the model to multiple GPUs) code as the standard deepseek-vl2 is too large to be handled on one single GPU?

bsakaguchi commented

Author

Owner

@williamium3000 commented on GitHub (Dec 22, 2024):

any clues? @HubHop

@williamium3000 commented on GitHub (Dec 22, 2024): any clues? @HubHop

bsakaguchi commented

Author

Owner

@HubHop commented on GitHub (Dec 22, 2024):

Hi @williamium3000, thanks for mentioning this feature. Due to many schedules, please kindly wait. We will consider adding this support later.

@HubHop commented on GitHub (Dec 22, 2024): Hi @williamium3000, thanks for mentioning this feature. Due to many schedules, please kindly wait. We will consider adding this support later.

bsakaguchi commented

Author

Owner

@HubHop commented on GitHub (Dec 23, 2024):

Hi @williamium3000, please let me know if this works for you: https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2558914207

@HubHop commented on GitHub (Dec 23, 2024): Hi @williamium3000, please let me know if this works for you: https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2558914207

bsakaguchi commented