deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #100] Visual gronding Bbox Shift #82

New issue
Open
opened 2026-10-26 21:11:35 +00:00 by bsakaguchi · 11 comments
bsakaguchi commented 2026-10-26 21:11:35 +00:00
Owner
Copy link

Originally created by @maryhh on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/100

this is the output ,i used vllm to deploy deepseek-vl2-small

Image

this is the output ,i used gradio demo you provided，

Image

the picture is 1920x1080 ,i also used pixels < 1000x1000 to test .Both results have the same problems.

how to slove it?

Originally created by @maryhh on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/100 this is the output ,i used vllm to deploy deepseek-vl2-small ![Image](https://github.com/user-attachments/assets/8eeccef5-deba-4105-b460-b0aecdb69447) this is the output ,i used gradio demo you provided， ![Image](https://github.com/user-attachments/assets/cc668884-6eff-45be-bba8-544a6a5574f1) the picture is 1920x1080 ,i also used pixels < 1000x1000 to test .Both results have the same problems. how to slove it?
bsakaguchi commented 2026-10-26 21:11:36 +00:00
Author
Owner
Copy link

@adrianJW421 commented on GitHub (Feb 25, 2025):

I also encounter similar problem:

my prompt is : <|grounding|>Find the cars in the scene.

my prediction is : <|Assistant|>: <|ref|>mazdaughters<|/ref|><|det|>220, <<|/det|><｜end▁of▁sentence｜>

after rescaled to original image size and visualize the bbox, the result is:

Image

seems the grounding result is not very accurate

<!-- gh-comment-id:2681514932 --> @adrianJW421 commented on GitHub (Feb 25, 2025): I also encounter similar problem: my prompt is : <|grounding|>Find the cars in the scene. my prediction is : <|Assistant|>: <|ref|>mazdaughters<|/ref|><|det|>[[220, <|ref|>compact car<|/ref|><|det|>[[220, 416, 417, 720]]<|/det|><｜end▁of▁sentence｜> after rescaled to original image size and visualize the bbox, the result is: <img width="772" alt="Image" src="https://github.com/user-attachments/assets/3029c8be-4f0c-4a49-b0b7-a79f88701985" /> seems the grounding result is not very accurate
bsakaguchi commented 2026-10-26 21:11:37 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Feb 26, 2025):

@maryhh @adrianJW421
Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions.

box = (
    int(box[0] / 999 * image_w), 
    int(box[1] / 999 * image_h), 
    int(box[2] / 999 * image_w), 
    int(box[3] / 999 * image_h), 
)

For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning.

<!-- gh-comment-id:2683888219 --> @charlesCXK commented on GitHub (Feb 26, 2025): @maryhh @adrianJW421 Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions. ``` box = ( int(box[0] / 999 * image_w), int(box[1] / 999 * image_h), int(box[2] / 999 * image_w), int(box[3] / 999 * image_h), ) ``` For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning.
bsakaguchi commented 2026-10-26 21:11:37 +00:00
Author
Owner
Copy link

@maryhh commented on GitHub (Feb 26, 2025):

@maryhh @adrianJW421 Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions.

box = (
    int(box[0] / 999 * image_w), 
    int(box[1] / 999 * image_h), 
    int(box[2] / 999 * image_w), 
    int(box[3] / 999 * image_h), 
)

For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning.

I have already done this operation and i used the prompt 找到<|ref|>明火<|/ref|> . By the way ,the gradio demo you provided also has the bbox shift problem.

<!-- gh-comment-id:2684026220 --> @maryhh commented on GitHub (Feb 26, 2025): > [@maryhh](https://github.com/maryhh) [@adrianJW421](https://github.com/adrianJW421) Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions. > > ``` > box = ( > int(box[0] / 999 * image_w), > int(box[1] / 999 * image_h), > int(box[2] / 999 * image_w), > int(box[3] / 999 * image_h), > ) > ``` > > For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning. I have already done this operation and i used the prompt 找到<|ref|>明火<|/ref|> . By the way ,the gradio demo you provided also has the bbox shift problem.
bsakaguchi commented 2026-10-26 21:11:38 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Feb 27, 2025):

@maryhh

I test the case you provide in the gradio, it works correctly.

Image

<!-- gh-comment-id:2686767908 --> @charlesCXK commented on GitHub (Feb 27, 2025): @maryhh I test the case you provide in the gradio, it works correctly. ![Image](https://github.com/user-attachments/assets/e81644d1-ca75-45f0-a152-72634c1e40be)
bsakaguchi commented 2026-10-26 21:11:38 +00:00
Author
Owner
Copy link

@xinlin-xiao commented on GitHub (Feb 27, 2025):

@adrianJW421 @maryhh @charlesCXK Hello，"I also use vLLM to deploy deepseek-vl2 and encounter the same bounding box misalignment issue. I have already used the following code to adjust the box:

box = (
    int(box[0] / 999 * image_w),
    int(box[1] / 999 * image_h),
    int(box[2] / 999 * image_w),
    int(box[3] / 999 * image_h),
)

This is my inference test code:

import argparse
import gradio as gr
from openai import OpenAI
import base64
import io 
import os
import time
import re
from PIL import Image, ImageDraw, ImageFont
from io import BytesIO
# 设置命令行参数解析
parser = argparse.ArgumentParser(
    description='带有可自定义参数的聊天机器人接口')
parser.add_argument('--model-url',
                    type=str,
                    default='http://localhost:8000/v1',
                    help='模型的URL地址')
parser.add_argument('-m',
                    '--model',
                    type=str,
                    required=True,
                    help='聊天机器人使用的模型名称')
parser.add_argument('--temp',
                    type=float,
                    default=0.1,
                    help='文本生成的温度参数')
parser.add_argument('--stop-token-ids',
                    type=str,
                    default='',
                    help='逗号分隔的停止标记ID')
parser.add_argument("--host", type=str, default='192.168.3.242', help="主机地址")
parser.add_argument("--port", type=int, default=8001)

# 新增一个控制是否使用历史记录的参数
parser.add_argument('--use-history',
                    action='store_true',
                    help='如果设置该参数，表示使用历史记录进行生成，否则仅使用当前输入')

# 添加一个控制是否处理多模态数据的参数，去掉了视频选项
parser.add_argument('--modality', type=str, default='text', choices=['text', 'image'],
                    help='指定输入的模态类型：文本或图像')

# 解析命令行参数
args = parser.parse_args()

# 设置OpenAI的API密钥和API地址，使用vLLM的API服务器
openai_api_key = "token-abc123"
openai_api_base = args.model_url

# 创建一个OpenAI客户端来与API服务器交互
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 定义图像转 Base64 的函数
def image_to_base64(image):
    """
    将PIL图像转换为Base64编码字符串
    """
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")  # 可以根据需要保存为其他格式
    return base64.b64encode(buffered.getvalue()).decode('utf-8')
# 定义颜色映射
BOX2COLOR = {
    0: (255, 0, 0),
    1: (0, 255, 0),
    2: (0, 0, 255),
    3: (0, 255, 255),
    4: (255, 255, 0),
    5: (255, 0, 255),
    6: (127, 127, 127),
    7: (255, 255, 127),
    8: (255, 127, 255),
    9: (127, 255, 255),
    10: (127, 127, 255),
    11: (127, 255, 127),
    12: (255, 127, 127),
}

# 将 Base64 编码的图像字符串转换为 PIL 图像
def base64_to_pil(image_base64: str) -> Image.Image:
    # 解码 Base64 字符串
    image_data = base64.b64decode(image_base64)
    # 将字节数据转换为 PIL 图像
    image = Image.open(BytesIO(image_data))
    return image
def pil_to_base64(
    image: Image.Image,
    alt: str = "user upload image",
    resize: bool = True,
    max_size: int = 800,
    min_size: int = 400,
    format: str = "JPEG",
    quality: int = 95
) -> str:
    if resize:
        max_hw, min_hw = max(image.size), min(image.size)
        aspect_ratio = max_hw / min_hw
        shortest_edge = int(min(max_size / aspect_ratio, min_size, min_hw))
        longest_edge = int(shortest_edge * aspect_ratio)
        W, H = image.size
        if H > W:
            H, W = longest_edge, shortest_edge
        else:
            H, W = shortest_edge, longest_edge
        image = image.resize((W, H))

    buffered = io.BytesIO()
    image.save(buffered, format=format, quality=quality)
    img_b64_str = base64.b64encode(buffered.getvalue()).decode()
    # img_str = f'<img src="data:image/png;base64,{img_b64_str}" alt="{alt}" />'

    return img_b64_str


def parse_ref_bbox(response, image: Image.Image, save_path="output_image_with_bboxes.jpg"):
    """
    解析API响应中的边界框信息，并绘制框在图像上。
    
    参数:
    - response: 从模型获取的响应，包含边界框和标签信息。
    - image: 原始图像对象，将在此图像上绘制边界框。
    - save_path: 保存图像的路径。

    返回:
    - 绘制了边界框的图像。
    """
    try:
        # 创建图像副本，避免修改原始图像
        image = image.copy()
        image_h, image_w = image.size
        print(f"Image size: {image_h}x{image_w}")
        draw = ImageDraw.Draw(image)

        # 使用正则提取标签和边界框
        ref = re.findall(r'<\|ref\|>.*?<\|/ref\|>', response)
        # print(f"ref:{ref}")
        bbox = re.findall(r'<\|det\|>.*?<\|/det\|>', response)
        # print(f"bbox:{bbox}")
        
        
        # 如果标签和边界框的数量不一致，报错
        assert len(ref) == len(bbox), f"Number of labels ({len(ref)}) does not match number of boxes ({len(bbox)})"
        
        if len(ref) == 0:
            print("No bounding boxes or labels found.")
            return None

        boxes, labels = [], []
        # 遍历所有提取到的边界框和标签
        for box, label in zip(bbox, ref):
            box = box.replace('<|det|>', '').replace('<|/det|>', '')
            label = label.replace('<|ref|>', '').replace('<|/ref|>', '')
            box = box[1:-1]  # 去除最外层的方括号
            print(f"box:{box}")
            for onebox in re.findall(r'\[.*?\]', box):
                boxes.append(eval(onebox))  # 解析为坐标
                # print(f"box:{boxes}")
                labels.append(label)
                # print(f"label:{labels}")

        # 绘制边界框和标签
        for indice, (box, label) in enumerate(zip(boxes, labels)):
            # 将归一化的坐标映射回图像大小
            box = (
                int(box[0] / 999 * image_w),
                int(box[1] / 999 * image_h),
                int(box[2] / 999 * image_w),
                int(box[3] / 999 * image_h),
            )
            print(f"normalized box:{box}")

            # 获取框的颜色，确保每个框有不同的颜色
            box_color = BOX2COLOR[indice % len(BOX2COLOR)]  # 使用 modulo 来循环不同的颜色
            box_width = 3
            # 绘制矩形框
            draw.rectangle(box, outline=box_color, width=box_width)

            # 绘制标签文本
            text_x = box[0]
            text_y = box[1] - 20  # 将文本放置在框上方
            text_color = box_color
            font = ImageFont.truetype("deepseek_vl2/serve/assets/simsun.ttc", size=20)  # 指定字体
            draw.text((text_x, text_y), label, font=font, fill=text_color)

        # 生成唯一的文件名，可以使用时间戳或者图像编号
        output_dir = "output_image"
        os.makedirs(output_dir, exist_ok=True)  # 确保输出目录存在
        timestamp = int(time.time() * 1000)  # 时间戳（毫秒）
        save_path = os.path.join(output_dir, f"output_image_{timestamp}.jpg")

        # 保存图像到指定路径
        image.save(save_path)
        print(f"Image with bounding boxes saved at: {save_path}")

        return image  # 返回带有边界框的图像
    except Exception as e:
        print(f"Error: {e}")
        return None
# 定义预测函数
def predict(message, history, image=None, prompt_type='image_question'):
    """
    用于预测生成模型的响应，支持历史记录和多模态输入。
    参数:
    - message: 当前用户输入的消息
    - history: 历史记录
    - image: 传入的图像，只有在modality为'image'时有效
    - prompt_type: 控制使用哪种prompt格式，'image_question' 或 'ref_question'
    """
    # 如果启用了历史记录（use_history为True），则包含历史记录
    if args.use_history:
        history_openai_format = [{
            "role": "system",  # 系统消息
            "content": "你是一个非常优秀的AI助手。"
        }]
        # 遍历历史记录，将每一轮对话添加到消息列表中
        for human, assistant in history:
            history_openai_format.append({"role": "user", "content": human})  # 用户消息
            history_openai_format.append({"role": "assistant", "content": assistant})  # AI助手回复
        history_openai_format.append({"role": "user", "content": message})  # 当前用户消息
    else:
        # 如果不使用历史记录，只传入当前用户的输入
        history_openai_format = [ ]
    
    # 根据输入的模态（text或image）调整生成的prompt
    if args.modality == 'image' and image:
        
        # 将图像转换为 Base64 编码
        image_base64 = image_to_base64(image)
        
        # 使用 pil_to_base64 函数将图像转换为 Base64 编码
        # image_base64 = pil_to_base64(image, alt="user uploaded image", max_size=800, min_size=400)
        if prompt_type == 'image_question':
            # 图像输入，生成适合图像问题的prompt
            img_question = f"<image>\n{message}"
        elif prompt_type == 'ref_question':
            # 图像输入，生成带有 <|ref|> 标签的 prompt
            img_question = f"<image>\n<|ref|>{message}.<|/ref|>"
        
        history_openai_format.append({"role": "user", "content":[
                {
                    "type": "text",
                    "text": img_question
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_base64}"
                        # "url": image_base64
                    },
                },
            ],
        })
    
    # 创建聊天生成请求，并发送给API服务器
    stream = client.chat.completions.create(
        model=args.model,  # 使用的模型名称
        messages=history_openai_format,  # 聊天记录或当前消息
        temperature=args.temp,  # 文本生成的温度
        stream=True,  # 开启流式响应
        extra_body={
            'repetition_penalty': 1,  # 重复惩罚
            'stop_token_ids': [
                int(id.strip()) for id in args.stop_token_ids.split(',') if id.strip()
            ] if args.stop_token_ids else []  # 停止标记ID（如果有的话）
        })

    # 从响应流中读取生成的文本
    partial_message = ""
    # print(f"stream:{stream}")
    for chunk in stream:
        # print(f"Chunk: {chunk}")
        partial_message += (chunk.choices[0].delta.content or "")
    print(f"Partial message: {partial_message}")
    
    
    # 解析并绘制边界框
    # 将 Base64 编码的图像转换为 PIL 图像
    # pil_image = base64_to_pil(image_base64)
    output_image = parse_ref_bbox(partial_message, image)
    
    if output_image:
        # 如果图像处理成功，返回文本和图像
        return partial_message, output_image
    
    # 如果没有图像处理，返回文本
    return partial_message, None





def main():
    with gr.Blocks() as demo:
        # 创建输入组件
        user_input = gr.Textbox(label="User Input", placeholder="输入问题...", lines=2)  # 用户输入的文本框
        history = gr.State()  # 用于存储历史记录的Gradio状态（只作为输入）
        image_input = gr.Image(type="pil", label="Upload Image")  # 上传图像的功能
        prompt_type_input = gr.Radio(choices=["image_question", "ref_question"], label="Prompt Type", value="image_question")  # 选择Prompt类型
        
        # 输出组件：使用Image而不是Textbox来显示图像
        output_image = gr.Image(label="Output Image with Bounding Boxes")  # 输出图像框
        output_text = gr.Textbox(label="Generated Text Output")  # 输出文本框

        # 创建一个按钮并绑定点击事件，点击时触发预测
        button = gr.Button("Send")
        button.click(predict, 
                     inputs=[user_input, history, image_input, prompt_type_input],
                     outputs=[output_text, output_image])  # 返回文本和图像

    # 启动Gradio界面
    demo.launch(server_name=args.host, server_port=args.port, share=True)

if __name__ == "__main__":
    main()

run：
python web-vllm-demo.py -m /mnt/data1/download_new/DeepSeek-VL2/model/deepseek-ai/deepseek-vl2 --modality image
result:
Image

Do you have a solution for this box offset issue?"

<!-- gh-comment-id:2687651392 --> @xinlin-xiao commented on GitHub (Feb 27, 2025): @adrianJW421 @maryhh @charlesCXK Hello，"I also use vLLM to deploy deepseek-vl2 and encounter the same bounding box misalignment issue. I have already used the following code to adjust the box: ```python box = ( int(box[0] / 999 * image_w), int(box[1] / 999 * image_h), int(box[2] / 999 * image_w), int(box[3] / 999 * image_h), ) ``` This is my inference test code: ```python import argparse import gradio as gr from openai import OpenAI import base64 import io import os import time import re from PIL import Image, ImageDraw, ImageFont from io import BytesIO # 设置命令行参数解析 parser = argparse.ArgumentParser( description='带有可自定义参数的聊天机器人接口') parser.add_argument('--model-url', type=str, default='http://localhost:8000/v1', help='模型的URL地址') parser.add_argument('-m', '--model', type=str, required=True, help='聊天机器人使用的模型名称') parser.add_argument('--temp', type=float, default=0.1, help='文本生成的温度参数') parser.add_argument('--stop-token-ids', type=str, default='', help='逗号分隔的停止标记ID') parser.add_argument("--host", type=str, default='192.168.3.242', help="主机地址") parser.add_argument("--port", type=int, default=8001) # 新增一个控制是否使用历史记录的参数 parser.add_argument('--use-history', action='store_true', help='如果设置该参数，表示使用历史记录进行生成，否则仅使用当前输入') # 添加一个控制是否处理多模态数据的参数，去掉了视频选项 parser.add_argument('--modality', type=str, default='text', choices=['text', 'image'], help='指定输入的模态类型：文本或图像') # 解析命令行参数 args = parser.parse_args() # 设置OpenAI的API密钥和API地址，使用vLLM的API服务器 openai_api_key = "token-abc123" openai_api_base = args.model_url # 创建一个OpenAI客户端来与API服务器交互 client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) # 定义图像转 Base64 的函数 def image_to_base64(image): """ 将PIL图像转换为Base64编码字符串 """ buffered = io.BytesIO() image.save(buffered, format="JPEG") # 可以根据需要保存为其他格式 return base64.b64encode(buffered.getvalue()).decode('utf-8') # 定义颜色映射 BOX2COLOR = { 0: (255, 0, 0), 1: (0, 255, 0), 2: (0, 0, 255), 3: (0, 255, 255), 4: (255, 255, 0), 5: (255, 0, 255), 6: (127, 127, 127), 7: (255, 255, 127), 8: (255, 127, 255), 9: (127, 255, 255), 10: (127, 127, 255), 11: (127, 255, 127), 12: (255, 127, 127), } # 将 Base64 编码的图像字符串转换为 PIL 图像 def base64_to_pil(image_base64: str) -> Image.Image: # 解码 Base64 字符串 image_data = base64.b64decode(image_base64) # 将字节数据转换为 PIL 图像 image = Image.open(BytesIO(image_data)) return image def pil_to_base64( image: Image.Image, alt: str = "user upload image", resize: bool = True, max_size: int = 800, min_size: int = 400, format: str = "JPEG", quality: int = 95 ) -> str: if resize: max_hw, min_hw = max(image.size), min(image.size) aspect_ratio = max_hw / min_hw shortest_edge = int(min(max_size / aspect_ratio, min_size, min_hw)) longest_edge = int(shortest_edge * aspect_ratio) W, H = image.size if H > W: H, W = longest_edge, shortest_edge else: H, W = shortest_edge, longest_edge image = image.resize((W, H)) buffered = io.BytesIO() image.save(buffered, format=format, quality=quality) img_b64_str = base64.b64encode(buffered.getvalue()).decode() # img_str = f'<img src="data:image/png;base64,{img_b64_str}" alt="{alt}" />' return img_b64_str def parse_ref_bbox(response, image: Image.Image, save_path="output_image_with_bboxes.jpg"): """ 解析API响应中的边界框信息，并绘制框在图像上。 参数: - response: 从模型获取的响应，包含边界框和标签信息。 - image: 原始图像对象，将在此图像上绘制边界框。 - save_path: 保存图像的路径。 返回: - 绘制了边界框的图像。 """ try: # 创建图像副本，避免修改原始图像 image = image.copy() image_h, image_w = image.size print(f"Image size: {image_h}x{image_w}") draw = ImageDraw.Draw(image) # 使用正则提取标签和边界框 ref = re.findall(r'<\|ref\|>.*?<\|/ref\|>', response) # print(f"ref:{ref}") bbox = re.findall(r'<\|det\|>.*?<\|/det\|>', response) # print(f"bbox:{bbox}") # 如果标签和边界框的数量不一致，报错 assert len(ref) == len(bbox), f"Number of labels ({len(ref)}) does not match number of boxes ({len(bbox)})" if len(ref) == 0: print("No bounding boxes or labels found.") return None boxes, labels = [], [] # 遍历所有提取到的边界框和标签 for box, label in zip(bbox, ref): box = box.replace('<|det|>', '').replace('<|/det|>', '') label = label.replace('<|ref|>', '').replace('<|/ref|>', '') box = box[1:-1] # 去除最外层的方括号 print(f"box:{box}") for onebox in re.findall(r'\[.*?\]', box): boxes.append(eval(onebox)) # 解析为坐标 # print(f"box:{boxes}") labels.append(label) # print(f"label:{labels}") # 绘制边界框和标签 for indice, (box, label) in enumerate(zip(boxes, labels)): # 将归一化的坐标映射回图像大小 box = ( int(box[0] / 999 * image_w), int(box[1] / 999 * image_h), int(box[2] / 999 * image_w), int(box[3] / 999 * image_h), ) print(f"normalized box:{box}") # 获取框的颜色，确保每个框有不同的颜色 box_color = BOX2COLOR[indice % len(BOX2COLOR)] # 使用 modulo 来循环不同的颜色 box_width = 3 # 绘制矩形框 draw.rectangle(box, outline=box_color, width=box_width) # 绘制标签文本 text_x = box[0] text_y = box[1] - 20 # 将文本放置在框上方 text_color = box_color font = ImageFont.truetype("deepseek_vl2/serve/assets/simsun.ttc", size=20) # 指定字体 draw.text((text_x, text_y), label, font=font, fill=text_color) # 生成唯一的文件名，可以使用时间戳或者图像编号 output_dir = "output_image" os.makedirs(output_dir, exist_ok=True) # 确保输出目录存在 timestamp = int(time.time() * 1000) # 时间戳（毫秒） save_path = os.path.join(output_dir, f"output_image_{timestamp}.jpg") # 保存图像到指定路径 image.save(save_path) print(f"Image with bounding boxes saved at: {save_path}") return image # 返回带有边界框的图像 except Exception as e: print(f"Error: {e}") return None # 定义预测函数 def predict(message, history, image=None, prompt_type='image_question'): """ 用于预测生成模型的响应，支持历史记录和多模态输入。 参数: - message: 当前用户输入的消息 - history: 历史记录 - image: 传入的图像，只有在modality为'image'时有效 - prompt_type: 控制使用哪种prompt格式，'image_question' 或 'ref_question' """ # 如果启用了历史记录（use_history为True），则包含历史记录 if args.use_history: history_openai_format = [{ "role": "system", # 系统消息 "content": "你是一个非常优秀的AI助手。" }] # 遍历历史记录，将每一轮对话添加到消息列表中 for human, assistant in history: history_openai_format.append({"role": "user", "content": human}) # 用户消息 history_openai_format.append({"role": "assistant", "content": assistant}) # AI助手回复 history_openai_format.append({"role": "user", "content": message}) # 当前用户消息 else: # 如果不使用历史记录，只传入当前用户的输入 history_openai_format = [ ] # 根据输入的模态（text或image）调整生成的prompt if args.modality == 'image' and image: # 将图像转换为 Base64 编码 image_base64 = image_to_base64(image) # 使用 pil_to_base64 函数将图像转换为 Base64 编码 # image_base64 = pil_to_base64(image, alt="user uploaded image", max_size=800, min_size=400) if prompt_type == 'image_question': # 图像输入，生成适合图像问题的prompt img_question = f"<image>\n{message}" elif prompt_type == 'ref_question': # 图像输入，生成带有 <|ref|> 标签的 prompt img_question = f"<image>\n<|ref|>{message}.<|/ref|>" history_openai_format.append({"role": "user", "content":[ { "type": "text", "text": img_question }, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{image_base64}" # "url": image_base64 }, }, ], }) # 创建聊天生成请求，并发送给API服务器 stream = client.chat.completions.create( model=args.model, # 使用的模型名称 messages=history_openai_format, # 聊天记录或当前消息 temperature=args.temp, # 文本生成的温度 stream=True, # 开启流式响应 extra_body={ 'repetition_penalty': 1, # 重复惩罚 'stop_token_ids': [ int(id.strip()) for id in args.stop_token_ids.split(',') if id.strip() ] if args.stop_token_ids else [] # 停止标记ID（如果有的话） }) # 从响应流中读取生成的文本 partial_message = "" # print(f"stream:{stream}") for chunk in stream: # print(f"Chunk: {chunk}") partial_message += (chunk.choices[0].delta.content or "") print(f"Partial message: {partial_message}") # 解析并绘制边界框 # 将 Base64 编码的图像转换为 PIL 图像 # pil_image = base64_to_pil(image_base64) output_image = parse_ref_bbox(partial_message, image) if output_image: # 如果图像处理成功，返回文本和图像 return partial_message, output_image # 如果没有图像处理，返回文本 return partial_message, None def main(): with gr.Blocks() as demo: # 创建输入组件 user_input = gr.Textbox(label="User Input", placeholder="输入问题...", lines=2) # 用户输入的文本框 history = gr.State() # 用于存储历史记录的Gradio状态（只作为输入） image_input = gr.Image(type="pil", label="Upload Image") # 上传图像的功能 prompt_type_input = gr.Radio(choices=["image_question", "ref_question"], label="Prompt Type", value="image_question") # 选择Prompt类型 # 输出组件：使用Image而不是Textbox来显示图像 output_image = gr.Image(label="Output Image with Bounding Boxes") # 输出图像框 output_text = gr.Textbox(label="Generated Text Output") # 输出文本框 # 创建一个按钮并绑定点击事件，点击时触发预测 button = gr.Button("Send") button.click(predict, inputs=[user_input, history, image_input, prompt_type_input], outputs=[output_text, output_image]) # 返回文本和图像 # 启动Gradio界面 demo.launch(server_name=args.host, server_port=args.port, share=True) if __name__ == "__main__": main() ``` run： `python web-vllm-demo.py -m /mnt/data1/download_new/DeepSeek-VL2/model/deepseek-ai/deepseek-vl2 --modality image` result: ![Image](https://github.com/user-attachments/assets/225a80cd-2e73-42e7-9faf-b54cc4c16d03) Do you have a solution for this box offset issue?"
bsakaguchi commented 2026-10-26 21:11:38 +00:00
Author
Owner
Copy link

@maryhh commented on GitHub (Feb 28, 2025):

@maryhh

I test the case you provide in the gradio, it works correctly.

Image

The first attempt was fine, but after several more tries, the offset problem occurred again.

<!-- gh-comment-id:2689895094 --> @maryhh commented on GitHub (Feb 28, 2025): > [@maryhh](https://github.com/maryhh) > > I test the case you provide in the gradio, it works correctly. > > ![Image](https://github.com/user-attachments/assets/e81644d1-ca75-45f0-a152-72634c1e40be) The first attempt was fine, but after several more tries, the offset problem occurred again.
bsakaguchi commented 2026-10-26 21:11:39 +00:00
Author
Owner
Copy link

@xinlin-xiao commented on GitHub (Mar 3, 2025):

@maryhh "A strange phenomenon. Could it be an issue with the model? Have you tried the official Gradio code? Will the offset still occur after multiple attempts?"

<!-- gh-comment-id:2693057844 --> @xinlin-xiao commented on GitHub (Mar 3, 2025): @maryhh "A strange phenomenon. Could it be an issue with the model? Have you tried the official Gradio code? Will the offset still occur after multiple attempts?"
bsakaguchi commented 2026-10-26 21:11:39 +00:00
Author
Owner
Copy link

@maryhh commented on GitHub (Mar 3, 2025):

@maryhh "A strange phenomenon. Could it be an issue with the model? Have you tried the official Gradio code? Will the offset still occur after multiple attempts?"

Yes，i used the official gradio, i have gived up.

<!-- gh-comment-id:2693331082 --> @maryhh commented on GitHub (Mar 3, 2025): > [@maryhh](https://github.com/maryhh) "A strange phenomenon. Could it be an issue with the model? Have you tried the official Gradio code? Will the offset still occur after multiple attempts?" Yes，i used the official gradio, i have gived up.
bsakaguchi commented 2026-10-26 21:11:40 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Mar 4, 2025):

@maryhh
"The first attempt was fine, but after several more tries, the offset problem occurred again."

Does it mean that it performs poorly in multi-round conversations? This is normal. The performance of large models in multi-round conversations is not as good as that in a single round.

<!-- gh-comment-id:2696463586 --> @charlesCXK commented on GitHub (Mar 4, 2025): @maryhh "_The first attempt was fine, but after several more tries, the offset problem occurred again._" Does it mean that it performs poorly in multi-round conversations? This is normal. The performance of large models in multi-round conversations is not as good as that in a single round.
bsakaguchi commented 2026-10-26 21:11:40 +00:00
Author
Owner
Copy link

@adrianJW421 commented on GitHub (Mar 4, 2025):

@maryhh @adrianJW421 Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions.

box = (
    int(box[0] / 999 * image_w), 
    int(box[1] / 999 * image_h), 
    int(box[2] / 999 * image_w), 
    int(box[3] / 999 * image_h), 
)

For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning.

Thanks for replying. I actually have done the transformation, so it was not the key problem for me. However, I found out the point that may have caused the problem. Initially I implemented the code according to the example code on huggingface page, compared to the inference.py code in this repo, the following part is different:

correct case:

with torch.no_grad():
        inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
        past_key_values = None
        # run the model to get the response
        outputs = vl_gpt.generate(
                # inputs_embeds=inputs_embeds[:, -1:],
                # input_ids=prepare_inputs.input_ids[:, -1:],
                inputs_embeds=inputs_embeds,
                input_ids=prepare_inputs.input_ids,
                images=prepare_inputs.images,
                images_seq_mask=prepare_inputs.images_seq_mask,
                images_spatial_crop=prepare_inputs.images_spatial_crop,

initial wrong case:

inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
past_key_values = None
# run the model to get the response
outputs = vl_gpt.generate(
        # inputs_embeds=inputs_embeds[:, -1:],
        # input_ids=prepare_inputs.input_ids[:, -1:],
        inputs_embeds=inputs_embeds,
        input_ids=prepare_inputs.input_ids,
        images=prepare_inputs.images,
        images_seq_mask=prepare_inputs.images_seq_mask,
        images_spatial_crop=prepare_inputs.images_spatial_crop,

Somehow although I have set vl_gpt to eval mode as in the example code, if with torch.no_grad() is missing before vl_gpt.generate, the result will be wrong.

<!-- gh-comment-id:2696814909 --> @adrianJW421 commented on GitHub (Mar 4, 2025): > [@maryhh](https://github.com/maryhh) [@adrianJW421](https://github.com/adrianJW421) Hello, the returned coordinates are relative coordinates normalized to the range [0, 999]. You need to multiply them by the original image's width and height to obtain the absolute positions. > > ``` > box = ( > int(box[0] / 999 * image_w), > int(box[1] / 999 * image_h), > int(box[2] / 999 * image_w), > int(box[3] / 999 * image_h), > ) > ``` > > For object localization, you can directly use <|ref|>something<|/ref|>. <|grounding|> is more suitable for scenarios that require a combination of object localization and reasoning. Thanks for replying. I actually have done the transformation, so it was not the key problem for me. However, I found out the point that may have caused the problem. Initially I implemented the code according to the example code on huggingface page, compared to the `inference.py` code in this repo, the following part is different: correct case: ``` with torch.no_grad(): inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) past_key_values = None # run the model to get the response outputs = vl_gpt.generate( # inputs_embeds=inputs_embeds[:, -1:], # input_ids=prepare_inputs.input_ids[:, -1:], inputs_embeds=inputs_embeds, input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, ``` initial wrong case: ``` inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) past_key_values = None # run the model to get the response outputs = vl_gpt.generate( # inputs_embeds=inputs_embeds[:, -1:], # input_ids=prepare_inputs.input_ids[:, -1:], inputs_embeds=inputs_embeds, input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, ``` Somehow although I have set vl_gpt to `eval` mode as in the example code, if `with torch.no_grad()` is missing before `vl_gpt.generate`, the result will be wrong.
bsakaguchi commented 2026-10-26 21:11:41 +00:00
Author
Owner
Copy link

@xinlin-xiao commented on GitHub (Mar 18, 2025):

@adrianJW421 hello，Have you tested vLLM inference? I still can't find the reason for the bounding box offset.

<!-- gh-comment-id:2731518802 --> @xinlin-xiao commented on GitHub (Mar 18, 2025): @adrianJW421 hello，Have you tested vLLM inference? I still can't find the reason for the bounding box offset.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#82
No description provided.