[GH-ISSUE #130] pure-text data format #107

New issue

Open

opened 2026-10-26 21:12:06 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:12:06 +00:00

Owner

Originally created by @hongjian on GitHub (Apr 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/130

I'm curious about the data format of pure-text used in pre-training in DeepSeek-VL2 but I haven't found specific details.
In LLM pre-training, the loss is typically computed over all tokens. However, in traditional MLLM pre-training, the data format often resembles a multi-turn QA setup, and only the answer tokens are used for loss computation — the system prompt and question are usually masked.
So, what approach does DeepSeek-VL2 use?

Originally created by @hongjian on GitHub (Apr 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/130 I'm curious about the data format of pure-text used in pre-training in DeepSeek-VL2 but I haven't found specific details. In LLM pre-training, the loss is typically computed over all tokens. However, in traditional MLLM pre-training, the data format often resembles a multi-turn QA setup, and only the answer tokens are used for loss computation — the system prompt and question are usually masked. So, what approach does DeepSeek-VL2 use?

bsakaguchi commented

2026-10-26 21:12:07 +00:00

Author

Owner

@zhaozhipeng1997 commented on GitHub (Apr 10, 2025):

       conversation = [
        {
            "role": "<|User|>",
            "content": "\n<|ref|>" + sample['question'] + "<|/ref|>.",
            "images": []
            
        },
        {
            "role": "<|Assistant|>",
            "content": ""
        }
        ]
        prepare_inputs = vl_chat_processor(
                conversations=conversation,
                images=[],
                force_batchify=True,
                system_prompt="",
            ).to(dtype=torch.bfloat16,device=vl_gpt.device)
        inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

@zhaozhipeng1997 commented on GitHub (Apr 10, 2025): conversation = [ { "role": "<|User|>", "content": "\n<|ref|>" + sample['question'] + "<|/ref|>.", "images": [] }, { "role": "<|Assistant|>", "content": "" } ] prepare_inputs = vl_chat_processor( conversations=conversation, images=[], force_batchify=True, system_prompt="", ).to(dtype=torch.bfloat16,device=vl_gpt.device) inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

bsakaguchi referenced this issue

2026-10-26 21:12:39 +00:00

[PR #107] Pytorch version update #145