deepseek-ai/Janus
1
0
Fork 0
mirror of https://github.com/deepseek-ai/Janus.git synced 2026-11-05 00:57:18 +00:00
Code Issues 183 Projects Packages Wiki Activity

[GH-ISSUE #5] Colab Gradio demo here #6

New issue
Closed
opened 2026-10-26 21:09:48 +00:00 by bsakaguchi · 13 comments
bsakaguchi commented 2026-10-26 21:09:48 +00:00
Owner
Copy link

Originally created by @0smboy on GitHub (Oct 18, 2024).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/5

image image

First you should new file app.py in Janus dir

import gradio as gr
import torch
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images
import PIL.Image
import numpy as np
import os

# Load model and processor
model_path = "deepseek-ai/Janus-1.3B"
vl_chat_processor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer
vl_gpt = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

# Multimodal Understanding function
def multimodal_understanding(image, question):
    conversation = [
        {
            "role": "User",
            "content": f"<image_placeholder>\n{question}",
            "images": [image],
        },
        {"role": "Assistant", "content": ""},
    ]
    
    pil_images = [PIL.Image.fromarray(image)]
    prepare_inputs = vl_chat_processor(
        conversations=conversation, images=pil_images, force_batchify=True
    ).to(vl_gpt.device)
    
    inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
    
    outputs = vl_gpt.language_model.generate(
        inputs_embeds=inputs_embeds,
        attention_mask=prepare_inputs.attention_mask,
        pad_token_id=tokenizer.eos_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        max_new_tokens=512,
        do_sample=False,
        use_cache=True,
    )
    
    answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
    return answer

# Text-to-Image Generation function
@torch.inference_mode()
def text_to_image(prompt, temperature=1, parallel_size=4, cfg_weight=5):
    conversation = [
        {"role": "User", "content": prompt},
        {"role": "Assistant", "content": ""},
    ]
    sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(
        conversations=conversation,
        sft_format=vl_chat_processor.sft_format,
        system_prompt="",
    )
    full_prompt = sft_format + vl_chat_processor.image_start_tag
    
    input_ids = vl_chat_processor.tokenizer.encode(full_prompt)
    input_ids = torch.LongTensor(input_ids)
    tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()
    for i in range(parallel_size*2):
        tokens[i, :] = input_ids
        if i % 2 != 0:
            tokens[i, 1:-1] = vl_chat_processor.pad_id
    
    inputs_embeds = vl_gpt.language_model.get_input_embeddings()(tokens)
    image_token_num_per_image = 576
    img_size = 384
    patch_size = 16
    
    generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()
    
    for i in range(image_token_num_per_image):
        outputs = vl_gpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)
        hidden_states = outputs.last_hidden_state
        
        logits = vl_gpt.gen_head(hidden_states[:, -1, :])
        logit_cond = logits[0::2, :]
        logit_uncond = logits[1::2, :]
        
        logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)
        probs = torch.softmax(logits / temperature, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        generated_tokens[:, i] = next_token.squeeze(dim=-1)
        next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)
        img_embeds = vl_gpt.prepare_gen_img_embeds(next_token)
        inputs_embeds = img_embeds.unsqueeze(dim=1)
    
    dec = vl_gpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])
    dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)
    dec = np.clip((dec + 1) / 2 * 255, 0, 255)
    visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)
    visual_img[:, :, :] = dec
    
    return [PIL.Image.fromarray(img) for img in visual_img]

# Gradio interface
with gr.Blocks() as demo:
    gr.Markdown("# Janus Model Demo")
    
    with gr.Tab("Multimodal Understanding"):
        with gr.Row():
            image_input = gr.Image()
            question_input = gr.Textbox(label="Question")
        understanding_button = gr.Button("Analyze")
        understanding_output = gr.Textbox(label="Answer")
        
    with gr.Tab("Text-to-Image Generation"):
        prompt_input = gr.Textbox(label="Prompt")
        with gr.Row():
            temperature_input = gr.Slider(minimum=0.1, maximum=2.0, value=1.0, label="Temperature")
            cfg_weight_input = gr.Slider(minimum=1, maximum=10, value=5, step=0.5, label="CFG Weight")
        generation_button = gr.Button("Generate Images")
        image_output = gr.Gallery(label="Generated Images", columns=2, rows=2)
    
    understanding_button.click(
        multimodal_understanding,
        inputs=[image_input, question_input],
        outputs=understanding_output
    )
    
    generation_button.click(
        text_to_image,
        inputs=[prompt_input, temperature_input, cfg_weight_input],
        outputs=image_output
    )

demo.launch(share=True)

the colab script
janus.ipynb

Honestly, the quality of the generated images isn’t great.

Originally created by @0smboy on GitHub (Oct 18, 2024). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/5 <img width="1574" alt="image" src="https://github.com/user-attachments/assets/a986043c-df2b-40df-9323-dffe824e29e9"> <img width="1624" alt="image" src="https://github.com/user-attachments/assets/967c8723-71a5-40d8-9326-e2b99d773d5a"> First you should **new file `app.py` in Janus dir** ```python import gradio as gr import torch from transformers import AutoModelForCausalLM from janus.models import MultiModalityCausalLM, VLChatProcessor from janus.utils.io import load_pil_images import PIL.Image import numpy as np import os # Load model and processor model_path = "deepseek-ai/Janus-1.3B" vl_chat_processor = VLChatProcessor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer vl_gpt = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True) vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval() # Multimodal Understanding function def multimodal_understanding(image, question): conversation = [ { "role": "User", "content": f"<image_placeholder>\n{question}", "images": [image], }, {"role": "Assistant", "content": ""}, ] pil_images = [PIL.Image.fromarray(image)] prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True ).to(vl_gpt.device) inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) outputs = vl_gpt.language_model.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True, ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True) return answer # Text-to-Image Generation function @torch.inference_mode() def text_to_image(prompt, temperature=1, parallel_size=4, cfg_weight=5): conversation = [ {"role": "User", "content": prompt}, {"role": "Assistant", "content": ""}, ] sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts( conversations=conversation, sft_format=vl_chat_processor.sft_format, system_prompt="", ) full_prompt = sft_format + vl_chat_processor.image_start_tag input_ids = vl_chat_processor.tokenizer.encode(full_prompt) input_ids = torch.LongTensor(input_ids) tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda() for i in range(parallel_size*2): tokens[i, :] = input_ids if i % 2 != 0: tokens[i, 1:-1] = vl_chat_processor.pad_id inputs_embeds = vl_gpt.language_model.get_input_embeddings()(tokens) image_token_num_per_image = 576 img_size = 384 patch_size = 16 generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda() for i in range(image_token_num_per_image): outputs = vl_gpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None) hidden_states = outputs.last_hidden_state logits = vl_gpt.gen_head(hidden_states[:, -1, :]) logit_cond = logits[0::2, :] logit_uncond = logits[1::2, :] logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond) probs = torch.softmax(logits / temperature, dim=-1) next_token = torch.multinomial(probs, num_samples=1) generated_tokens[:, i] = next_token.squeeze(dim=-1) next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1) img_embeds = vl_gpt.prepare_gen_img_embeds(next_token) inputs_embeds = img_embeds.unsqueeze(dim=1) dec = vl_gpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size]) dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1) dec = np.clip((dec + 1) / 2 * 255, 0, 255) visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8) visual_img[:, :, :] = dec return [PIL.Image.fromarray(img) for img in visual_img] # Gradio interface with gr.Blocks() as demo: gr.Markdown("# Janus Model Demo") with gr.Tab("Multimodal Understanding"): with gr.Row(): image_input = gr.Image() question_input = gr.Textbox(label="Question") understanding_button = gr.Button("Analyze") understanding_output = gr.Textbox(label="Answer") with gr.Tab("Text-to-Image Generation"): prompt_input = gr.Textbox(label="Prompt") with gr.Row(): temperature_input = gr.Slider(minimum=0.1, maximum=2.0, value=1.0, label="Temperature") cfg_weight_input = gr.Slider(minimum=1, maximum=10, value=5, step=0.5, label="CFG Weight") generation_button = gr.Button("Generate Images") image_output = gr.Gallery(label="Generated Images", columns=2, rows=2) understanding_button.click( multimodal_understanding, inputs=[image_input, question_input], outputs=understanding_output ) generation_button.click( text_to_image, inputs=[prompt_input, temperature_input, cfg_weight_input], outputs=image_output ) demo.launch(share=True) ``` the colab script [janus.ipynb](https://nbviewer.jupyter.org/gist/0smboy/d778ca0b93b5300bd6769157d5f27706#file-janus-ipynb) Honestly, the quality of the generated images isn’t great.
bsakaguchi commented 2026-10-26 21:09:49 +00:00
Author
Owner
Copy link

@aj47 commented on GitHub (Oct 18, 2024):

there is no app.py

<!-- gh-comment-id:2422661677 --> @aj47 commented on GitHub (Oct 18, 2024): there is no app.py
bsakaguchi commented 2026-10-26 21:09:49 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 18, 2024):

@aj47 new file app.py in Janus dir，app file content is above.

<!-- gh-comment-id:2422685086 --> @0smboy commented on GitHub (Oct 18, 2024): @aj47 new file app.py in Janus dir，app file content is above.
bsakaguchi commented 2026-10-26 21:09:49 +00:00
Author
Owner
Copy link

@aj47 commented on GitHub (Oct 18, 2024):

Thanks! sorry i missed that.
unfortunately doesn't seem to work with the t4 google colab GPU
image

<!-- gh-comment-id:2422714435 --> @aj47 commented on GitHub (Oct 18, 2024): Thanks! sorry i missed that. unfortunately doesn't seem to work with the t4 google colab GPU <img width="1037" alt="image" src="https://github.com/user-attachments/assets/03200e84-a04a-4d03-b634-3e8d2b4f4db7">
bsakaguchi commented 2026-10-26 21:09:50 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 18, 2024):

@aj47 try L4

<!-- gh-comment-id:2422731890 --> @0smboy commented on GitHub (Oct 18, 2024): @aj47 try L4
bsakaguchi commented 2026-10-26 21:09:50 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 18, 2024):

@aj47 with T4 gpu you can make it work by doing this:

  1. find model config
!find /root -name "config.json"

my config file is:
/root/.cache/huggingface/hub/models--deepseek-ai--Janus-1.3B/snapshots/825ec516a2c161d21f42d384b2745fa59d61560c/config.json

  1. Edit the config.json -> Replace "_attn_implementation": "flash_attention_2" with "_attn_implementation": "eager"
  2. execute python app.py

without flash attention it will run slowly

<!-- gh-comment-id:2422806176 --> @0smboy commented on GitHub (Oct 18, 2024): @aj47 with T4 gpu you can make it work by doing this: 1. find model config ``` !find /root -name "config.json" ``` > my config file is: `/root/.cache/huggingface/hub/models--deepseek-ai--Janus-1.3B/snapshots/825ec516a2c161d21f42d384b2745fa59d61560c/config.json` 2. Edit the config.json -> Replace "_attn_implementation": "flash_attention_2" with "_attn_implementation": "eager" 3. execute `python app.py` without flash attention it will run slowly
bsakaguchi commented 2026-10-26 21:09:50 +00:00
Author
Owner
Copy link

@aj47 commented on GitHub (Oct 18, 2024):

thank you @0smboy ! the find command didnt work but i manually made the change and now its working on the t4
image

<!-- gh-comment-id:2422892661 --> @aj47 commented on GitHub (Oct 18, 2024): thank you @0smboy ! the find command didnt work but i manually made the change and now its working on the t4 <img width="570" alt="image" src="https://github.com/user-attachments/assets/3d94bbc3-d9b9-4300-869e-aed6d923d89e">
bsakaguchi commented 2026-10-26 21:09:50 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 18, 2024):

@aj47 colab code cell execute cmd
should add ! first,like this:

!find /root -name "config.json"
<!-- gh-comment-id:2423346302 --> @0smboy commented on GitHub (Oct 18, 2024): @aj47 colab code cell execute cmd should add **!** first,like this: ```python !find /root -name "config.json" ```
bsakaguchi commented 2026-10-26 21:09:50 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Oct 19, 2024):

@0smboy
Thank you for your interest in our work! In the field of visual generation, generating portraits is a particularly challenging task. On one hand, when collecting training data, we focused more on general scenes and did not add additional data specifically for the "human portrait" category, which may lead to some imperfections in portrait generation. On the other hand, specifying "digital art" in the prompt or designing a more detailed prompt can help produce better images. Please see the following examples (we use default setting, temperature=1, cfg=5):

  • a pretty girl, digital art
    img_v3_00fq_838af863-79b8-414f-959f-5ce669e11d9g

  • In a sunlit meadow, a pretty girl stands gracefully, her chestnut hair shimmering in the golden light. Her emerald eyes sparkle with curiosity and tranquility. Dressed in a flowing lavender dress adorned with lace Wildflowers bloom around her, and butterflies flutter nearby. Her expression is one of quiet contemplation, blending innocence with wisdom. She embodies a harmonious blend of inner and outer beauty, inviting the viewer to explore her enchanting story.
    img_v3_02fq_a0383ed6-9790-4bc4-a33d-815bea18931g

<!-- gh-comment-id:2423734927 --> @charlesCXK commented on GitHub (Oct 19, 2024): @0smboy Thank you for your interest in our work! In the field of visual generation, generating portraits is a particularly challenging task. On one hand, when collecting training data, we focused more on general scenes and did not add additional data specifically for the "human portrait" category, which may lead to some imperfections in portrait generation. On the other hand, specifying "**digital art**" in the prompt or designing **a more detailed prompt** can help produce better images. Please see the following examples (we use default setting, temperature=1, cfg=5): - a pretty girl, **digital art** ![img_v3_00fq_838af863-79b8-414f-959f-5ce669e11d9g](https://github.com/user-attachments/assets/eb98e73d-0ce1-4919-a90e-1d702157c960) - In a sunlit meadow, a pretty girl stands gracefully, her chestnut hair shimmering in the golden light. Her emerald eyes sparkle with curiosity and tranquility. Dressed in a flowing lavender dress adorned with lace Wildflowers bloom around her, and butterflies flutter nearby. Her expression is one of quiet contemplation, blending innocence with wisdom. She embodies a harmonious blend of inner and outer beauty, inviting the viewer to explore her enchanting story. ![img_v3_02fq_a0383ed6-9790-4bc4-a33d-815bea18931g](https://github.com/user-attachments/assets/4d46821b-4ab9-4af3-9fc9-f4b779e71825)
bsakaguchi commented 2026-10-26 21:09:51 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 19, 2024):

image image image @charlesCXK Great！, I tried a few similar prompts, and it seems to work really well! Thanks for presenting your work!
<!-- gh-comment-id:2423763298 --> @0smboy commented on GitHub (Oct 19, 2024): <img width="1642" alt="image" src="https://github.com/user-attachments/assets/75fa411d-b6e7-4e8e-9586-887115ee1703"> <img width="1637" alt="image" src="https://github.com/user-attachments/assets/920a8b79-bb5b-4b6b-8799-06152137d37a"> <img width="1645" alt="image" src="https://github.com/user-attachments/assets/4efa6e35-614f-4902-a945-feb212198123"> @charlesCXK Great！, I tried a few similar prompts, and it seems to work really well! Thanks for presenting your work!
bsakaguchi commented 2026-10-26 21:09:51 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Oct 20, 2024):

@0smboy
We found a bug in the tokenizer_config.json, where we forgot to set the pad_token, which caused issues with the classifier-free guidance in visual generation, resulting in poor performance. The correct file has already been uploaded to Huggingface.

<!-- gh-comment-id:2424827910 --> @charlesCXK commented on GitHub (Oct 20, 2024): @0smboy We found a bug in the `tokenizer_config.json`, where we forgot to set the `pad_token`, which caused issues with the classifier-free guidance in visual generation, resulting in poor performance. The correct file has already been uploaded to Huggingface.
bsakaguchi commented 2026-10-26 21:09:51 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 20, 2024):

@charlesCXK Good, I will try the new effects again!

<!-- gh-comment-id:2424859310 --> @0smboy commented on GitHub (Oct 20, 2024): @charlesCXK Good, I will try the new effects again!
bsakaguchi commented 2026-10-26 21:09:51 +00:00
Author
Owner
Copy link

@0smboy commented on GitHub (Oct 20, 2024):

image @charlesCXK I tried a few, but it seems like the new results didn’t improve much.
<!-- gh-comment-id:2424913584 --> @0smboy commented on GitHub (Oct 20, 2024): <img width="1647" alt="image" src="https://github.com/user-attachments/assets/5e09da67-9d1e-484b-be52-61bea8c5411a"> @charlesCXK I tried a few, but it seems like the new results didn’t improve much.
bsakaguchi commented 2026-10-26 21:09:51 +00:00
Author
Owner
Copy link

@charlesCXK commented on GitHub (Oct 20, 2024):

@0smboy
It should have improvements in most cases, as the original file would cause the classifier-free guidance to not work. 😉

<!-- gh-comment-id:2424991269 --> @charlesCXK commented on GitHub (Oct 20, 2024): @0smboy It should have improvements in most cases, as the original file would cause the classifier-free guidance to not work. 😉
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/Janus#6
No description provided.