[GH-ISSUE #12] The design of semantic tokenizer #9

New issue

Closed

opened 2026-10-26 21:09:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:09:53 +00:00

Owner

Originally created by @youngsheen on GitHub (Oct 19, 2024).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/12

Hi, Excellent work!

Have you tried other SSL models rather than CLIP for semantic tokenizer training?

I also find that the feature of SSL models can significantly boost the performance of autoregressive generative models (ref DiGIT at https://github.com/DAMO-NLP-SG/DiGIT). What I do is train an offline tokenizer with K-Means on DINOv2 hidden states. And we find that patch level optimization objective (DINO) is better than image level optimization objective (MoCo or CLIP maybe).

Also, in the paper you mention that the method can be applied to other modalities, for example audio. However, the audio CLIP model still lags behind audio SSL models. We also validate the above methodology in speech data (ref GPST at https://github.com/youngsheen/GPST).

Originally created by @youngsheen on GitHub (Oct 19, 2024). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/12 Hi, Excellent work! Have you tried other SSL models rather than CLIP for semantic tokenizer training? I also find that the feature of SSL models can significantly boost the performance of autoregressive generative models (ref DiGIT at https://github.com/DAMO-NLP-SG/DiGIT). What I do is train an offline tokenizer with K-Means on DINOv2 hidden states. And we find that patch level optimization objective (DINO) is better than image level optimization objective (MoCo or CLIP maybe). Also, in the paper you mention that the method can be applied to other modalities, for example audio. However, the audio CLIP model still lags behind audio SSL models. We also validate the above methodology in speech data (ref GPST at https://github.com/youngsheen/GPST).

bsakaguchi closed this issue

2026-10-26 21:09:53 +00:00

bsakaguchi commented

2026-10-26 21:09:54 +00:00

Author

Owner

@charlesCXK commented on GitHub (Oct 19, 2024):

Thank you for your attention! Regarding the semantic tokenizer, we would like to emphasize the following points:

It is only used in the ablation study, not in Janus itself. We use it as a stronger comparative baseline and have not tried other alternatives. In Janus, to maintain simplicity, the visual generation part still adopts the basic VQ Tokenizer.
The design of the semantic tokenizer aims to strengthen semantic representation to perform better in multimodal understanding tasks, rather than being designed specifically for visual generation. Therefore, we distilled it using methods like SigLIP, which are commonly used in understanding tasks.

Regarding your point that SSL features might be more suitable than CLIP features for visual generation tasks, I think that makes sense. On the one hand, CLIP-style models focus more on the overall semantics of the image during training, while SSL methods—since the label is the image itself—might capture low-level information more effectively. Recent work like REPresentation Alignment (REPA) also found that using DINOv2 features to guide the training of diffusion models can significantly accelerate convergence, proving that SSL models indeed have great benefits for image generation.

Lastly, since Janus is not only a model but also a framework—with the core idea being the decoupling of input encoding for different tasks—it’s easy to replace the current VQ Tokenizer with a more powerful one, such as one distilled via DINOv2, for image generation. Similarly, if audio encoder is added, one can also utilize an SSL encoder (if an SSL encoder is indeed better than a CLIP-style encoder in the context of audio).

@charlesCXK commented on GitHub (Oct 19, 2024): Thank you for your attention! Regarding the semantic tokenizer, we would like to emphasize the following points: - It is **only used in the ablation study**, not in Janus itself. We use it as a stronger comparative baseline and have not tried other alternatives. In Janus, to maintain simplicity, the visual generation part still adopts the basic VQ Tokenizer. - The design of the semantic tokenizer aims to strengthen semantic representation to **perform better in multimodal understanding tasks, rather than being designed specifically for visual generation**. Therefore, we distilled it using methods like SigLIP, which are commonly used in understanding tasks. Regarding your point that SSL features might be more suitable than CLIP features for visual generation tasks, I think that makes sense. On the one hand, CLIP-style models focus more on the overall semantics of the image during training, while SSL methods—since the label is the image itself—might capture low-level information more effectively. Recent work like REPresentation Alignment (REPA) also found that using DINOv2 features to guide the training of diffusion models can significantly accelerate convergence, proving that SSL models indeed have great benefits for image generation. Lastly, since Janus is not only a model but also a framework—with the core idea being the decoupling of input encoding for different tasks—it’s easy to replace the current VQ Tokenizer with a more powerful one, such as one distilled via DINOv2, for image generation. Similarly, if audio encoder is added, one can also utilize an SSL encoder (if an SSL encoder is indeed better than a CLIP-style encoder in the context of audio).

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/Janus#9

No description provided.

Rows
Columns