[GH-ISSUE #19] A question from beginner: How Chinese chars converted in the vocab? #17

New issue

Open

opened 2026-10-26 21:10:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:10:17 +00:00

Owner

Originally created by @Adenialzz on GitHub (Dec 24, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/19

I notice that in deepseek (llama) vocab, there is no Chinese chars in that, instead of something like åĲ¦, and that can be decoded to Chinese 否. I wonder how this happened. I have checked the source code in transformers and did not find the implementation about this.

(Pdb) mllm.tokenizer.convert_ids_to_tokens(3636)
'åĲ¦'
(Pdb) mllm.tokenizer.decode([3636])
'否'
(Pdb)

Originally created by @Adenialzz on GitHub (Dec 24, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/19 I notice that in [deepseek (llama) vocab](https://huggingface.co/deepseek-ai/deepseek-vl2/raw/main/tokenizer.json), there is no Chinese chars in that, instead of something like `åĲ¦`, and that can be decoded to Chinese `否`. I wonder how this happened. I have checked the source code in transformers and did not find the implementation about this. ```shell (Pdb) mllm.tokenizer.convert_ids_to_tokens(3636) 'åĲ¦' (Pdb) mllm.tokenizer.decode([3636]) '否' (Pdb) ```

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-VL2#17

No description provided.

Rows
Columns