deepseek-ai/DeepSeek-Coder
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-Coder.git synced 2026-11-05 00:57:19 +00:00
Code Issues 177 Projects Packages Wiki Activity

[GH-ISSUE #50] tokenizer.model #41

New issue
Closed
opened 2026-10-26 21:02:40 +00:00 by bsakaguchi · 12 comments
bsakaguchi commented 2026-10-26 21:02:40 +00:00
Owner
Copy link

Originally created by @SinanAkkoyun on GitHub (Nov 24, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/50

Hey!

I'd like to work on implementing exllama support, but the tokenizer.model is missing.

https://discord.com/channels/1169871344037548062/1170016413428228168/1171019891839615077
https://discord.com/channels/1169871344037548062/1170016413428228168/1171040772229967952

A couple of weeks ago I asked for this in the discord, I was told that you are working on that

I would highly appreciate if you published the tokenizer.model :)

Originally created by @SinanAkkoyun on GitHub (Nov 24, 2023). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/50 Hey! I'd like to work on implementing exllama support, but the `tokenizer.model` is missing. https://discord.com/channels/1169871344037548062/1170016413428228168/1171019891839615077 https://discord.com/channels/1169871344037548062/1170016413428228168/1171040772229967952 A couple of weeks ago I asked for this in the discord, I was told that you are working on that I would highly appreciate if you published the `tokenizer.model` :)
bsakaguchi 2026-10-26 21:02:40 +00:00
  • closed this issue
  • added the
    WIP
         label
bsakaguchi commented 2026-10-26 21:02:41 +00:00
Author
Owner
Copy link

@guoday commented on GitHub (Nov 24, 2023):

Thank you for your interest in DeepSeek Coder. Currently we can not provide a sentencepiece model (tokenizer.model) that achieves the same output as the existing HuggingFace Tokenizer. We are working hard to help the model quantization community directly support Huggingface's Tokenizer, and updates on our progress will be timely synced to ​https://github.com/deepseek-ai/DeepSeek-Coder#7-qa.

<!-- gh-comment-id:1825811410 --> @guoday commented on GitHub (Nov 24, 2023): Thank you for your interest in DeepSeek Coder. Currently we can not provide a sentencepiece model (tokenizer.model) that achieves the same output as the existing HuggingFace Tokenizer. We are working hard to help the model quantization community directly support Huggingface's Tokenizer, and updates on our progress will be timely synced to ​https://github.com/deepseek-ai/DeepSeek-Coder#7-qa.
bsakaguchi commented 2026-10-26 21:02:41 +00:00
Author
Owner
Copy link

@guoday commented on GitHub (Nov 24, 2023):

For exllamaV2, we are also working on it.

<!-- gh-comment-id:1825812594 --> @guoday commented on GitHub (Nov 24, 2023): For exllamaV2, we are also working on it.
bsakaguchi commented 2026-10-26 21:02:41 +00:00
Author
Owner
Copy link

@SinanAkkoyun commented on GitHub (Nov 25, 2023):

Thank you for your fast reply.

For exllamaV2, we are also working on it.

So, would it totally suffice if Exllamav2 also supported the HF tokenzier? Is the deepseek model architecture equivalent to Llama only except for the tokenizer?

<!-- gh-comment-id:1826173495 --> @SinanAkkoyun commented on GitHub (Nov 25, 2023): Thank you for your fast reply. > For exllamaV2, we are also working on it. So, would it totally suffice if Exllamav2 also supported the HF tokenzier? Is the deepseek model architecture equivalent to Llama only except for the tokenizer?
bsakaguchi commented 2026-10-26 21:02:41 +00:00
Author
Owner
Copy link

@guoday commented on GitHub (Nov 25, 2023):

Yes, we use the same architecture as Llama except for the tokenizer. We attempted to convert the HuggingFace Tokenizer into a sentencepiece model (tokenizer.model), but couldn't achieve the same output. Therefore, we have to try to enable Exllamav2 to support the HF tokenizer. The current issue is that Exllamav2 uses a feature (ignore special token) within the sentencepiece model, which the HF tokenizer doesn't have. We are actively working on a solution.

<!-- gh-comment-id:1826191954 --> @guoday commented on GitHub (Nov 25, 2023): Yes, we use the same architecture as Llama except for the tokenizer. We attempted to convert the HuggingFace Tokenizer into a sentencepiece model (tokenizer.model), but couldn't achieve the same output. Therefore, we have to try to enable Exllamav2 to support the HF tokenizer. The current issue is that Exllamav2 uses a feature (ignore special token) within the sentencepiece model, which the HF tokenizer doesn't have. We are actively working on a solution.
bsakaguchi commented 2026-10-26 21:02:42 +00:00
Author
Owner
Copy link

@SinanAkkoyun commented on GitHub (Nov 26, 2023):

That's awesome, thank you for the work!!!

<!-- gh-comment-id:1826933663 --> @SinanAkkoyun commented on GitHub (Nov 26, 2023): That's awesome, thank you for the work!!!
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@BingxuanWang commented on GitHub (Nov 28, 2023):

@SinanAkkoyun Hi, we just updated our readme about exllamav2, you could try out the forked version which supports huggingface tokenizer and we'd like to hear from your feedback.

<!-- gh-comment-id:1829265855 --> @BingxuanWang commented on GitHub (Nov 28, 2023): @SinanAkkoyun Hi, we just updated our readme about exllamav2, you could try out the forked version which supports huggingface tokenizer and we'd like to hear from your feedback.
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@SinanAkkoyun commented on GitHub (Nov 28, 2023):

Hi! Thank you so much, I will test your fork asap

<!-- gh-comment-id:1829268969 --> @SinanAkkoyun commented on GitHub (Nov 28, 2023): Hi! Thank you so much, I will test your fork asap
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@SinanAkkoyun commented on GitHub (Nov 28, 2023):

@BingxuanWang I just tried to run the fork and the models only output nonsense

I even followed the exact prompt formatting, I am using GPTQ quants of TheBloke of the 6.7B and 1.3B model

python examples/chat.py -m../../models/deepseek/deepseek-coder-1.3b-instruct-GPTQ/ -mode deepseek
 -- Model: ../../models/deepseek/deepseek-coder-1.3b-instruct-GPTQ/
 -- Options: ['rope_scale 1.0', 'rope_alpha 1.0']
 -- Loading model...
 -- Loading tokenizer...
 -- Prompt format: deepseek
 -- System prompt:

You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.

User: How can I tell the time in python?

���ııiiiii iii iv i i ii io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io

6.7B shows very similar behaviour, but often returns some kind of invisible token in an infinite loop

<!-- gh-comment-id:1829735316 --> @SinanAkkoyun commented on GitHub (Nov 28, 2023): @BingxuanWang I just tried to run the fork and the models only output nonsense I even followed the exact prompt formatting, I am using GPTQ quants of TheBloke of the 6.7B and 1.3B model ``` python examples/chat.py -m../../models/deepseek/deepseek-coder-1.3b-instruct-GPTQ/ -mode deepseek -- Model: ../../models/deepseek/deepseek-coder-1.3b-instruct-GPTQ/ -- Options: ['rope_scale 1.0', 'rope_alpha 1.0'] -- Loading model... -- Loading tokenizer... -- Prompt format: deepseek -- System prompt: You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer. User: How can I tell the time in python? ���ııiiiii iii iv i i ii io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io io ``` 6.7B shows very similar behaviour, but often returns some kind of invisible token in an infinite loop
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@SinanAkkoyun commented on GitHub (Nov 28, 2023):

Turboderp told me that RoPe scale 4 is needed, now the model outputs english sentences. The 6.7B model works fine, the 1.3B quickly runs into repetition loops, but that seems to be a matter of quantization bitrate rather than pure inference

Please add the the chat_formats from here to the PR:
https://github.com/SinanAkkoyun/exllama-v2/tree/deepseek

Thank you for the work!

<!-- gh-comment-id:1830360713 --> @SinanAkkoyun commented on GitHub (Nov 28, 2023): Turboderp told me that RoPe scale 4 is needed, now the model outputs english sentences. The 6.7B model works fine, the 1.3B quickly runs into repetition loops, but that seems to be a matter of quantization bitrate rather than pure inference Please add the the chat_formats from here to the PR: https://github.com/SinanAkkoyun/exllama-v2/tree/deepseek Thank you for the work!
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@leedrake5 commented on GitHub (Dec 27, 2023):

Wanted to check in on this - trying to compile for llama.cpp but missing tokenizer.model is hamstringing that effort.

<!-- gh-comment-id:1870626499 --> @leedrake5 commented on GitHub (Dec 27, 2023): Wanted to check in on this - trying to compile for llama.cpp but missing tokenizer.model is hamstringing that effort.
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@BingxuanWang commented on GitHub (Dec 28, 2023):

@leedrake5 Please refer to https://github.com/deepseek-ai/DeepSeek-Coder#7-qa for the tokenizer.model issue

<!-- gh-comment-id:1870732164 --> @BingxuanWang commented on GitHub (Dec 28, 2023): @leedrake5 Please refer to https://github.com/deepseek-ai/DeepSeek-Coder#7-qa for the tokenizer.model issue
bsakaguchi commented 2026-10-26 21:02:44 +00:00
Author
Owner
Copy link

@leedrake5 commented on GitHub (Dec 28, 2023):

@BingxuanWang That worked perfectly, thanks!

<!-- gh-comment-id:1870801302 --> @leedrake5 commented on GitHub (Dec 28, 2023): @BingxuanWang That worked perfectly, thanks!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ydj/leetcode
bingxuan/dev
ydj/mbpp
No results found.
Labels
Clear labels   
documentation

   
ENV

   
pull-request
Mirrored from GitHub Pull Request

  
research question

   
WIP
No labels
documentation
ENV
pull-request
research question
WIP
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-Coder#41
No description provided.