[GH-ISSUE #151] Duplicate bos tokens after using tokenizer.apply_chat_template and tokenize #123

New issue

Closed

opened 2026-10-26 21:14:48 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:14:48 +00:00

Owner

Originally created by @yanyongyu on GitHub (Jan 29, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/151

I'm trying to use deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, the add_bos_token option in the tokenizer_config is true. After applying chat template and tokenization, there are two bos tokens in the input ids.

I notice that the add_bos_token option has been changed twice in the commit b640c47 and 4831ee1. I'm not sure if the behavior is as expected and whether affects the inferencing and training process.

Re-produce code:

from transformers import AutoTokenizer

t = AutoTokenizer.from_pretrained("./data/DeepSeek-R1-Distill-Qwen-7B", use_fast=True)

# tokenize
content = t.apply_chat_template([{"role": "user", "content": "hello"}], tokenize=False)
result = t(content)
result # {'input_ids': [151646, 151646, 151644, 14990], 'attention_mask': [1, 1, 1, 1]}

Originally created by @yanyongyu on GitHub (Jan 29, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/151 I'm trying to use deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, the `add_bos_token` option in the tokenizer_config is `true`. After applying chat template and tokenization, there are two bos tokens in the input ids. I notice that the `add_bos_token` option has been changed twice in the commit [b640c47](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/commit/b640c47fb9acbc6333ced6216252a17ed3abc4f6) and [4831ee1](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/commit/4831ee1375be5b4ff5a4abf7984e13628db44e35). I'm not sure if the behavior is as expected and whether affects the inferencing and training process. Re-produce code: ```python from transformers import AutoTokenizer t = AutoTokenizer.from_pretrained("./data/DeepSeek-R1-Distill-Qwen-7B", use_fast=True) # tokenize content = t.apply_chat_template([{"role": "user", "content": "hello"}], tokenize=False) result = t(content) result # {'input_ids': [151646, 151646, 151644, 14990], 'attention_mask': [1, 1, 1, 1]} ```

bsakaguchi

2026-10-26 21:14:48 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:14:49 +00:00

Author

Owner

@Adam-fei commented on GitHub (Feb 10, 2025):

Hello，I met the same issue.
Have you resolve this problem?

I see. You should modify the tokenizer_config.json. Change add_bos_token from true to false
But I don't understand why it is true by default.

@Adam-fei commented on GitHub (Feb 10, 2025): Hello，I met the same issue. Have you resolve this problem? --------------------------------- I see. You should modify the` tokenizer_config.json`. Change `add_bos_token` from true to false But I don't understand why it is true by default.

bsakaguchi commented

2026-10-26 21:14:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 7, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Apr 7, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:14:50 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 9, 2025):

false

@github-actions[bot] commented on GitHub (May 9, 2025): false

bsakaguchi referenced this issue

2026-10-26 21:26:29 +00:00

[PR #123] docs: Add key features and improve contact section in README #761