[GH-ISSUE #352 ] 本地部署DeepSeek-R1-Distill-Qwen-32B,输出仅有&lt;/think&gt;，没有&lt;think&gt; #271

Author

Owner

@haowang02 commented on GitHub (Feb 10, 2025):

我想你遇到这个问题是因为昨天模型仓库更新后 chat_template 末尾增加了 <think>\\n，也就是说 <think> 标签在 prompt 中，而非生成的内容中。

不妨尝试将 chat_template 中的 <think>\\n 去掉再进行尝试。

参考如下代码：

template = tokenizer.chat_template
template = template.replace("<think>\\n", "")  # 去掉末尾的 <think>\\n
tokenizer.chat_template = template  # Set the new template

对比结果如下：

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")

conversation = [
    {
        "role": "user",
        "content": "你好"
    },
]

print('原始 chat template')
print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False))

template = tokenizer.chat_template
template = template.replace("<think>\\n", "")  # 去掉末尾的 <think>\\n
tokenizer.chat_template = template  # Set the new template

print('修改后 chat template')
print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False))

输出：

原始 chat template
<｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜><think>

修改后 chat template
<｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜>

@haowang02 commented on GitHub (Feb 10, 2025): 我想你遇到这个问题是因为昨天模型仓库更新后 chat_template 末尾增加了 `<think>\\n`，也就是说 `<think>` 标签在 prompt 中，而非生成的内容中。不妨尝试将 chat_template 中的 `<think>\\n` 去掉再进行尝试。参考如下代码： ```python template = tokenizer.chat_template template = template.replace("<think>\\n", "") # 去掉末尾的 <think>\\n tokenizer.chat_template = template # Set the new template ``` 对比结果如下： ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B") conversation = [ { "role": "user", "content": "你好" }, ] print('原始 chat template') print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)) template = tokenizer.chat_template template = template.replace("<think>\\n", "") # 去掉末尾的 <think>\\n tokenizer.chat_template = template # Set the new template print('修改后 chat template') print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)) ``` 输出： ```text 原始 chat template <｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜><think> 修改后 chat template <｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜> ```

bsakaguchi commented

Author

Owner

@haowang02 commented on GitHub (Feb 10, 2025):

当然这样并不是一个好的解决问题的方法，因为官方说：

Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting <think>\n\n</think>) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with <think>\n at the beginning of every output.

@haowang02 commented on GitHub (Feb 10, 2025): 当然这样并不是一个好的解决问题的方法，因为官方说： > Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting `<think>\n\n</think>`) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with `<think>\n` at the beginning of every output.

bsakaguchi commented

Author

Owner

@zhaob1n commented on GitHub (Feb 10, 2025):

如果是vllm部署的该如何解决？

@zhaob1n commented on GitHub (Feb 10, 2025): 如果是vllm部署的该如何解决？

bsakaguchi commented

2026-10-26 21:18:29 +00:00

Author

Owner

@Kenwwww commented on GitHub (Feb 10, 2025):

如果是vllm部署的该如何解决？

新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

@Kenwwww commented on GitHub (Feb 10, 2025): > 如果是vllm部署的该如何解决？新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

bsakaguchi commented

Author

Owner

@goufbest commented on GitHub (Feb 10, 2025):

我想你遇到这个问题是因为昨天模型仓库更新后 chat_template 末尾增加了 <think>\\n，也就是说 <think> 标签在 prompt 中，而非生成的内容中。

不妨尝试将 chat_template 中的 <think>\\n 去掉再进行尝试。

参考如下代码：

template = tokenizer.chat_template
template = template.replace("\n", "") # 去掉末尾的 \n
tokenizer.chat_template = template # Set the new template
对比结果如下：

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")

conversation = [
{
"role": "user",
"content": "你好"
},
]

print('原始 chat template')
print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False))

template = tokenizer.chat_template
template = template.replace("\n", "") # 去掉末尾的 \n
tokenizer.chat_template = template # Set the new template

print('修改后 chat template')
print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False))
输出：
原始 chat template
<｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜><think>

修改后 chat template
<｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜>

感谢您的解决方案

@goufbest commented on GitHub (Feb 10, 2025): > 我想你遇到这个问题是因为昨天模型仓库更新后 chat_template 末尾增加了 `<think>\\n`，也就是说 `<think>` 标签在 prompt 中，而非生成的内容中。 > > 不妨尝试将 chat_template 中的 `<think>\\n` 去掉再进行尝试。 > > 参考如下代码： > > template = tokenizer.chat_template > template = template.replace("<think>\\n", "") # 去掉末尾的 <think>\\n > tokenizer.chat_template = template # Set the new template > 对比结果如下： > > from transformers import AutoTokenizer > > tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B") > > conversation = [ > { > "role": "user", > "content": "你好" > }, > ] > > print('原始 chat template') > print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)) > > template = tokenizer.chat_template > template = template.replace("<think>\\n", "") # 去掉末尾的 <think>\\n > tokenizer.chat_template = template # Set the new template > > print('修改后 chat template') > print(tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)) > 输出： > > ``` > 原始 chat template > <｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜><think> > > 修改后 chat template > <｜begin▁of▁sentence｜><｜User｜>你好<｜Assistant｜> > ``` 感谢您的解决方案

bsakaguchi commented

2026-10-26 21:18:29 +00:00

Author

Owner

@haowang02 commented on GitHub (Feb 10, 2025):

@xcpky 您好，给您提供一个我目前的解决方法。

如果是vllm部署的该如何解决？

新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

如果不方便修改 deepseek_r1_reasoning_parser.py，可以在 vllm serve 命令中指定 chat template。

具体而言，首先需要创建一个 template_deepseek_r1.jinja 文件，写入如下 chat template：

{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

上述 template 是官方模型仓库中更新前的 template

之后在 vllm serve 命令中指定该 template 即可（--chat-template ./template_deepseek_r1.jinja）：

vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --enforce-eager \
  --enable-reasoning \
  --reasoning-parser deepseek_r1 \
  --chat-template ./template_deepseek_r1.jinja

@haowang02 commented on GitHub (Feb 10, 2025): @xcpky 您好，给您提供一个我目前的解决方法。 > > 如果是vllm部署的该如何解决？ > > 新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下如果不方便修改 `deepseek_r1_reasoning_parser.py`，可以在 vllm serve 命令中指定 chat template。具体而言，首先需要创建一个 `template_deepseek_r1.jinja` 文件，写入如下 chat template： ``` {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %} ``` *上述 template 是官方模型仓库中更新前的 template* 之后在 vllm serve 命令中指定该 template 即可（`--chat-template ./template_deepseek_r1.jinja`）： ``` vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \ --tensor-parallel-size 4 \ --max-model-len 32768 \ --enforce-eager \ --enable-reasoning \ --reasoning-parser deepseek_r1 \ --chat-template ./template_deepseek_r1.jinja ```

bsakaguchi commented

2026-10-26 21:18:29 +00:00

Author

Owner

@zhaob1n commented on GitHub (Feb 10, 2025):

@xcpky 您好，给您提供一个我目前的解决方法。

如果是vllm部署的该如何解决？

新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

如果不方便修改 deepseek_r1_reasoning_parser.py，可以在 vllm serve 命令中指定 chat template。

具体而言，首先需要创建一个 template_deepseek_r1.jinja 文件，写入如下 chat template：

{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

上述 template 是官方模型仓库中更新前的 template

之后在 vllm serve 命令中指定该 template 即可（--chat-template ./template_deepseek_r1.jinja）：

vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --enforce-eager \
  --enable-reasoning \
  --reasoning-parser deepseek_r1 \
  --chat-template ./template_deepseek_r1.jinja

我刚才直接修改了huggingface下载的文件里面的template。我在想能否有更好的办法，比如在content输出中直接添加<think>？

@zhaob1n commented on GitHub (Feb 10, 2025): > [@xcpky](https://github.com/xcpky) 您好，给您提供一个我目前的解决方法。 > > > > 如果是vllm部署的该如何解决？ > > > > > > 新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下 > > 如果不方便修改 `deepseek_r1_reasoning_parser.py`，可以在 vllm serve 命令中指定 chat template。 > > 具体而言，首先需要创建一个 `template_deepseek_r1.jinja` 文件，写入如下 chat template： > > ``` > {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %} > ``` > > _上述 template 是官方模型仓库中更新前的 template_ > > 之后在 vllm serve 命令中指定该 template 即可（`--chat-template ./template_deepseek_r1.jinja`）： > > ``` > vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \ > --tensor-parallel-size 4 \ > --max-model-len 32768 \ > --enforce-eager \ > --enable-reasoning \ > --reasoning-parser deepseek_r1 \ > --chat-template ./template_deepseek_r1.jinja > ``` 我刚才直接修改了huggingface下载的文件里面的template。我在想能否有更好的办法，比如在content输出中直接添加`<think>`？

bsakaguchi commented

Author

Owner

@haowang02 commented on GitHub (Feb 10, 2025):

@xcpky 您好，给您提供一个我目前的解决方法。

如果是vllm部署的该如何解决？

新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

如果不方便修改 deepseek_r1_reasoning_parser.py，可以在 vllm serve 命令中指定 chat template。
具体而言，首先需要创建一个 template_deepseek_r1.jinja 文件，写入如下 chat template：

{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

上述 template 是官方模型仓库中更新前的 template
之后在 vllm serve 命令中指定该 template 即可（--chat-template ./template_deepseek_r1.jinja）：

vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --enforce-eager \
  --enable-reasoning \
  --reasoning-parser deepseek_r1 \
  --chat-template ./template_deepseek_r1.jinja

我刚才直接修改了huggingface下载的文件里面的template。我在想能否有更好的办法，比如在content输出中直接添加<think>？

那试试修改 vllm 的 deepseek_r1_reasoning_parser.py? 还能顺便给 vllm 提个 PR 👍

@haowang02 commented on GitHub (Feb 10, 2025): > > [@xcpky](https://github.com/xcpky) 您好，给您提供一个我目前的解决方法。 > > > > 如果是vllm部署的该如何解决？ > > > > > > > > > 新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下 > > > > > > 如果不方便修改 `deepseek_r1_reasoning_parser.py`，可以在 vllm serve 命令中指定 chat template。 > > 具体而言，首先需要创建一个 `template_deepseek_r1.jinja` 文件，写入如下 chat template： > > ``` > > {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %} > > ``` > > > > > > > > > > > > > > > > > > > > > > > > _上述 template 是官方模型仓库中更新前的 template_ > > 之后在 vllm serve 命令中指定该 template 即可（`--chat-template ./template_deepseek_r1.jinja`）： > > ``` > > vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \ > > --tensor-parallel-size 4 \ > > --max-model-len 32768 \ > > --enforce-eager \ > > --enable-reasoning \ > > --reasoning-parser deepseek_r1 \ > > --chat-template ./template_deepseek_r1.jinja > > ``` > > 我刚才直接修改了huggingface下载的文件里面的template。我在想能否有更好的办法，比如在content输出中直接添加`<think>`？那试试修改 vllm 的 deepseek_r1_reasoning_parser.py? 还能顺便给 vllm 提个 PR 👍

bsakaguchi commented

Author

Owner

@ariesfish commented on GitHub (Feb 10, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的<think>\\n，vLLM重新部署一下就可以了。

加上系统提示词:

think first and then answer user's question as below format every time:
<think>reasoning content</think>
content

可以大幅提高模型思考的概率

@ariesfish commented on GitHub (Feb 10, 2025): 直接修改tokenizer_config.json里的chat-template，去掉末尾的\<think\>\\\\n，vLLM重新部署一下就可以了。加上系统提示词: ``` think first and then answer user's question as below format every time: <think>reasoning content</think> content ``` 可以大幅提高模型思考的概率

bsakaguchi commented

Author

Owner

@hicaonmb commented on GitHub (Feb 10, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的<think>\\n，vLLM重新部署一下就可以了。

加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率

系统提示词在哪里加？

@hicaonmb commented on GitHub (Feb 10, 2025): > 直接修改tokenizer_config.json里的chat-template，去掉末尾的\<think\>\\\\n，vLLM重新部署一下就可以了。 > > 加上系统提示词: > ``` > think first and then answer user's question as below format every time: > <think>reasoning content</think> > content > ``` > 可以大幅提高模型思考的概率系统提示词在哪里加？

bsakaguchi commented

huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B@3865e12a1e

Author

Owner

@hwb96 commented on GitHub (Feb 11, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。
加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率
系统提示词在哪里加？

直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。

@hwb96 commented on GitHub (Feb 11, 2025): > > 直接修改tokenizer_config.json里的chat-template，去掉末尾的<think>\\n，vLLM重新部署一下就可以了。 > > 加上系统提示词: > > ``` > > think first and then answer user's question as below format every time: > > <think>reasoning content</think> > > content > > ``` > > > > > > > > > > > > > > > > > > > > > > > > 可以大幅提高模型思考的概率 > > 系统提示词在哪里加？ https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/commit/3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。

bsakaguchi commented

Author

Owner

@joze2016 commented on GitHub (Feb 12, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。
加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率
系统提示词在哪里加？
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B@3865e12a1e

直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。

这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。

@joze2016 commented on GitHub (Feb 12, 2025): > > > 直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。 > > > 加上系统提示词: > > > ``` > > > think first and then answer user's question as below format every time: > > > <think>reasoning content</think> > > > content > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 可以大幅提高模型思考的概率 > > > > > > 系统提示词在哪里加？ > > https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/commit/3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 > > 直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。

bsakaguchi commented

Author

Owner

@547414 commented on GitHub (Feb 13, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。
加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率
系统提示词在哪里加？
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B@3865e12a1e
直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。
这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。

要加上提示词才行，提示词可以加在prompt上，或者如果用了第三方web ui, 可以在设置中配置提示词

提示词参考：

先思考，然后按照以下格式回答用户的问题：
<think>推理内容</think>
内容

prompt:

maxkb:

@547414 commented on GitHub (Feb 13, 2025): > > > > 直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。 > > > > 加上系统提示词: > > > > ``` > > > > think first and then answer user's question as below format every time: > > > > <think>reasoning content</think> > > > > content > > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 可以大幅提高模型思考的概率 > > > > > > > > > 系统提示词在哪里加？ > > > > > > https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/commit/3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 > > 直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。 > > 这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。要加上提示词才行，提示词可以加在prompt上，或者如果用了第三方web ui, 可以在设置中配置提示词提示词参考： ``` 先思考，然后按照以下格式回答用户的问题： <think>推理内容</think> 内容 ``` prompt: ![Image](https://github.com/user-attachments/assets/7566ad4d-950a-45bb-92f6-d4743d5ae282) maxkb: ![Image](https://github.com/user-attachments/assets/cadf759b-a272-44f4-808f-c799c960259b)

bsakaguchi commented

Author

Owner

@ch9hn commented on GitHub (Feb 13, 2025):

We did exactly the same steps and run it, here the code:

{
  "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
  "messages": [
    {"role": "user", "content": "Question: Who are you? Requirement: Think first, and then answer the user's question in the following format: <think reasoning content</think>content"}
  ]
}

Chat Template (ending):

Instead of:

We put in this:

{% if add_generation_prompt and not ns.is_tool %}
{{'<｜Assistant｜>'}}
{% endif %}"

instead of

{% if add_generation_prompt and not ns.is_tool %}
{{'<｜Assistant｜><think>\\n'}}
{% endif %}"

Still the same result:

{
  "id": "chatcmpl-1aa0ba0b3cb34fd68ef6c452b880a8bd",
  "object": "chat.completion",
  "created": 1739435473,
  "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": null,
        "content": "Who are you?\" is a question that seeks to understand the identity or nature of the being or entity being addressed.\n\nI am an AI language model created to assist with answering various questions and providing information.\n\nThe appropriate response should clearly and concisely convey this information to the user.\n</think>\n\nI am an AI assistant created to provide helpful and honest responses to your questions.",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 36,
    "total_tokens": 111,
    "completion_tokens": 75,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}

@ch9hn commented on GitHub (Feb 13, 2025): We did exactly the same steps and run it, here the code: ``` { "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B", "messages": [ {"role": "user", "content": "Question: Who are you? Requirement: Think first, and then answer the user's question in the following format: <think reasoning content</think>content"} ] } ``` Chat Template (ending): Instead of: We put in this: ``` {% if add_generation_prompt and not ns.is_tool %} {{'<｜Assistant｜>'}} {% endif %}" ``` instead of ``` {% if add_generation_prompt and not ns.is_tool %} {{'<｜Assistant｜><think>\\n'}} {% endif %}" ``` Still the same result: ``` { "id": "chatcmpl-1aa0ba0b3cb34fd68ef6c452b880a8bd", "object": "chat.completion", "created": 1739435473, "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B", "choices": [ { "index": 0, "message": { "role": "assistant", "reasoning_content": null, "content": "Who are you?\" is a question that seeks to understand the identity or nature of the being or entity being addressed.\n\nI am an AI language model created to assist with answering various questions and providing information.\n\nThe appropriate response should clearly and concisely convey this information to the user.\n</think>\n\nI am an AI assistant created to provide helpful and honest responses to your questions.", "tool_calls": [] }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ], "usage": { "prompt_tokens": 36, "total_tokens": 111, "completion_tokens": 75, "prompt_tokens_details": null }, "prompt_logprobs": null } ```

bsakaguchi commented

Author

Owner

@ibrahimsteed commented on GitHub (Feb 14, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。
加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率
系统提示词在哪里加？
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B@3865e12a1e
直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。
这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。
要加上提示词才行，提示词可以加在prompt上，或者如果用了第三方web ui, 可以在设置中配置提示词

提示词参考：
先思考，然后按照以下格式回答用户的问题：
<think>推理内容</think>
内容
prompt:

maxkb:

你的接口调用结果content字段的标签内容，是不是应该在reasoning_content返回？

@ibrahimsteed commented on GitHub (Feb 14, 2025): > > > > > 直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。 > > > > > 加上系统提示词: > > > > > ``` > > > > > think first and then answer user's question as below format every time: > > > > > <think>reasoning content</think> > > > > > content > > > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 可以大幅提高模型思考的概率 > > > > > > > > > > > > 系统提示词在哪里加？ > > > > > > > > > https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/commit/3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 > > > 直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。 > > > > > > 这样修改后会造成有的问题没有模型思考。这个修改应该就是解决让模型每次都思考的。 > > 要加上提示词才行，提示词可以加在prompt上，或者如果用了第三方web ui, 可以在设置中配置提示词 > > 提示词参考： > > ``` > 先思考，然后按照以下格式回答用户的问题： > <think>推理内容</think> > 内容 > ``` > > prompt: ![Image](https://github.com/user-attachments/assets/7566ad4d-950a-45bb-92f6-d4743d5ae282) > > maxkb: ![Image](https://github.com/user-attachments/assets/cadf759b-a272-44f4-808f-c799c960259b) 你的接口调用结果content字段的<think>标签内容，是不是应该在reasoning_content返回？

bsakaguchi commented

Author

Owner

@liwei-rk commented on GitHub (Feb 14, 2025):

当然这样并不是一个好的解决问题的方法，因为官方说：

Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting <think>\n\n</think>) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with <think>\n at the beginning of every output.

为什么加上<think>\n，模型就一定会进入思考模式？<think>\n\n</think>，难道不是模型的输出吗？本质上没有区别啊

@liwei-rk commented on GitHub (Feb 14, 2025): > 当然这样并不是一个好的解决问题的方法，因为官方说： > > > Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting `<think>\n\n</think>`) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with `<think>\n` at the beginning of every output. 为什么加上`<think>\n`，模型就一定会进入思考模式？`<think>\n\n</think>`，难道不是模型的输出吗？本质上没有区别啊

bsakaguchi commented

Author

Owner

@ch9hn commented on GitHub (Feb 14, 2025):

This is actually true, with the old Chat Template the model is not responding with thinking text.

@ch9hn commented on GitHub (Feb 14, 2025): This is actually true, with the old Chat Template the model is not responding with thinking text.

bsakaguchi commented

Author

Owner

@Kenwwww commented on GitHub (Feb 17, 2025):

当然这样并不是一个好的解决问题的方法，因为官方说：

Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting <think>\n\n</think>) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with <think>\n at the beginning of every output.

为什么加上<think>\n，模型就一定会进入思考模式？<think>\n\n</think>，难道不是模型的输出吗？本质上没有区别啊

一样的你预加一个相当于诱导模型接着生成，思考不思考什么的都是模型输出，只是不输出，解析部分就不会把他解析到reasoning_content里面

@Kenwwww commented on GitHub (Feb 17, 2025): > > 当然这样并不是一个好的解决问题的方法，因为官方说： > > > Additionally, we have observed that the DeepSeek-R1 series models tend to bypass thinking pattern (i.e., outputting `<think>\n\n</think>`) when responding to certain queries, which can adversely affect the model's performance. To ensure that the model engages in thorough reasoning, we recommend enforcing the model to initiate its response with `<think>\n` at the beginning of every output. > > 为什么加上`<think>\n`，模型就一定会进入思考模式？`<think>\n\n</think>`，难道不是模型的输出吗？本质上没有区别啊一样的你预加一个<think>相当于诱导模型接着<think>生成，思考不思考什么的都是模型输出，只是不输出<think>，解析部分就不会把他解析到reasoning_content里面

bsakaguchi commented

Author

Owner

@jussker commented on GitHub (Feb 17, 2025):

个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导:

<think>
Please reason step by step.
Below is the actual prompt to ask the AI model:

对于蒸馏模型：修改 HuggingFace 模型的消息模版

这里以 DeepSeek-R1-Distill-Qwen-32B 为例子。

去模型文件里，有一个 tokenizer_config.json 文件，把里面chat_template的 <think>\\n 删除掉,

旧的：

"chat_template":"...{{'<｜Assistant｜><think>\\n'}}..."

删除后，新的：

"chat_template":"...{{'<｜Assistant｜>'}}..."

同时，设置用户消息增加前缀，把旧的：

"chat_template":"...{{'<｜User｜>' + message['content']}}..."

替换为新的:

"chat_template":"...{{'<｜User｜>' + '<think>\\nPlease reason step by step.\\nBelow is the actual prompt to ask the AI model:\\n\\n' + message['content']}}..."

@jussker commented on GitHub (Feb 17, 2025): 个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导: ``` <think> Please reason step by step. Below is the actual prompt to ask the AI model: ``` --- 对于蒸馏模型：修改 HuggingFace 模型的消息模版这里以 [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/blob/main/tokenizer_config.json) 为例子。去模型文件里，有一个 tokenizer_config.json 文件，把里面`chat_template`的 `<think>\\n` 删除掉, 旧的： ```JSON "chat_template":"...{{'<｜Assistant｜><think>\\n'}}..." ``` 删除后，新的： ```JSON "chat_template":"...{{'<｜Assistant｜>'}}..." ``` 同时，设置用户消息增加前缀，把旧的： ```JSON "chat_template":"...{{'<｜User｜>' + message['content']}}..." ``` 替换为新的: ```JSON "chat_template":"...{{'<｜User｜>' + '<think>\\nPlease reason step by step.\\nBelow is the actual prompt to ask the AI model:\\n\\n' + message['content']}}..." ```

bsakaguchi commented

Author

Owner

@Cyclones-Y commented on GitHub (Feb 18, 2025):

个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导:
<think>
Please reason step by step.
Below is the actual prompt to ask the AI model:
对于蒸馏模型：修改 HuggingFace 模型的消息模版

这里以 DeepSeek-R1-Distill-Qwen-32B 为例子。

去模型文件里，有一个 tokenizer_config.json 文件，把里面chat_template的 <think>\\n 删除掉,

旧的：

"chat_template":"...{{'<｜Assistant｜>\n'}}..."
删除后，新的：

"chat_template":"...{{'<｜Assistant｜>'}}..."
同时，设置用户消息增加前缀，把旧的：

"chat_template":"...{{'<｜User｜>' + message['content']}}..."
替换为新的:

"chat_template":"...{{'<｜User｜>' + '\nPlease reason step by step.\nBelow is the actual prompt to ask the AI model:\n\n' + message['content']}}..."

使用了这个模板，但还是会出现"reasoning_content":"\n\n"的情况，模型会避开思维推理；
配合这段系统提示词效果会好很多

先思考，然后按照以下格式回答用户的问题：
<think>推理内容</think>
内容

@Cyclones-Y commented on GitHub (Feb 18, 2025): > 个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导: > > ``` > <think> > Please reason step by step. > Below is the actual prompt to ask the AI model: > ``` > > 对于蒸馏模型：修改 HuggingFace 模型的消息模版 > > 这里以 [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/blob/main/tokenizer_config.json) 为例子。 > > 去模型文件里，有一个 tokenizer_config.json 文件，把里面`chat_template`的 `<think>\\n` 删除掉, > > 旧的： > > "chat_template":"...{{'<｜Assistant｜><think>\\n'}}..." > 删除后，新的： > > "chat_template":"...{{'<｜Assistant｜>'}}..." > 同时，设置用户消息增加前缀，把旧的： > > "chat_template":"...{{'<｜User｜>' + message['content']}}..." > 替换为新的: > > "chat_template":"...{{'<｜User｜>' + '<think>\\nPlease reason step by step.\\nBelow is the actual prompt to ask the AI model:\\n\\n' + message['content']}}..." 1. 使用了这个模板，但还是会出现"reasoning_content":"\n\n"的情况，模型会避开思维推理； 2. 配合这段系统提示词效果会好很多 ``` 先思考，然后按照以下格式回答用户的问题： <think>推理内容</think> 内容 ``` ![Image](https://github.com/user-attachments/assets/c9a14615-9f7a-4bd0-9897-e798c407f3d3)

bsakaguchi commented

Author

Owner

@ibrahimsteed commented on GitHub (Feb 18, 2025):

个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导:
<think>
Please reason step by step.
Below is the actual prompt to ask the AI model:
对于蒸馏模型：修改 HuggingFace 模型的消息模版
这里以 DeepSeek-R1-Distill-Qwen-32B 为例子。
去模型文件里，有一个 tokenizer_config.json 文件，把里面chat_template的 <think>\\n 删除掉,
旧的：
"chat_template":"...{{'<｜Assistant｜>\n'}}..."
删除后，新的：
"chat_template":"...{{'<｜Assistant｜>'}}..."
同时，设置用户消息增加前缀，把旧的：
"chat_template":"...{{'<｜User｜>' + message['content']}}..."
替换为新的:
"chat_template":"...{{'<｜User｜>' + '\nPlease reason step by step.\nBelow is the actual prompt to ask the AI model:\n\n' + message['content']}}..."
使用了这个模板，但还是会出现"reasoning_content":"\n\n"的情况，模型会避开思维推理；

配合这段系统提示词效果会好很多
先思考，然后按照以下格式回答用户的问题：
<think>推理内容</think>
内容

是存在这个问题，但模型实际是做了思维推理，只不过把推理过程放到了<content>中

@ibrahimsteed commented on GitHub (Feb 18, 2025): > > 个人使用的方法，激活思考不在 Asssistant 消息上加引导，而是在 User 消息上加引导: > > ``` > > <think> > > Please reason step by step. > > Below is the actual prompt to ask the AI model: > > ``` > > > > > > > > > > > > > > > > > > > > > > > > 对于蒸馏模型：修改 HuggingFace 模型的消息模版 > > 这里以 [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/blob/main/tokenizer_config.json) 为例子。 > > 去模型文件里，有一个 tokenizer_config.json 文件，把里面`chat_template`的 `<think>\\n` 删除掉, > > 旧的： > > "chat_template":"...{{'<｜Assistant｜>\n'}}..." > > 删除后，新的： > > "chat_template":"...{{'<｜Assistant｜>'}}..." > > 同时，设置用户消息增加前缀，把旧的： > > "chat_template":"...{{'<｜User｜>' + message['content']}}..." > > 替换为新的: > > "chat_template":"...{{'<｜User｜>' + '\nPlease reason step by step.\nBelow is the actual prompt to ask the AI model:\n\n' + message['content']}}..." > > 1. 使用了这个模板，但还是会出现"reasoning_content":"\n\n"的情况，模型会避开思维推理； > 2. 配合这段系统提示词效果会好很多 > > ``` > 先思考，然后按照以下格式回答用户的问题： > <think>推理内容</think> > 内容 > ``` > > ![Image](https://github.com/user-attachments/assets/c9a14615-9f7a-4bd0-9897-e798c407f3d3) 是存在这个问题，但模型实际是做了思维推理，只不过把推理过程放到了`<content>`中

bsakaguchi commented

Author

Owner

@isky-cd commented on GitHub (Feb 19, 2025):

但是通过加系统提示词的方式会存在一个问题就是，如果系统提示词是英文的，那虽然提问是用中文提问的但是他回答的是英文，相反，你系统提示词中文的，如果你使用英文问问题，他可能回答的是中文

@isky-cd commented on GitHub (Feb 19, 2025): 但是通过加系统提示词的方式会存在一个问题就是，如果系统提示词是英文的，那虽然提问是用中文提问的但是他回答的是英文，相反，你系统提示词中文的，如果你使用英文问问题，他可能回答的是中文

bsakaguchi commented

Author

Owner

@hahmad2008 commented on GitHub (Feb 19, 2025):

@jussker

I tried this chat template and now I can see the reasoning content, however I there is \n \n\n in the beginning of the reasoning content and the content.

example:

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"Alright"},"logprobs":null,"finish_reason":null}]}

...

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":".\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"\n\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"The"},"logprobs":null,"finish_reason":null}]}

chat-template:

 "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

@hahmad2008 commented on GitHub (Feb 19, 2025): @jussker I tried this chat template and now I can see the reasoning content, however I there is `\n` `\n\n` in the beginning of the reasoning content and the content. example: ``` data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"\n"},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"Alright"},"logprobs":null,"finish_reason":null}]} ... data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":".\n"},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"\n\n"},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"The"},"logprobs":null,"finish_reason":null}]} ``` chat-template: ``` "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}" ```

bsakaguchi commented

Author

Owner

@jussker commented on GitHub (Feb 19, 2025):

@jussker

I tried this chat template and now I can see the reasoning content, however I there is \n \n\n in the beginning of the reasoning content and the content.

example:

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"Alright"},"logprobs":null,"finish_reason":null}]}

...

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":".\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"\n\n"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"The"},"logprobs":null,"finish_reason":null}]}

chat-template:

 "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

@hahmad2008

After comparing the chat-template differences, found that you missed adding a prefix to user messages.

The modified chat-template is as follows:

"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + '<think>\\nPlease reason step by step.\\nBelow is the actual prompt to ask the AI model:\\n\\n' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

I have tested the modified chat-template on DeepSeek-R1-Distill-Qwen-1.5B, and it works normally.

@jussker commented on GitHub (Feb 19, 2025): > [@jussker](https://github.com/jussker) > > I tried this chat template and now I can see the reasoning content, however I there is `\n` `\n\n` in the beginning of the reasoning content and the content. > > example: > > ``` > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]} > > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"\n"},"logprobs":null,"finish_reason":null}]} > > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":"Alright"},"logprobs":null,"finish_reason":null}]} > > ... > > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"reasoning_content":".\n"},"logprobs":null,"finish_reason":null}]} > > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"\n\n"},"logprobs":null,"finish_reason":null}]} > > data: {"id":"chatcmpl-string","object":"chat.completion.chunk","created":1739869517,"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","choices":[{"index":0,"delta":{"content":"The"},"logprobs":null,"finish_reason":null}]} > ``` > > chat-template: > > ``` > "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}" > ``` @hahmad2008 After comparing the `chat-template` differences, found that you missed adding a prefix to user messages. <img width="1112" alt="Image" src="https://github.com/user-attachments/assets/a0084dc3-1806-4553-8b8c-17a9331d44da" /> --- The modified `chat-template` is as follows: ``` "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + '<think>\\nPlease reason step by step.\\nBelow is the actual prompt to ask the AI model:\\n\\n' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}" ``` I have tested the modified `chat-template` on `DeepSeek-R1-Distill-Qwen-1.5B`, and it works normally.

bsakaguchi commented

Author

Owner

@hahmad2008 commented on GitHub (Feb 19, 2025):

@jussker Thanks for your prompt response.
I tried the chatemplate you provided. but still I got the \n \n\n in the beginning of the reasoning content and the content.

@hahmad2008 commented on GitHub (Feb 19, 2025): @jussker Thanks for your prompt response. I tried the chatemplate you provided. but still I got the `\n` `\n\n` in the beginning of the reasoning content and the content.

bsakaguchi commented

Author

Owner

@verigle commented on GitHub (Feb 20, 2025):

sglang 用户如何解决这个问题？

@verigle commented on GitHub (Feb 20, 2025): sglang 用户如何解决这个问题？

bsakaguchi commented

Author

Owner

@newledge commented on GitHub (Feb 27, 2025):

如果是vllm部署的该如何解决？

新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下

什么版本？

@newledge commented on GitHub (Feb 27, 2025): > > 如果是vllm部署的该如何解决？ > > 新版vllm的话，可以把deepseek_r1_reasoning_parser.py改了，里面解析think标签的部分自定义一下什么版本？

bsakaguchi commented

https://github.com/vllm-project/vllm/issues/13375#issuecomment-2676496362

Author

Owner

@xring commented on GitHub (Feb 28, 2025):

if you are using vllm, upgrade to 0.7.3 or later should solve this problem.

I tried myself with DeepSeek-R1-Distill-Qwen-32B, worked.

@xring commented on GitHub (Feb 28, 2025): if you are using `vllm`, upgrade to 0.7.3 or later should solve this problem. https://github.com/vllm-project/vllm/issues/13375#issuecomment-2676496362 I tried myself with `DeepSeek-R1-Distill-Qwen-32B`, worked.

bsakaguchi commented

Author

Owner

@lpb1 commented on GitHub (Feb 28, 2025):

if you are using vllm, upgrade to 0.7.3 or later should solve this problem.

vllm-project/vllm#13375 (comment)

I tried myself with DeepSeek-R1-Distill-Qwen-32B, worked.

The reasoning_content is worked, but <think></think> tag still lost the first one

@lpb1 commented on GitHub (Feb 28, 2025): > if you are using `vllm`, upgrade to 0.7.3 or later should solve this problem. > > [vllm-project/vllm#13375 (comment)](https://github.com/vllm-project/vllm/issues/13375#issuecomment-2676496362) > > I tried myself with `DeepSeek-R1-Distill-Qwen-32B`, worked. The `reasoning_content` is worked, but `<think></think>` tag still lost the first one

bsakaguchi commented

2026-10-26 21:18:34 +00:00

Author

Owner

@xring commented on GitHub (Feb 28, 2025):

if you are using vllm, upgrade to 0.7.3 or later should solve this problem.
vllm-project/vllm#13375 (comment)
I tried myself with DeepSeek-R1-Distill-Qwen-32B, worked.

The reasoning_content is worked, but <think></think> tag still lost the first one

true, we use reasoning_content only.

@xring commented on GitHub (Feb 28, 2025): > > if you are using `vllm`, upgrade to 0.7.3 or later should solve this problem. > > [vllm-project/vllm#13375 (comment)](https://github.com/vllm-project/vllm/issues/13375#issuecomment-2676496362) > > I tried myself with `DeepSeek-R1-Distill-Qwen-32B`, worked. > > The `reasoning_content` is worked, but `<think></think>` tag still lost the first one true, we use `reasoning_content` only.

bsakaguchi commented

Author

Owner

@Modas-Li commented on GitHub (Mar 1, 2025):

same issue

@Modas-Li commented on GitHub (Mar 1, 2025): same issue

bsakaguchi commented

2026-10-26 21:18:34 +00:00

Author

Owner

@wifewriter commented on GitHub (Mar 2, 2025):

ollama本地部署deepseek r1也出现了这个问题，模型输出没有一开始的标签，应该怎么编写modelfile呢？

@wifewriter commented on GitHub (Mar 2, 2025): ollama本地部署deepseek r1也出现了这个问题，模型输出没有一开始的<think>标签，应该怎么编写modelfile呢？

bsakaguchi commented

Author

Owner

@Liusuqing commented on GitHub (Mar 5, 2025):

xinference 里也一样。

Xinference Chat Bot : deepseek-r1-distill-qwen 🚀
Model ID: deepseek-r1-distill-qwen
Model Size: 14 Billion Parameters
Model Format: awq
Model Quantization: Int4

您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助。
</think>

您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助

@Liusuqing commented on GitHub (Mar 5, 2025): xinference 里也一样。 ``` Xinference Chat Bot : deepseek-r1-distill-qwen 🚀 Model ID: deepseek-r1-distill-qwen Model Size: 14 Billion Parameters Model Format: awq Model Quantization: Int4 ``` ``` 您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助。 </think> 您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助 ```

bsakaguchi commented

Author

Owner

@shuzhiweb commented on GitHub (Mar 11, 2025):

sglang 用户如何解决这个问题？

升级sglang最新版, 并保留chat_template 中的 think标签

@shuzhiweb commented on GitHub (Mar 11, 2025): > sglang 用户如何解决这个问题？升级sglang最新版, 并保留chat_template 中的 think标签

bsakaguchi commented

Author

Owner

@Liusuqing commented on GitHub (Mar 11, 2025):

sglang 用户如何解决这个问题？

升级sglang最新版, 并保留chat_template 中的 think标

好像说反了吧？
另外，模型不同，解决方法好像也有差异。
以 qwen 32 4bit量化版为例，直接删除

@Liusuqing commented on GitHub (Mar 11, 2025): > > sglang 用户如何解决这个问题？ > > 升级sglang最新版, 并保留chat_template 中的 think标好像说反了吧？另外，模型不同，解决方法好像也有差异。以 qwen 32 4bit量化版为例，直接删除 </think>

bsakaguchi commented

Author

Owner

@shuzhiweb commented on GitHub (Mar 11, 2025):

sglang 用户如何解决这个问题？

升级sglang最新版, 并保留chat_template 中的 think标

好像说反了吧？另外，模型不同，解决方法好像也有差异。以 qwen 32 4bit量化版为例，直接删除

你说的可能是对的，目前我们用的是满血版的，以上是我们的解决方案

@shuzhiweb commented on GitHub (Mar 11, 2025): > > > sglang 用户如何解决这个问题？ > > > > > > 升级sglang最新版, 并保留chat_template 中的 think标 > > 好像说反了吧？另外，模型不同，解决方法好像也有差异。以 qwen 32 4bit量化版为例，直接删除你说的可能是对的，目前我们用的是满血版的，以上是我们的解决方案

bsakaguchi commented

Author

Owner

@CiaranYoung commented on GitHub (Mar 14, 2025):

直接vllm离线推理好像也没有

@CiaranYoung commented on GitHub (Mar 14, 2025): 直接vllm离线推理好像也没有

bsakaguchi commented

Author

Owner

@longshine commented on GitHub (Apr 7, 2025):

如果是 vllm 部署，加上这个参数：
--enable-reasoning --reasoning-parser deepseek_r1
亲测 vllm v0.8.3 有效

@longshine commented on GitHub (Apr 7, 2025): 如果是 vllm 部署，加上这个参数： `--enable-reasoning --reasoning-parser deepseek_r1` 亲测 vllm v0.8.3 有效 See also: https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#quickstart

bsakaguchi commented

Author

Owner

@Modas-Li commented on GitHub (Apr 7, 2025):

Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

@Modas-Li commented on GitHub (Apr 7, 2025): Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

bsakaguchi commented

Author

Owner

@sgr-china commented on GitHub (Apr 24, 2025):

直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。
加上系统提示词:
think first and then answer user's question as below format every time:
<think>reasoning content</think>
content
可以大幅提高模型思考的概率
系统提示词在哪里加？
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B@3865e12a1e

直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。

这个确实有效，vLLM 0.6.3，DeepSeek-R1-Distill-Qwen-14B和QwQ-32B都有效

@sgr-china commented on GitHub (Apr 24, 2025): > > > 直接修改tokenizer_config.json里的chat-template，去掉末尾的\n，vLLM重新部署一下就可以了。 > > > 加上系统提示词: > > > ``` > > > think first and then answer user's question as below format every time: > > > <think>reasoning content</think> > > > content > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 可以大幅提高模型思考的概率 > > > > > > 系统提示词在哪里加？ > > https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/commit/3865e12a1eb7cbd641ab3f9dfc28c588c6b0c1e9 > > 直接查看文本差异，拿到之前的，修改tokenizer_config.json文件，重新启动模型就不需要这么麻烦了。这个确实有效，vLLM 0.6.3，DeepSeek-R1-Distill-Qwen-14B和QwQ-32B都有效

bsakaguchi commented

Author

Owner

@lhhssd commented on GitHub (Apr 27, 2025):

当然这样并不是一个好的解决问题的方法，因为官方说：

此外，我们观察到 DeepSeek-R1 系列模型在响应某些查询时往往会绕过思维模式（即输出），这可能会对模型的性能产生不利影响。为了确保模型进行彻底的推理，我们建议强制模型在每次输出的开头启动其响应。<think>\n\n</think>``<think>\n

请问， \n\n<think>\n 添加到具体哪个位置呢？？是tokenizer_config.json文件的chat_template开头还是末尾的'<｜Assistant｜><think>\\n' 改为'<｜Assistant｜><think>\n\n</think>\n' 呢？？？

@lhhssd commented on GitHub (Apr 27, 2025): > 当然这样并不是一个好的解决问题的方法，因为官方说： > > > 此外，我们观察到 DeepSeek-R1 系列模型在响应某些查询时往往会绕过思维模式（即输出），这可能会对模型的性能产生不利影响。为了确保模型进行彻底的推理，我们建议强制模型在每次输出的开头启动其响应。`<think>\n\n</think>``<think>\n` 请问， <think>\n\n</think>``<think>\n 添加到具体哪个位置呢？？是tokenizer_config.json文件的chat_template开头还是末尾的'<｜Assistant｜><think>\\n' 改为'<｜Assistant｜><think>\n\n</think>``<think>\n' 呢？？？

bsakaguchi commented

Author

Owner

@T-DeBuger commented on GitHub (May 15, 2025):

起因：msr2000 commited on Feb 9 更新了 tokenizer_config.json
huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B@0d6d11a6ea

如果遇到回答中的 <think> 丢失问题，可以修改模型文件夹下的 tokenizer_config.json 文件。
将末尾的 {{'<｜Assistant｜><think>\\n'}} 改为 {{'<｜Assistant｜>'}} 即可，也就是删除 '<think>\\n' 部分。

@T-DeBuger commented on GitHub (May 15, 2025): _起因：msr2000 commited on Feb 9 更新了 tokenizer_config.json_ https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B/commit/0d6d11a6ea1187363aa7b78543f824fc02e06b14 ``` 如果遇到回答中的 <think> 丢失问题，可以修改模型文件夹下的 tokenizer_config.json 文件。将末尾的 {{'<｜Assistant｜><think>\\n'}} 改为 {{'<｜Assistant｜>'}} 即可，也就是删除 '<think>\\n' 部分。 ```

bsakaguchi commented

Author

Owner

@Modas-Li commented on GitHub (May 15, 2025):

Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

@Modas-Li commented on GitHub (May 15, 2025): Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

bsakaguchi commented

Author

Owner

@HZOK2022 commented on GitHub (May 15, 2025):

如果是 vllm 部署，加上这个参数： --enable-reasoning --reasoning-parser deepseek_r1 亲测 vllm v0.8.3 有效

See also: https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#quickstart

vllm 部署时在--reasoning-parser deepseek_r1前加上--enable-reasoning，推理内容会输出到reasoning_content字段内。并且不需要修改tokenizer_config.json中的模板，即不需要删掉think标签，也能正常输出。

vllm version: 0.8.5.post1

@HZOK2022 commented on GitHub (May 15, 2025): > 如果是 vllm 部署，加上这个参数： `--enable-reasoning --reasoning-parser deepseek_r1` 亲测 vllm v0.8.3 有效 > > See also: https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#quickstart vllm 部署时在--reasoning-parser deepseek_r1前加上--enable-reasoning，推理内容会输出到reasoning_content字段内。并且不需要修改tokenizer_config.json中的模板，即不需要删掉think标签，也能正常输出。 vllm version: 0.8.5.post1 ![Image](https://github.com/user-attachments/assets/1f948b9b-7d45-4461-919e-1a59a0b43cec)

bsakaguchi commented

Author

Owner

@llc2peak commented on GitHub (May 20, 2025):

使用ollama部署的，也是一样这个问题，开始没有，而且好像并不只是没有think，还会丢失一小段话，请问要怎么解决

@llc2peak commented on GitHub (May 20, 2025): 使用ollama部署的，也是一样这个问题，开始没有<think>，而且好像并不只是没有think，还会丢失一小段话，请问要怎么解决

bsakaguchi commented

Author

Owner

@github-actions[bot] commented on GitHub (Jun 22, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Jun 22, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

Author

Owner

@Modas-Li commented on GitHub (Jun 22, 2025):

Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

@Modas-Li commented on GitHub (Jun 22, 2025): Thank you for your email !您好，你的邮件我已收到，我会尽快给你回复。

bsakaguchi commented

Author

Owner

@github-actions[bot] commented on GitHub (Jul 25, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Jul 25, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented