deepseek-ai/DeepSeek-MoE

Fork 0

mirror of https://github.com/deepseek-ai/DeepSeek-MoE.git synced 2026-11-05 00:57:43 +00:00

[PR #54] fix: use the tokenizer's eos_token as the fine-tuning target terminator #51

New issue

Open

opened 2026-10-26 21:02:09 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:02:09 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepSeek-MoE/pull/54
Author: @PerryLink
Created: 9/17/2026
Status: 🔄 Open

Base: main ← Head: fix/eot-token-from-tokenizer

📝 Commits (1)

0716312 fix: use the tokenizer's eos_token as the fine-tuning target terminator

📊 Changes

1 file changed (+13 additions, -4 deletions)

View changed files

📝 finetune/finetune.py (+13 -4)

📄 Description

Summary

finetune/finetune.py appends the hard-coded string "<|EOT|>" to every training target,
but <|EOT|> is not a token in the DeepSeekMoE tokenizer. This PR derives the terminator
from the loaded tokenizer's eos_token instead, with an optional --eot_token override.

Fixes #37.

Problem

finetune/finetune.py:20 (at 66edeee) hard-codes:

EOT_TOKEN = "<|EOT|>"

and finetune/finetune.py:178 builds the targets as:

targets = [f"{output}\n{EOT_TOKEN}" for output in examples['output']]

EOT_TOKEN is used nowhere else in the file.

That string is not in this model's vocabulary:

deepseek-ai/deepseek-moe-16b-base / -chat: eos_token = pad_token = the
<|end of sentence|> special token (DeepSeek spells it with full-width vertical bars),
added_tokens_decoder is empty and additional_special_tokens is empty.
In tokenizer.json of deepseek-moe-16b-base the literal <|EOT|> does not occur at all;
its 15 added_tokens are the two <|begin/end of sentence|> special tokens (again spelled
with full-width bars) plus 13 single-character tokens. So <|EOT|> is not a token: it is
shredded into ordinary sub-word/byte tokens.

Because preprocess() keeps the target suffix inside the loss and the real eos never
appears in the labels, a model fine-tuned with this script never learns to end a turn.
That is exactly what was reported in #28, where the output degenerates into
<|EOT|>, <EOT>, <EOTOT> fragments repeated indefinitely; the resolution given there by
a maintainer was to use the tokenizer's eos_token.

The constant is a leftover from DeepSeek-Coder/finetune/finetune_deepseekcoder.py, where
line 14 is byte-identical (EOT_TOKEN = "<|EOT|>") and line 117 is byte-identical to line 178
here. It is correct for the Coder models (whose eos_token really is <|EOT|>) and wrong
for DeepSeekMoE.

Changes

Remove the module-level EOT_TOKEN constant.
train_tokenize_function(examples, tokenizer, eot_token=None) now resolves the terminator
at the point of use: eot_token = tokenizer.eos_token when no override is given, and raises
a clear ValueError if the tokenizer defines no eos_token (previously that case would have
written the literal string None into the targets).
Add --eot_token to the existing DataArguments dataclass, threaded through the existing
fn_kwargs plumbing.

The trailing \n before the terminator is kept: this script uses the Coder-style
### Instruction: / ### Response: prompt, which is also where the \n comes from. Note that
the official MoE chat template ('Assistant: ' + content + eos_token) has no \n before the
eos; if you would rather match that, it is a one-token change and I am happy to make it.

Effect on existing users

Already-trained checkpoints and adapters are unaffected: only new training runs are affected.
Re-running training now ends each target with the real eos, which is the point of the fix.
To reproduce the old behaviour byte-for-byte, pass --eot_token '<|EOT|>'.
If you cached a tokenized dataset, clear the cache directory once (the map fingerprint
should change with the function body, but clearing is the safe route).

Test plan

python -m py_compile finetune/finetune.py - passes (Python 3.12).
The patched train_tokenize_function was exercised with a stub preprocess:
the model's <|end of sentence|> token gives answer\n<|end of sentence|>; a Coder-style
tokenizer gives answer\n<|EOT|>; --eot_token overrides; no eos and no override raises.
Not run: an actual fine-tuning job (no GPU/DeepSpeed environment available to me), so the
end-to-end "the model now stops" claim is not empirically verified here.

Notes

The default model_name_or_path on line 39 is deepseek-ai/deepseek-moe-16b, which does not
resolve on the Hub (the existing repos are -base and -chat). Left untouched here to keep
this PR to one defect; flagging it since it may deserve its own fix.
PR #40 and #50 touch this same file, but in different regions; this change does not overlap.

AI assistance was used to prepare this patch (issue triage, evidence gathering, patch and test
drafting). The findings above were verified against the repository contents and the model
tokenizer files; please review as usual.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepSeek-MoE/pull/54 **Author:** [@PerryLink](https://github.com/PerryLink) **Created:** 9/17/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/eot-token-from-tokenizer` --- ### 📝 Commits (1) - [`0716312`](https://github.com/deepseek-ai/DeepSeek-MoE/commit/07163123d6557301282baa506e5a8a3c58733476) fix: use the tokenizer's eos_token as the fine-tuning target terminator ### 📊 Changes **1 file changed** (+13 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `finetune/finetune.py` (+13 -4) </details> ### 📄 Description ## Summary `finetune/finetune.py` appends the hard-coded string `"<|EOT|>"` to every training target, but `<|EOT|>` is not a token in the DeepSeekMoE tokenizer. This PR derives the terminator from the loaded tokenizer's `eos_token` instead, with an optional `--eot_token` override. Fixes #37. ## Problem `finetune/finetune.py:20` (at `66edeee`) hard-codes: EOT_TOKEN = "<|EOT|>" and `finetune/finetune.py:178` builds the targets as: targets = [f"{output}\n{EOT_TOKEN}" for output in examples['output']] `EOT_TOKEN` is used nowhere else in the file. That string is not in this model's vocabulary: * `deepseek-ai/deepseek-moe-16b-base` / `-chat`: `eos_token` = `pad_token` = the `<|end of sentence|>` special token (DeepSeek spells it with full-width vertical bars), `added_tokens_decoder` is empty and `additional_special_tokens` is empty. * In `tokenizer.json` of `deepseek-moe-16b-base` the literal `<|EOT|>` does not occur at all; its 15 `added_tokens` are the two `<|begin/end of sentence|>` special tokens (again spelled with full-width bars) plus 13 single-character tokens. So `<|EOT|>` is not a token: it is shredded into ordinary sub-word/byte tokens. Because `preprocess()` keeps the target suffix inside the loss and the real `eos` never appears in the labels, a model fine-tuned with this script never learns to end a turn. That is exactly what was reported in #28, where the output degenerates into `<|EOT|>`, `<EOT>`, `<EOTOT>` fragments repeated indefinitely; the resolution given there by a maintainer was to use the tokenizer's `eos_token`. The constant is a leftover from `DeepSeek-Coder/finetune/finetune_deepseekcoder.py`, where line 14 is byte-identical (`EOT_TOKEN = "<|EOT|>"`) and line 117 is byte-identical to line 178 here. It is correct for the Coder models (whose `eos_token` really is `<|EOT|>`) and wrong for DeepSeekMoE. ## Changes * Remove the module-level `EOT_TOKEN` constant. * `train_tokenize_function(examples, tokenizer, eot_token=None)` now resolves the terminator at the point of use: `eot_token = tokenizer.eos_token` when no override is given, and raises a clear `ValueError` if the tokenizer defines no `eos_token` (previously that case would have written the literal string `None` into the targets). * Add `--eot_token` to the existing `DataArguments` dataclass, threaded through the existing `fn_kwargs` plumbing. The trailing `\n` before the terminator is kept: this script uses the Coder-style `### Instruction:` / `### Response:` prompt, which is also where the `\n` comes from. Note that the official MoE chat template (`'Assistant: ' + content + eos_token`) has no `\n` before the `eos`; if you would rather match that, it is a one-token change and I am happy to make it. ## Effect on existing users * Already-trained checkpoints and adapters are unaffected: only new training runs are affected. * Re-running training now ends each target with the real `eos`, which is the point of the fix. * To reproduce the old behaviour byte-for-byte, pass `--eot_token '<|EOT|>'`. * If you cached a tokenized dataset, clear the cache directory once (the `map` fingerprint should change with the function body, but clearing is the safe route). ## Test plan * `python -m py_compile finetune/finetune.py` - passes (Python 3.12). * The patched `train_tokenize_function` was exercised with a stub `preprocess`: the model's `<|end of sentence|>` token gives `answer\n<|end of sentence|>`; a Coder-style tokenizer gives `answer\n<|EOT|>`; `--eot_token` overrides; no `eos` and no override raises. * Not run: an actual fine-tuning job (no GPU/DeepSpeed environment available to me), so the end-to-end "the model now stops" claim is not empirically verified here. ## Notes * The default `model_name_or_path` on line 39 is `deepseek-ai/deepseek-moe-16b`, which does not resolve on the Hub (the existing repos are `-base` and `-chat`). Left untouched here to keep this PR to one defect; flagging it since it may deserve its own fix. * PR #40 and #50 touch this same file, but in different regions; this change does not overlap. --- AI assistance was used to prepare this patch (issue triage, evidence gathering, patch and test drafting). The findings above were verified against the repository contents and the model tokenizer files; please review as usual. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 21:02:09 +00:00

No labels

pull-request

WIP

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-MoE#51

No description provided.

Rows
Columns