mirror of
https://github.com/deepseek-ai/DeepSeek-Coder.git
synced 2026-11-05 00:57:19 +00:00
[GH-ISSUE #99] How to extended window size during train step2? #85
Labels
No labels
documentation
ENV
pull-request
research question
WIP
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-Coder#85
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jiejie1993 on GitHub (Jan 17, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/99
https://github.com/deepseek-ai/DeepSeek-Coder#model-trainingdescibe "Further Pre-training using an extended 16K window size on an additional 200B tokens", How to extend the window size during training? Just modify max_length and max_position_embeddings in the “config.json” file? or what I need to do?@guoday commented on GitHub (Jan 30, 2024):
please check our technical report. https://arxiv.org/pdf/2401.14196.pdf
@tangbo-sh commented on GitHub (Jul 9, 2024):
The technical report shows that only 8B data is used to train the long context, while the readme shows that 200B is trained on the 16K window. So I am a little confused here. If the pretrain truncation does have 200B data for 16K window training, then when was the base frequency and scaling factor of the rope modified? Thank you.
Content in the technical report: "The model underwent an additional
1000 steps of training, using a batch size of 512 and a sequence length of 16K. "
@guoday