mirror of
https://github.com/deepseek-ai/DeepSeek-Coder.git
synced 2026-11-05 00:57:19 +00:00
[GH-ISSUE #9] Impact of repo-level code concat #7
Labels
No labels
documentation
ENV
pull-request
research question
WIP
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-Coder#7
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @feiyuvl on GitHub (Nov 3, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/9
Very impressive work. Does the repo-level code conact improve humaneval result?
@guoday commented on GitHub (Nov 4, 2023):
In our experiment, repo-level code would slightly hurt performance of benchmark. However, it will enhance long context modeling and repo-level completion.
@feiyuvl commented on GitHub (Nov 6, 2023):
@guoday Thanx for your reply. Another question, how many epochs do you train on the 2T tokens dataset?
@guoday commented on GitHub (Nov 6, 2023):
4-5 epochs
@feiyuvl commented on GitHub (Nov 6, 2023):
So the total training tokens is about 10T? Amazing
@guoday commented on GitHub (Nov 6, 2023):
No. The total data is about 400-500B. We train the model for 4-5 epochs, resulting 2T.
@feiyuvl commented on GitHub (Nov 6, 2023):
This amount is equal to the starcoder dataset. Do you possibly resuse the stack dataset, and apply a more advanced data filter alogrithm, or do you download more code data than stack, and implment more stricted data cleaning algorithms?
@feiyuvl commented on GitHub (Nov 6, 2023):
Deepseek use the repo-level files, so it can't resuse the stack dataset. Which is the key contrbution to this amazing improvment over starcoder, code quaility filtering, second pretraining or something else?
@guoday commented on GitHub (Nov 6, 2023):
The primary contributions come from many aspects, including data collection, deduplication, filtering, the total number of tokens for pre-training, and the methods of pre-training. The optimization of each process is essential for enhancing the model. We will analyze this further in our later technical reports.
@feiyuvl commented on GitHub (Nov 6, 2023):
Thank you, looking forward to your technical reports
@wentinghome commented on GitHub (Jul 31, 2024):
Hey, thanks for the great work! can you share the script of dependency parsing and repo level dedup?