mirror of
https://github.com/deepseek-ai/DeepSeek-R1.git
synced 2026-11-05 00:47:42 +00:00
[GH-ISSUE #14] Request for the Release of the SFT Dataset for Distillation #10
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-R1#10
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jakeju92 on GitHub (Jan 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/14
Hi,
Congratulations on such a fantastic piece of work!
In Section 2.4 of your paper, you mention employing an 800k curated dataset for SFT on the LLama and Qwen models. I was wondering if there is any possibility of open-sourcing this dataset? It would be immensely valuable for the community and could greatly aid related research efforts.
@corbt commented on GitHub (Jan 21, 2025):
Also, could you clarify what "distillation" means in this context? Were the distilled models simply trained using standard SFT on the generated traces/answers from Deepseek R1? Or were they trained with logit-level distillation of some kind?
@hancheolcho commented on GitHub (Jan 22, 2025):
@corbt As I understand, they ran SFT on the 800k sft data.
@EoinKenny commented on GitHub (Feb 5, 2025):
Yes please this is exactly what I need too, i want to train interpretable by design LLMs that can reason with good performance, this would be perfect to help that research goal. Please open source this data so we can replicate the distillation process.
@AnnaGolubeva commented on GitHub (Feb 19, 2025):
+1
@lw309637554 commented on GitHub (Feb 25, 2025):
+1
@github-actions[bot] commented on GitHub (Apr 13, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@DA21S321D commented on GitHub (Apr 15, 2025):
mysterous so-called "distillation" i will never know, after searching the entire web tutorial :(
1.does the distillation compute the loss between teacher and student logits ?
2.or does it just collect a high-quality SFT data using rejection sampling and then perform SFT on the small parameters model?
3.does distillation related to lora? ( i have seen many tutorials teaching us reproducing "distillation" by lora fine-tuning)
@github-actions[bot] commented on GitHub (Jun 14, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@github-actions[bot] commented on GitHub (Jun 29, 2025):
false