mirror of
https://github.com/deepseek-ai/DeepSeek-R1.git
synced 2026-11-05 00:47:42 +00:00
[GH-ISSUE #335] How does the Deepseekr1-zero model distillation work with base models like LLaMA or Qwen? #259
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-R1#259
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @QuantaoYao on GitHub (Feb 8, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/335
I have some confusion regarding the distillation implementation details of deepseekr1. The reference answers are generated by a teacher model (such as deepseekr1-zero) and knowledge is transferred through softened labels (such as probability distributions). However, when looking at the specific reproduction, it seems to adopt a one-question-multiple-answers format. For a given question, deepseekr1-zero outputs multiple answers, and then these answers with tags are used for Rejection Sampling and traditional SFT. This does not seem to reflect the process of label softening. How is this specifically implemented?
@github-actions[bot] commented on GitHub (Mar 26, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@github-actions[bot] commented on GitHub (May 1, 2025):
false