[GH-ISSUE #412] 关于DeepSeek-R1强化学习训练过程的一个小疑问 #324

New issue

Closed

opened 2026-10-26 21:19:25 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 21:19:25 +00:00

Owner

Originally created by @maohaotiancs on GitHub (Feb 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/412

您的论文中提到：
“In this work, we collect thousands of cold-start data to fine-tune the DeepSeek-V3-Base as the starting point for RL.”
“After fine-tuning DeepSeek-V3-Base on the cold start data, we apply the same large-scale reinforcement learning training process as employed in DeepSeek-R1-Zero.”
“When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round.”
“We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples.”
“To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.”
意思是在在DeepSeek-V3-Base上进行冷启动的微调，微调后的模型再进行和DeepSeek-R1-Zero一样的强化学习阶段，在强化学习收敛之后，收集该checkpoint生成的回复数据作为训练语料再返回到DeepSeek-V3-Base进行微调，最后在微调后的DeepSeek-V3-Base模型上再进行一次强化学习最终得到DeepSeek-R1？
那在前两步中得到的checkpoint仅仅是为了生成回复数据吗？最终的DeepSeek-R1模型实际是DeepSeek-V3-Base进行了一次800k样本的微调和一次强化学习微调得到的？

Originally created by @maohaotiancs on GitHub (Feb 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/412 您的论文中提到： “In this work, we collect thousands of cold-start data to fine-tune the DeepSeek-V3-Base as the starting point for RL.” “After fine-tuning DeepSeek-V3-Base on the cold start data, we apply the same large-scale reinforcement learning training process as employed in DeepSeek-R1-Zero.” “When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round.” “We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples.” “To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.” 意思是在在DeepSeek-V3-Base上进行冷启动的微调，微调后的模型再进行和DeepSeek-R1-Zero一样的强化学习阶段，在强化学习收敛之后，收集该checkpoint生成的回复数据作为训练语料再返回到DeepSeek-V3-Base进行微调，最后在微调后的DeepSeek-V3-Base模型上再进行一次强化学习最终得到DeepSeek-R1？那在前两步中得到的checkpoint仅仅是为了生成回复数据吗？最终的DeepSeek-R1模型实际是DeepSeek-V3-Base进行了一次800k样本的微调和一次强化学习微调得到的？

bsakaguchi closed this issue

2026-10-26 21:19:26 +00:00

bsakaguchi commented

2026-10-26 21:19:27 +00:00

Author

Owner

@noc-turne commented on GitHub (Feb 18, 2025):

同问

@noc-turne commented on GitHub (Feb 18, 2025): 同问

bsakaguchi commented

2026-10-26 21:19:27 +00:00

Author

Owner

@lesterlee89 commented on GitHub (Feb 20, 2025):

同问

@lesterlee89 commented on GitHub (Feb 20, 2025): 同问

bsakaguchi commented

2026-10-26 21:19:27 +00:00

Author

Owner

@cjw-d commented on GitHub (Mar 7, 2025):

同问

@cjw-d commented on GitHub (Mar 7, 2025): 同问

bsakaguchi commented

2026-10-26 21:19:27 +00:00

Author

Owner

@Peacer68 commented on GitHub (Apr 18, 2025):

你的理解是对的

您的论文中提到： “In this work, we collect thousands of cold-start data to fine-tune the DeepSeek-V3-Base as the starting point for RL.” “After fine-tuning DeepSeek-V3-Base on the cold start data, we apply the same large-scale reinforcement learning training process as employed in DeepSeek-R1-Zero.” “When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round.” “We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples.” “To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.” 意思是在在DeepSeek-V3-Base上进行冷启动的微调，微调后的模型再进行和DeepSeek-R1-Zero一样的强化学习阶段，在强化学习收敛之后，收集该checkpoint生成的回复数据作为训练语料再返回到DeepSeek-V3-Base进行微调，最后在微调后的DeepSeek-V3-Base模型上再进行一次强化学习最终得到DeepSeek-R1？那在前两步中得到的checkpoint仅仅是为了生成回复数据吗？最终的DeepSeek-R1模型实际是DeepSeek-V3-Base进行了一次800k样本的微调和一次强化学习微调得到的？

@Peacer68 commented on GitHub (Apr 18, 2025): 你的理解是对的 > 您的论文中提到： “In this work, we collect thousands of cold-start data to fine-tune the DeepSeek-V3-Base as the starting point for RL.” “After fine-tuning DeepSeek-V3-Base on the cold start data, we apply the same large-scale reinforcement learning training process as employed in DeepSeek-R1-Zero.” “When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round.” “We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples.” “To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.” 意思是在在DeepSeek-V3-Base上进行冷启动的微调，微调后的模型再进行和DeepSeek-R1-Zero一样的强化学习阶段，在强化学习收敛之后，收集该checkpoint生成的回复数据作为训练语料再返回到DeepSeek-V3-Base进行微调，最后在微调后的DeepSeek-V3-Base模型上再进行一次强化学习最终得到DeepSeek-R1？那在前两步中得到的checkpoint仅仅是为了生成回复数据吗？最终的DeepSeek-R1模型实际是DeepSeek-V3-Base进行了一次800k样本的微调和一次强化学习微调得到的？