[GH-ISSUE #319] DeepSeek Train pipeline Problem #247

New issue

Closed

opened 2026-10-26 21:17:51 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:17:51 +00:00

Owner

Originally created by @SCismycat on GitHub (Feb 8, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/319

Hi，
From the paper，Its said：

Specifically, we begin by collecting thousands of cold-start data to fine-tune the DeepSeek-V3-Base model. Following this, we perform reasoning-oriented RL like DeepSeek-R1-Zero. Upon nearing convergence in the RL process, we create new SFT data through rejection sampling on the RL checkpoint, combined with supervised data from DeepSeek-V3 in domains such as writing, factual QA, and self-cognition, and then retrain the DeepSeek-V3-Base model.

and：

We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples. --From：Rejection Sampling and Supervised Fine-Tuning

So, the cold start in Step 1 and the RL in Step 2 of R1 are only for synthesizing 800K data. The actual initiation of the R1 training process is V3-base-model-->SFT (using the 800K data) --> reinforcement learning for all scenarios. Can it be understood this way?

Originally created by @SCismycat on GitHub (Feb 8, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/319 Hi， From the paper，Its said： > Specifically, we begin by collecting thousands of cold-start data to fine-tune the DeepSeek-V3-Base model. Following this, we perform reasoning-oriented RL like DeepSeek-R1-Zero. Upon nearing convergence in the RL process, we create new SFT data through rejection sampling on the RL checkpoint, combined with supervised data from DeepSeek-V3 in domains such as writing, factual QA, and self-cognition, and then retrain the DeepSeek-V3-Base model. and： > We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples. --From：Rejection Sampling and Supervised Fine-Tuning So, the cold start in Step 1 and the RL in Step 2 of R1 are only for synthesizing 800K data. The actual initiation of the R1 training process is V3-base-model-->SFT (using the 800K data) --> reinforcement learning for all scenarios. Can it be understood this way? ![Image](https://github.com/user-attachments/assets/11d814d2-21cb-4824-93c5-392c08868cfd)

bsakaguchi

2026-10-26 21:17:51 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:17:52 +00:00

Author

Owner

@sflowing commented on GitHub (Feb 13, 2025):

Good question, and I have the exact same question!

Is this pipeline right?

@sflowing commented on GitHub (Feb 13, 2025): Good question, and I have the exact same question! ![Image](https://github.com/user-attachments/assets/29909a79-fd05-491e-b31d-7d34c8f2cc5c) Is this pipeline right?

bsakaguchi commented

2026-10-26 21:17:52 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Mar 27, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Mar 27, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:17:52 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 3, 2025):

false

@github-actions[bot] commented on GitHub (May 3, 2025): false