[GH-ISSUE #69] Question regarding Llama base model #56

New issue

Closed

opened 2026-10-26 21:12:28 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 21:12:28 +00:00

Owner

Originally created by @spaidataiga on GitHub (Jan 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/69

Hi,

This question partially relates to Issue #14 , as it might be answered by seeing the data used for knowledge distillation. According to the README and Paper provided, the base model for DeepSeek-R1-Distill-Llama-8B is the original base model Llama-3.1-8B, i.e. not the instruct-finetuned version with chat functionality. This is interesting given that the Distilled version appears to be instruction fine-tuned and accepts chat templates. Furthermore, all the other base models (including the other Llama model) are instruction fine-tuned.

I want to confirm that the non-instruction fine-tuned version of Llama-3.1-8B was the base model, and that all the instruction-following and chat-functionality abilities were distilled from DeepSeek-R1? If so, why? It makes some comparisons between the base model and the distilled model difficult, as they must be differentially prompted.

Thanks!

Originally created by @spaidataiga on GitHub (Jan 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/69 Hi, This question partially relates to Issue #14 , as it might be answered by seeing the data used for knowledge distillation. According to the README and Paper provided, the base model for DeepSeek-R1-Distill-Llama-8B is the original base model [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), i.e. not the instruct-finetuned version with chat functionality. This is interesting given that the Distilled version appears to be instruction fine-tuned and accepts chat templates. Furthermore, all the other base models (including the other Llama model) are instruction fine-tuned. I want to confirm that the non-instruction fine-tuned version of Llama-3.1-8B was the base model, and that all the instruction-following and chat-functionality abilities were distilled from DeepSeek-R1? If so, why? It makes some comparisons between the base model and the distilled model difficult, as they must be differentially prompted. Thanks!

bsakaguchi

2026-10-26 21:12:28 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:12:29 +00:00

Author

Owner

@davidogundepo commented on GitHub (Jan 27, 2025):

Yes, I can confirm the base model choice is intentional.

Using non-instruction-tuned Llama-3.1-8B as the base allows us to better evaluate how effectively the knowledge distillation process transfers capabilities from DeepSeek-R1. The instruction-following and chat abilities in the distilled model come purely from the distillation process, not the base model.

This approach actually helps validate the distillation method's effectiveness, since we can clearly see what capabilities were successfully transferred from teacher to student model. While it does require different prompting approaches for comparisons, this helps isolate and measure the impact of the distillation process itself.

@davidogundepo commented on GitHub (Jan 27, 2025): Yes, I can confirm the base model choice is intentional. Using non-instruction-tuned Llama-3.1-8B as the base allows us to better evaluate how effectively the knowledge distillation process transfers capabilities from DeepSeek-R1. The instruction-following and chat abilities in the distilled model come purely from the distillation process, not the base model. This approach actually helps validate the distillation method's effectiveness, since we can clearly see what capabilities were successfully transferred from teacher to student model. While it does require different prompting approaches for comparisons, this helps isolate and measure the impact of the distillation process itself.

bsakaguchi commented

2026-10-26 21:12:29 +00:00

Author

Owner

@wudidiandian commented on GitHub (Feb 5, 2025):

I tried to ask a few questions about python code generation, but the distilled model is much worse than the original basic model, so I don’t quite understand the meaning of distilling a small parameter model.

@wudidiandian commented on GitHub (Feb 5, 2025): I tried to ask a few questions about python code generation, but the distilled model is much worse than the original basic model, so I don’t quite understand the meaning of distilling a small parameter model.

bsakaguchi commented

2026-10-26 21:12:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Mar 15, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Mar 15, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:12:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 12, 2025):

false

@github-actions[bot] commented on GitHub (Apr 12, 2025): false