[GH-ISSUE #48] Is the compute calculation wrong for Chinchilla in the paper? #31

New issue

Open

opened 2026-10-26 21:01:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:01:53 +00:00

Owner

Originally created by @yzlnew on GitHub (Apr 30, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-LLM/issues/48

From the paper, Eq.2 list Chinchilla compute calculation as

6N_2 = 72 n_\text{layer}d_\text{model}^2 + 6n_\text{vocab}d_\text{model}

The first term comes from the 6ND estimate for non-embedding FLOPs (exclude lm_head parameters as well, maybe because of the tied embeddings), but the second term is not what Chinchilla used to calculate embedding FLOPs, see Appendix F from the Chinchilla paper, total forward pass FLOPs include embeddings and logits calculations.

So, N_2 should be larger than what is used in the paper (double the second term)?

Originally created by @yzlnew on GitHub (Apr 30, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-LLM/issues/48 From [the paper](https://arxiv.org/abs/2401.02954), Eq.2 list Chinchilla compute calculation as $$6N_2 = 72 n_\text{layer}d_\text{model}^2 + 6n_\text{vocab}d_\text{model}$$ The first term comes from the $6ND$ estimate for non-embedding FLOPs (exclude lm_head parameters as well, maybe because of the tied embeddings), but the second term is not what Chinchilla used to calculate embedding FLOPs, see Appendix F from the Chinchilla paper, total forward pass FLOPs include embeddings and logits calculations. So, $N_2$ should be larger than what is used in the paper (double the second term)?

bsakaguchi commented

2026-10-26 21:01:53 +00:00

Author

Owner

@yzlnew commented on GitHub (Apr 30, 2024):

Starting from Table 3, I re-add the missing part to N_2 and recalculate 6N_2/M, the results are [2.19, 1.50, 1.01, 1.01, 0.97, 0.95, 0.95], which is much closer to 1.0 for larger models.

Could you share the data of Figure 6 for further study? Thanks!

@yzlnew commented on GitHub (Apr 30, 2024): Starting from Table 3, I re-add the missing part to $N_2$ and recalculate $6N_2/M$, the results are [2.19, 1.50, 1.01, 1.01, 0.97, 0.95, 0.95], which is much closer to 1.0 for larger models. Could you share the data of Figure 6 for further study? Thanks!

bsakaguchi referenced this issue

2026-10-26 21:02:38 +00:00

[PR #31] [MERGED] Update README.md #94