[GH-ISSUE #1025] Reproduction Inquiry: DeepSeek-V3.1 AIME2024 & AIME2025 Results #629

New issue

Closed

opened 2026-10-26 21:24:18 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:24:18 +00:00

Owner

Originally created by @HermitSun on GitHub (Oct 27, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/1025

Hi DeepSeek Team,

First of all, thank you for open-sourcing DeepSeek-V3.1 and sharing the impressive benchmark results.
Our team has been working to reproduce the results reported in your official release, but we’ve encountered some discrepancies that we’d like to understand better.

Reproduction Setup

Inference: vLLM 0.11.0 on H200 * 8
Model weights: from the official Hugging Face repository (commit c0781d0, main branch)

Sampling parameters:

temperature = 0.6
top_p = 0.95
max_tokens = 32768
k = 64  # for pass@1 computation, following DeepSeek-R1 paper

Datasets:
- HuggingFaceH4/aime_2024
- opencompass/AIME2025

Prompt format:

Solve the following math problem step by step. The last line of your response should be of the form Answer: $ANSWER (without quotes) where $ANSWER is the answer to the problem.

{question}

Remember to put your answer on its own line after "Answer:", and you do not need to use a \boxed command.

Our Reproduction Results

Mode	AIME2024 (report → ours)	AIME2025 (report → ours)
Non-thinking	66.3 → 50.4	49.8 → 38.2
Thinking	93.1 → 50.7	88.4 → 79.7

We have double-checked dataset integrity, prompts, and sampling settings, but still observed a noticeable performance gap.

Questions

Could you please share a bit more about your evaluation configuration? For example:

The exact inference environment or framework version used
Whether any custom decoding, filtering, or re-ranking steps were applied
Any additional prompt preprocessing or formatting details

Any clarification would be greatly appreciated!
Thanks again for releasing such a high-quality model.

Originally created by @HermitSun on GitHub (Oct 27, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/1025 Hi DeepSeek Team, First of all, thank you for open-sourcing **DeepSeek-V3.1** and sharing the impressive benchmark results. Our team has been working to reproduce the results reported in your official release, but we’ve encountered some discrepancies that we’d like to understand better. --- ### Reproduction Setup - **Inference:** `vLLM 0.11.0` on H200 * 8 - **Model weights:** from the official [Hugging Face repository](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) (`commit c0781d0`, `main` branch) - **Sampling parameters:** ``` temperature = 0.6 top_p = 0.95 max_tokens = 32768 k = 64 # for pass@1 computation, following DeepSeek-R1 paper ``` - **Datasets:** - [HuggingFaceH4/aime_2024](https://huggingface.co/datasets/HuggingFaceH4/aime_2024) - [opencompass/AIME2025](https://huggingface.co/datasets/opencompass/AIME2025) - **Prompt format:** ```text Solve the following math problem step by step. The last line of your response should be of the form Answer: $ANSWER (without quotes) where $ANSWER is the answer to the problem. {question} Remember to put your answer on its own line after "Answer:", and you do not need to use a \boxed command. ``` --- ### Our Reproduction Results | Mode | AIME2024 (report → ours) | AIME2025 (report → ours) | |------|--------------------------|--------------------------| | Non-thinking | 66.3 → 50.4 | 49.8 → 38.2 | | Thinking | 93.1 → 50.7 | 88.4 → 79.7 | We have double-checked dataset integrity, prompts, and sampling settings, but still observed a noticeable performance gap. --- ### Questions Could you please share a bit more about your evaluation configuration? For example: - The exact inference environment or framework version used - Whether any custom decoding, filtering, or re-ranking steps were applied - Any additional prompt preprocessing or formatting details Any clarification would be greatly appreciated! Thanks again for releasing such a high-quality model.

bsakaguchi

2026-10-26 21:24:18 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:24:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Nov 27, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Nov 27, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:24:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Dec 11, 2025):

false

@github-actions[bot] commented on GitHub (Dec 11, 2025): false