[GH-ISSUE #570] 无法复现DeepSeek R1官方公布的在cevel、mmlu、math500数据集上的精度 #457

New issue

Closed

opened 2026-10-26 21:21:38 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:21:38 +00:00

Owner

Originally created by @Keithwwa on GitHub (Mar 13, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/570

个人使用opencompass，测DS R1在cevel、mmlu、math500数据集尝试复现官方精度，结果和官方差1~4分，请告知可能的原因。

Originally created by @Keithwwa on GitHub (Mar 13, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/570 个人使用opencompass，测DS R1在cevel、mmlu、math500数据集尝试复现官方精度，结果和官方差1~4分，请告知可能的原因。

bsakaguchi closed this issue

2026-10-26 21:21:38 +00:00

bsakaguchi commented

2026-10-26 21:21:39 +00:00

Author

Owner

@lzxzy commented on GitHub (Mar 15, 2025):

It's the same issue, Can anyone answer the question.

@lzxzy commented on GitHub (Mar 15, 2025): It's the same issue, Can anyone answer the question.

bsakaguchi commented

2026-10-26 21:21:39 +00:00

Author

Owner

@zhangshan123 commented on GitHub (Mar 25, 2025):

楼主，请问你知道Deepseek给出来的分数是基于opencompass跑出来的吗？还是结合其他tool跑的？

@zhangshan123 commented on GitHub (Mar 25, 2025): 楼主，请问你知道Deepseek给出来的分数是基于opencompass跑出来的吗？还是结合其他tool跑的？

bsakaguchi commented

2026-10-26 21:21:40 +00:00

Author

Owner

@Keithwwa commented on GitHub (Apr 8, 2025):

https://gitee.com/aisbench/benchmark
请尝试这个测评工具，我们在部分常用数据集比如math500 aime2024 ceval等数据集上复现了R1的评估结果
Please try this evaluation tool. We have reproduced the evaluation results of R1 on some common datasets such as math500 aime2024 ceval and other datasets.

@Keithwwa commented on GitHub (Apr 8, 2025): https://gitee.com/aisbench/benchmark 请尝试这个测评工具，我们在部分常用数据集比如math500 aime2024 ceval等数据集上复现了R1的评估结果 Please try this evaluation tool. We have reproduced the evaluation results of R1 on some common datasets such as math500 aime2024 ceval and other datasets.