[GH-ISSUE #814] 多语种MMMLU测试集结果复现求助 #472

New issue

Closed

opened 2026-10-26 21:21:51 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:21:51 +00:00

Owner

Originally created by @TangPeiwang on GitHub (Mar 26, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/814

你好，我尝试着复现base模型在mmmlu上的结果，发现总是相差了2-3个点左右。
为了更好地理解可能存在的差异，我希望能获得以下方面的更多信息：

您们对 MMMLU 的具体构建流程和设置。
是否选择OpenAI开源的所有MMMLU 测试集（具体评测的是哪些语种），5-shot的来源，计算ppl是否需要Length Normalization，是否包含COT。
如果可以的话，希望能提供您在内部评估中使用的示例输入/输出或参考测试集，以确保我们的实现与您方法的一致性。
感谢你们的帮助。

Originally created by @TangPeiwang on GitHub (Mar 26, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/814 你好，我尝试着复现base模型在mmmlu上的结果，发现总是相差了2-3个点左右。为了更好地理解可能存在的差异，我希望能获得以下方面的更多信息： 1. 您们对 MMMLU 的具体构建流程和设置。 2. 是否选择OpenAI开源的所有MMMLU 测试集（具体评测的是哪些语种），5-shot的来源，计算ppl是否需要Length Normalization，是否包含COT。 3. 如果可以的话，希望能提供您在内部评估中使用的示例输入/输出或参考测试集，以确保我们的实现与您方法的一致性。感谢你们的帮助。

bsakaguchi

2026-10-26 21:21:51 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:21:52 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 26, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Apr 26, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:21:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 10, 2025):

false

@github-actions[bot] commented on GitHub (May 10, 2025): false