[GH-ISSUE #33] TriviaQA结果复现求助 #19

New issue

Open

opened 2026-10-26 21:01:43 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 21:01:43 +00:00

Owner

Originally created by @HYZ17 on GitHub (Jan 14, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-LLM/issues/33

Originally assigned to: @DeepSeekPH on GitHub.

你好，我尝试着复现base模型（7B和67B）在TriviaQA上的结果。发现使用tech report 中的prompt格式，结果还是相差了7个点左右。请问可以提供复现的代码吗？感谢你的帮助。

Originally created by @HYZ17 on GitHub (Jan 14, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-LLM/issues/33 Originally assigned to: @DeepSeekPH on GitHub. 你好，我尝试着复现base模型（7B和67B）在TriviaQA上的结果。发现使用tech report 中的prompt格式，结果还是相差了7个点左右。请问可以提供复现的代码吗？感谢你的帮助。

bsakaguchi commented

2026-10-26 21:01:44 +00:00

Author

Owner

@luofuli commented on GitHub (Feb 4, 2024):

@hwxu20 @DeepSeekPH

@luofuli commented on GitHub (Feb 4, 2024): @hwxu20 @DeepSeekPH

bsakaguchi commented

2026-10-26 21:01:44 +00:00

Author

Owner

@hwxu20 commented on GitHub (Feb 5, 2024):

TriviaQA我们测试的是web的子集，实际评测时每个样本选择的few-shot example是随机从train里面挑选的，tech report中只是给出了其中的一个示例。
另外评估结果差7个点可能是对答案的后处理不一致，我们使用的后处理脚本供参考：

def normalize_answer(s):
    """Lower text and remove punctuation, articles and extra whitespace."""

    def remove_articles(text):
        return re.sub(r"\b(a|an|the)\b", " ", text)

    def white_space_fix(text):
        return " ".join(text.split())

    def handle_punc(text):
        exclude = set(string.punctuation + "".join(["‘", "’", "´", "`"]))
        return "".join(ch if ch not in exclude else " " for ch in text)

    def lower(text):
        return text.lower()

    def replace_underscore(text):
        return text.replace("_", " ")

    return white_space_fix(remove_articles(handle_punc(lower(replace_underscore(s))))).strip()

@hwxu20 commented on GitHub (Feb 5, 2024): TriviaQA我们测试的是web的子集，实际评测时每个样本选择的few-shot example是随机从train里面挑选的，tech report中只是给出了其中的一个示例。另外评估结果差7个点可能是对答案的后处理不一致，我们使用的后处理脚本供参考： ``` def normalize_answer(s): """Lower text and remove punctuation, articles and extra whitespace.""" def remove_articles(text): return re.sub(r"\b(a|an|the)\b", " ", text) def white_space_fix(text): return " ".join(text.split()) def handle_punc(text): exclude = set(string.punctuation + "".join(["‘", "’", "´", "`"])) return "".join(ch if ch not in exclude else " " for ch in text) def lower(text): return text.lower() def replace_underscore(text): return text.replace("_", " ") return white_space_fix(remove_articles(handle_punc(lower(replace_underscore(s))))).strip() ```

bsakaguchi commented

2026-10-26 21:01:45 +00:00

Author

Owner

@RoacherM commented on GitHub (Mar 18, 2024):

哪里可以看到你们的测评脚本？想复线一波？

@RoacherM commented on GitHub (Mar 18, 2024): 哪里可以看到你们的测评脚本？想复线一波？

bsakaguchi commented

2026-10-26 21:01:45 +00:00

Author

Owner

@hwxu20 commented on GitHub (Mar 21, 2024):