[GH-ISSUE #84] 使用fim后human eval分数很低？ #71

New issue

Open

opened 2026-10-26 21:03:04 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 21:03:04 +00:00

Owner

Originally created by @nullxjx on GitHub (Dec 26, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/84

不使用fim，跑出来的分数跟官方给的比较接近，但是使用fim后分数很低，低了8个点，以下是我使用的fim格式，请问有啥问题吗？

def get_full_prompt(model_name, prompt_prefix, prompt, suffix="\n"):
    """
    get_full_prompt 通过匹配模型名，使用不同的FIM格式来准备prompt
    """
    output = prompt
    if "star-coder" in model_name.lower():
        ...
    elif "deep-seek" in model_name.lower() or "deepseek" in model_name.lower():
        template = Template("<｜fim▁begin｜>${preprefix}${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>")
        output = template.substitute(preprefix=prompt_prefix, prefix=prompt, suffix=suffix)
    else:
        if prompt_prefix:
            output = prompt_prefix + output
    return output

Originally created by @nullxjx on GitHub (Dec 26, 2023). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/84 不使用fim，跑出来的分数跟官方给的比较接近，但是使用fim后分数很低，低了8个点，以下是我使用的fim格式，请问有啥问题吗？ ``` def get_full_prompt(model_name, prompt_prefix, prompt, suffix="\n"): """ get_full_prompt 通过匹配模型名，使用不同的FIM格式来准备prompt """ output = prompt if "star-coder" in model_name.lower(): ... elif "deep-seek" in model_name.lower() or "deepseek" in model_name.lower(): template = Template("<｜fim▁begin｜>${preprefix}${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>") output = template.substitute(preprefix=prompt_prefix, prefix=prompt, suffix=suffix) else: if prompt_prefix: output = prompt_prefix + output return output ```

bsakaguchi commented

2026-10-26 21:03:05 +00:00

Author

Owner

@guoday commented on GitHub (Jan 26, 2024):

看起来格式有点问题，请看这个readme
https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion

@guoday commented on GitHub (Jan 26, 2024): 看起来格式有点问题，请看这个readme https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion

bsakaguchi commented

2026-10-26 21:03:05 +00:00

Author

Owner

@xidongyuandong commented on GitHub (Feb 29, 2024):

看起来格式有点问题，请看这个readme https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion

看起来跟官方文档是一致的，他这种 fim格式问题出在哪里?

@xidongyuandong commented on GitHub (Feb 29, 2024): > 看起来格式有点问题，请看这个readme https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion 看起来跟官方文档是一致的，他这种 fim格式问题出在哪里?

bsakaguchi commented

2026-10-26 21:03:05 +00:00

Author

Owner

@guoday commented on GitHub (Feb 29, 2024):

"<｜fim▁begin｜>${preprefix}${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>"

"<｜fim▁begin｜>${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>"

看起来格式有点问题，请看这个readme https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion

看起来跟官方文档是一致的，他这种 fim格式问题出在哪里?

"<｜fim▁begin｜>${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>" 多了${preprefix}

@guoday commented on GitHub (Feb 29, 2024): > "<｜fim▁begin｜>${preprefix}${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>" "<｜fim▁begin｜>${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>" > > 看起来格式有点问题，请看这个readme https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion > > 看起来跟官方文档是一致的，他这种 fim格式问题出在哪里? "<｜fim▁begin｜>${prefix}<｜fim▁hole｜>${suffix}<｜fim▁end｜>" 多了${preprefix}

bsakaguchi commented

2026-10-26 21:03:06 +00:00

Author

Owner

@xidongyuandong commented on GitHub (Feb 29, 2024):