[GH-ISSUE #130] Reproduce FIM Evaluation #114

New issue

Closed

opened 2026-10-26 21:03:41 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:03:41 +00:00

Owner

Originally created by @Hambaobao on GitHub (Feb 29, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/130

Hello, thanks a lot for your great work.
Could you kindly advise on how to reproduce the evaluation results of FIM as shown in Table 6 of your paper? Xnip2024-02-29_21-30-00
I tried to use bigcode-evaluation on Santacoder_FIM with the Insertion prompt structure from this repository, but it still generates very long content and I can't reproduce the results in the paper.

Originally created by @Hambaobao on GitHub (Feb 29, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/130 Hello, thanks a lot for your great work. Could you kindly advise on how to reproduce the evaluation results of FIM as shown in Table 6 of your [paper](https://arxiv.org/abs/2401.14196)?<img width="1022" alt="Xnip2024-02-29_21-30-00" src="https://github.com/deepseek-ai/DeepSeek-Coder/assets/48345096/68fd3fbc-90bc-478d-b74b-660b72f20d37"> I tried to use [bigcode-evaluation](https://github.com/bigcode-project/bigcode-evaluation-harness) on `Santacoder_FIM` with the [Insertion prompt](https://github.com/deepseek-ai/DeepSeek-Coder#2-code-insertion) structure from this repository, but it still generates very long content and I can't reproduce the results in the paper.

bsakaguchi closed this issue

2026-10-26 21:03:41 +00:00

bsakaguchi commented

2026-10-26 21:03:42 +00:00

Author

Owner

@pkuzqh commented on GitHub (Mar 6, 2024):

<｜fim▁begin｜>def quick_sort(arr):
if len(arr) <= 1:
return arr
pivot = arr[0]
left = []
right = []
<｜fim▁hole｜>
if arr[i] < pivot:
left.append(arr[i])
else:
right.append(arr[i])
return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜>
you need to use this prompt

@pkuzqh commented on GitHub (Mar 6, 2024): <｜fim▁begin｜>def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[0] left = [] right = [] <｜fim▁hole｜> if arr[i] < pivot: left.append(arr[i]) else: right.append(arr[i]) return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜> you need to use this prompt