[GH-ISSUE #26] Cannot reproduce the results of HumanEval-Python #19

New issue

Closed

opened 2026-10-26 21:02:18 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:02:18 +00:00

Owner

Originally created by @Cheungki on GitHub (Nov 10, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/26

Thanks for your great work! I'm using the scripts in your repo to evaluate the released DeepSeek-Coder 1.3B (both base version and instruct version) with 8 V100s (therefore I use fp16 as the mixed precision).

But there is a performance gap between my results and those reported in this repo (3% for the base version and around 20% for the instruct version).

I'm wondering how to reproduce the reported results and might this is caused by fp16?

Originally created by @Cheungki on GitHub (Nov 10, 2023). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/26 Thanks for your great work! I'm using the scripts in your repo to evaluate the released DeepSeek-Coder 1.3B (both base version and instruct version) with 8 V100s (therefore I use fp16 as the mixed precision). But there is a performance gap between my results and those reported in this repo (3% for the base version and around 20% for the instruct version). I'm wondering how to reproduce the reported results and might this is caused by fp16?

bsakaguchi closed this issue

2026-10-26 21:02:18 +00:00

bsakaguchi commented

2026-10-26 21:02:19 +00:00

Author

Owner

@pkuzqh commented on GitHub (Nov 10, 2023):

The difference of the base model may be caused by the precision and the devices (we evaluate the models on A100s). The difference between the instruct version is caused by the post-processing. You can use this file to re-evaluate the model https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/Evaluation/HumanEval/eval_instruct.py .

@pkuzqh commented on GitHub (Nov 10, 2023): The difference of the base model may be caused by the precision and the devices (we evaluate the models on A100s). The difference between the instruct version is caused by the post-processing. You can use this file to re-evaluate the model https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/Evaluation/HumanEval/eval_instruct.py .