deepseek-ai/DeepSeek-Coder
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-Coder.git synced 2026-11-05 00:57:19 +00:00
Code Issues 177 Projects Packages Wiki Activity

[GH-ISSUE #5] Reproducing HumanEval scores #1

New issue
Closed
opened 2026-10-26 21:01:53 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 21:01:53 +00:00
Owner
Copy link

Originally created by @ganler on GitHub (Nov 2, 2023).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/5

Hi thanks for the great work.

I have just evaluated the 33b base model on both HumanEval and HumanEval+. However, there is a certain gap between the claimed 56.1% pass@1 from https://github.com/deepseek-ai/DeepSeek-Coder#2-evaluation-results and my regenerated 51.2%. I am curious if there is anything that I misimplemented for DeepSeek-Coder? We were using torch.bfloat16 and the default prompt from HumanEval. Meanwhile, I am using greedy decoding with top_p=0.95 and top_k=None.

image

Originally created by @ganler on GitHub (Nov 2, 2023). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-Coder/issues/5 Hi thanks for the great work. I have just evaluated the 33b base model on both HumanEval and HumanEval+. However, there is a certain gap between the claimed 56.1% pass@1 from https://github.com/deepseek-ai/DeepSeek-Coder#2-evaluation-results and my regenerated 51.2%. I am curious if there is anything that I misimplemented for `DeepSeek-Coder`? We were using `torch.bfloat16` and the default prompt from HumanEval. Meanwhile, I am using greedy decoding with top_p=0.95 and top_k=None. ![image](https://github.com/deepseek-ai/DeepSeek-Coder/assets/38074777/818c12a1-2f20-492c-97f9-3fcdfc721180)
bsakaguchi commented 2026-10-26 21:01:54 +00:00
Author
Owner
Copy link

@ganler commented on GitHub (Nov 2, 2023):

An interesting finding tho: https://evalplus.github.io/leaderboard.html

image

From our independent evaluation, codellama-34b is a bit stronger than deepseekcoder-33b on the original HumanEval but instead on HumanEval+ deepseekcoder-33b is found to be better, indicating deepseekcoder is producing more robust code (i.e., HumanEval+ adds more test-cases for HumanEval whose tests might be insufficient for showing the correctness).

<!-- gh-comment-id:1791452191 --> @ganler commented on GitHub (Nov 2, 2023): An interesting finding tho: https://evalplus.github.io/leaderboard.html ![image](https://github.com/deepseek-ai/DeepSeek-Coder/assets/38074777/90361c45-46d4-44a1-a616-307566c1afc8) From our independent evaluation, codellama-34b is a bit stronger than deepseekcoder-33b on the original HumanEval but instead on HumanEval+ deepseekcoder-33b is found to be better, indicating deepseekcoder is producing more robust code (i.e., HumanEval+ adds more test-cases for HumanEval whose tests might be insufficient for showing the correctness).
bsakaguchi commented 2026-10-26 21:01:55 +00:00
Author
Owner
Copy link

@ganler commented on GitHub (Nov 2, 2023):

BTW, could you indicate the instruction that is used for the deepseek-ai/deepseek-coder-33b-instruct on HumanEval? Thanks!

<!-- gh-comment-id:1791465445 --> @ganler commented on GitHub (Nov 2, 2023): BTW, could you indicate the instruction that is used for the `deepseek-ai/deepseek-coder-33b-instruct` on HumanEval? Thanks!
bsakaguchi commented 2026-10-26 21:01:55 +00:00
Author
Owner
Copy link

@ganler commented on GitHub (Nov 2, 2023):

Update: I am able to get the following result using the prompt here and some post cleaning to make all produced compilable:

image

<!-- gh-comment-id:1791566659 --> @ganler commented on GitHub (Nov 2, 2023): Update: I am able to get the following result using the prompt [here](https://github.com/evalplus/evalplus/blob/9b82c5230ba5acc18050501532e2e6a132c76c7b/codegen/model.py#L334) and some post cleaning to make all produced compilable: ![image](https://github.com/deepseek-ai/DeepSeek-Coder/assets/38074777/99fd6bd6-3626-405b-b5e1-42b0177b13e2)
bsakaguchi commented 2026-10-26 21:01:55 +00:00
Author
Owner
Copy link

@SivilTaram commented on GitHub (Nov 3, 2023):

@ganler

If following the exact same evaluation setting with bigcode-evaluation-harness, I can obtain the following pass@1 results for the 33B model as below:

Evaluating generations...                                    
{                                                           
  "humaneval": {                                             
    "pass@1": 0.5365853658536586                            
  },                                                         
  "config": {                                                 
    "prefix": "",                                                  
    "do_sample": false,                                         
    "temperature": 0.2,                                            
    "top_k": 0,                                                      
    "top_p": 0.95,                                                 
    "n_samples": 1,                                                  
    "eos": "<|endoftext|>",                                             
    "seed": 0,                                                       
    "model": "deepseek-ai/deepseek-coder-33b-base",                     
    "modeltype": "causal",                                                
    "peft_model": null,                                                 
    "revision": null,                                                     
    "use_auth_token": false,                                                 
    "trust_remote_code": true,                                            
    "tasks": "humaneval",                                                    
    "instruction_tokens": null,                                             
    "batch_size": 1,                                                         
    "max_length_generation": 650,                                            
    "precision": "fp32",              
    "load_in_8bit": false,            
    "load_in_4bit": false,            
    "limit": null,                    
    "limit_start": 0,                 
    "postprocess": true,              
    "allow_code_execution": true,                                            
    "generation_only": false,         
    "load_generations_path": null,                                           
    "load_data_path": null,           
    "metric_output_path": "evaluation_results.json",                         
    "save_generations": false,                                               
    "save_generations_path": "generations.json",                             
    "save_references": false,         
    "prompt": "prompt",               
    "max_memory_per_gpu": "auto",                                            
    "check_references": false         
  }                                   
}

Considering that the slightly different operations during post-processing, I think the claimed pass@1 should be reproducible.

<!-- gh-comment-id:1792079352 --> @SivilTaram commented on GitHub (Nov 3, 2023): @ganler If following the exact same evaluation setting with [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness), I can obtain the following pass@1 results for the [33B model](https://huggingface.co/deepseek-ai/deepseek-coder-33b-base) as below: ``` Evaluating generations... { "humaneval": { "pass@1": 0.5365853658536586 }, "config": { "prefix": "", "do_sample": false, "temperature": 0.2, "top_k": 0, "top_p": 0.95, "n_samples": 1, "eos": "<|endoftext|>", "seed": 0, "model": "deepseek-ai/deepseek-coder-33b-base", "modeltype": "causal", "peft_model": null, "revision": null, "use_auth_token": false, "trust_remote_code": true, "tasks": "humaneval", "instruction_tokens": null, "batch_size": 1, "max_length_generation": 650, "precision": "fp32", "load_in_8bit": false, "load_in_4bit": false, "limit": null, "limit_start": 0, "postprocess": true, "allow_code_execution": true, "generation_only": false, "load_generations_path": null, "load_data_path": null, "metric_output_path": "evaluation_results.json", "save_generations": false, "save_generations_path": "generations.json", "save_references": false, "prompt": "prompt", "max_memory_per_gpu": "auto", "check_references": false } } ``` Considering that the slightly different operations during post-processing, I think the claimed pass@1 should be reproducible.
bsakaguchi commented 2026-10-26 21:01:55 +00:00
Author
Owner
Copy link

@ganler commented on GitHub (Nov 3, 2023):

@SivilTaram Thanks for the reference! I think these results are all possible by a better/optimal generation/inference parameters (e.g., we use bf16 and your config is calling fp32; our max generation length is 512 while yours is larger; etc.) since I believe our post-processing is robust as only two output solutions from deepseek-coder are not compilable (and we manually checked it is due to infinite enumeration of outputs) while the rest all look good.

Anyways,I also get a copy of the author-provided solutions (thanks to the authors!) which can produce the claimed results:

image

I will close the issue as I don't have further questions. Thanks for the attention and the great work! :)

<!-- gh-comment-id:1793036761 --> @ganler commented on GitHub (Nov 3, 2023): @SivilTaram Thanks for the reference! I think these results are all possible by a better/optimal generation/inference parameters (e.g., we use `bf16` and your config is calling `fp32`; our max generation length is 512 while yours is larger; etc.) since I believe our [post-processing](https://github.com/evalplus/evalplus/blob/master/tools/sanitize.py) is robust as only two output solutions from deepseek-coder are not compilable (and we manually checked it is due to infinite enumeration of outputs) while the rest all look good. Anyways,I also get a copy of the author-provided solutions (thanks to the authors!) which can produce the claimed results: ![image](https://github.com/deepseek-ai/DeepSeek-Coder/assets/38074777/2619b7ee-ca23-4607-b15e-dbc701b062a8) I will close the issue as I don't have further questions. Thanks for the attention and the great work! :)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ydj/leetcode
bingxuan/dev
ydj/mbpp
No results found.
Labels
Clear labels   
documentation

   
ENV

   
pull-request
Mirrored from GitHub Pull Request

  
research question

   
WIP
No labels
documentation
ENV
pull-request
research question
WIP
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-Coder#1
No description provided.