deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #367] 关于技术报告中livecodebench的pass@1-cot指标的疑问 #286

New issue
Closed
opened 2026-10-26 21:18:49 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:18:49 +00:00
Owner
Copy link

Originally created by @stackoverflowwwww on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/367

技术报告中提到livecodebench指标为pass@1-cot，请问和livecodebench官方默认用的pass@1有什么区别吗。如果是prompt的区别的话，方便透露一下你们使用的prompt吗

Image
Originally created by @stackoverflowwwww on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/367 技术报告中提到livecodebench指标为pass@1-cot，请问和livecodebench官方默认用的pass@1有什么区别吗。如果是prompt的区别的话，方便透露一下你们使用的prompt吗 <img width="887" alt="Image" src="https://github.com/user-attachments/assets/0d859fb6-3d0f-4ec0-bb9d-bb4a3e3316d9" />
bsakaguchi 2026-10-26 21:18:49 +00:00
bsakaguchi commented 2026-10-26 21:18:50 +00:00
Author
Owner
Copy link

@davidxwwang commented on GitHub (Feb 12, 2025):

pass@1-cot 也是基于 pass@1 的评价方式，但加上了 Chain-of-Thought (CoT)，即链式推理的步骤。
CoT 是指模型在解答过程中，展示其推理过程的中间步骤，而不仅仅是给出最终答案。这意味着，模型不仅仅给出正确答案，还需要能展示出一个合理的推理过程，才能被认为通过测试。
pass@1-cot 的评价要求模型给出正确的解答，并且在此过程中能够展示其推理过程。如果没有推理过程或者推理过程错误，即便最终答案正确，模型也不会被认为是成功的。

<!-- gh-comment-id:2652511465 --> @davidxwwang commented on GitHub (Feb 12, 2025): pass@1-cot 也是基于 pass@1 的评价方式，但加上了 Chain-of-Thought (CoT)，即链式推理的步骤。 CoT 是指模型在解答过程中，展示其推理过程的中间步骤，而不仅仅是给出最终答案。这意味着，模型不仅仅给出正确答案，还需要能展示出一个合理的推理过程，才能被认为通过测试。 pass@1-cot 的评价要求模型给出正确的解答，并且在此过程中能够展示其推理过程。如果没有推理过程或者推理过程错误，即便最终答案正确，模型也不会被认为是成功的。
bsakaguchi commented 2026-10-26 21:18:51 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Mar 24, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:2746569837 --> @github-actions[bot] commented on GitHub (Mar 24, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:18:51 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Apr 28, 2025):

false

<!-- gh-comment-id:2833712904 --> @github-actions[bot] commented on GitHub (Apr 28, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#286
No description provided.