[PR #724] fix(eval): robust code block extraction across evaluation scripts #295

New issue

Open

opened 2026-10-26 21:05:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:05:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepSeek-Coder/pull/724
Author: @reginaldalfret
Created: 10/25/2026
Status: 🔄 Open

Base: main ← Head: fix/evaluation-code-extraction

📝 Commits (1)

f5c8e58 fix(eval): robust code block extraction across evaluation scripts

📊 Changes

4 files changed (+149 additions, -17 deletions)

View changed files

📝 Evaluation/HumanEval/utils/utils.py (+27 -11)
📝 Evaluation/LeetCode/evaluate_leetcode.py (+8 -4)
📝 Evaluation/MBPP/eval_instruct.py (+5 -2)
➕ tests/test_code_extraction.py (+109 -0)

📄 Description

Problem

CRLF & Whitespace Mismatch in Code Blocks: In Evaluation/HumanEval/utils/utils.py, Evaluation/MBPP/eval_instruct.py, and Evaluation/LeetCode/evaluate_leetcode.py, regex patterns for markdown code blocks used strict \n without supporting \r?\n or trailing whitespace after the language identifier (e.g., ```python). On Windows environments or when responses contain CRLF, all extracted code blocks failed to match and fell back to raw/unparsed output.
Unclosed / Truncated Code Blocks: When model generation reaches max_new_tokens, the terminating ``` fence is often not generated. The previous regex patterns strictly required closing fences, discarding otherwise valid completions.
Index Underflow in Indentation Parsing: In Evaluation/HumanEval/utils/utils.py, the loop while start - indent >= 0 and code_block[start - indent - 1] == ' ': evaluated code_block[-1] when start == indent. If the generated code block ended with trailing whitespace, code_block[-1] == ' ' was True, causing indent to erroneously increment to 1 and corrupting the closing brace search.
Indented Function Detection: get_function_name required func_lines[i].startswith("def "), failing when function definitions had leading whitespace (e.g. methods in class templates).

Solution

Update code block regexes across HumanEval, MBPP, and LeetCode to handle CRLF line endings (\r?\n), optional horizontal whitespace after language tags ([^\S\r\n]*), and unclosed code blocks ((?:```|$)).
Add fallback to generic code blocks if language identifier is omitted.
Fix the indentation detection condition to start - indent > 0 to prevent Python negative index wrap-around.
Support indented def declarations in get_function_name().
Add unit tests in tests/test_code_extraction.py covering all extraction edge cases (CRLF, trailing space, unclosed fence, negative index safety, and LeetCode fallbacks).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepSeek-Coder/pull/724 **Author:** [@reginaldalfret](https://github.com/reginaldalfret) **Created:** 10/25/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/evaluation-code-extraction` --- ### 📝 Commits (1) - [`f5c8e58`](https://github.com/deepseek-ai/DeepSeek-Coder/commit/f5c8e58d7b7de97ebf75f279e3d664fe49a138a3) fix(eval): robust code block extraction across evaluation scripts ### 📊 Changes **4 files changed** (+149 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `Evaluation/HumanEval/utils/utils.py` (+27 -11) 📝 `Evaluation/LeetCode/evaluate_leetcode.py` (+8 -4) 📝 `Evaluation/MBPP/eval_instruct.py` (+5 -2) ➕ `tests/test_code_extraction.py` (+109 -0) </details> ### 📄 Description ### Problem 1. **CRLF & Whitespace Mismatch in Code Blocks**: In `Evaluation/HumanEval/utils/utils.py`, `Evaluation/MBPP/eval_instruct.py`, and `Evaluation/LeetCode/evaluate_leetcode.py`, regex patterns for markdown code blocks used strict `\n` without supporting `\r?\n` or trailing whitespace after the language identifier (e.g., ` ```python `). On Windows environments or when responses contain CRLF, all extracted code blocks failed to match and fell back to raw/unparsed output. 2. **Unclosed / Truncated Code Blocks**: When model generation reaches `max_new_tokens`, the terminating ` ``` ` fence is often not generated. The previous regex patterns strictly required closing fences, discarding otherwise valid completions. 3. **Index Underflow in Indentation Parsing**: In `Evaluation/HumanEval/utils/utils.py`, the loop `while start - indent >= 0 and code_block[start - indent - 1] == ' ':` evaluated `code_block[-1]` when `start == indent`. If the generated code block ended with trailing whitespace, `code_block[-1] == ' '` was `True`, causing `indent` to erroneously increment to `1` and corrupting the closing brace search. 4. **Indented Function Detection**: `get_function_name` required `func_lines[i].startswith("def ")`, failing when function definitions had leading whitespace (e.g. methods in class templates). ### Solution 1. Update code block regexes across HumanEval, MBPP, and LeetCode to handle CRLF line endings (`\r?\n`), optional horizontal whitespace after language tags (`[^\S\r\n]*`), and unclosed code blocks (`(?:```|$)`). 2. Add fallback to generic code blocks if language identifier is omitted. 3. Fix the indentation detection condition to `start - indent > 0` to prevent Python negative index wrap-around. 4. Support indented `def ` declarations in `get_function_name()`. 5. Add unit tests in `tests/test_code_extraction.py` covering all extraction edge cases (CRLF, trailing space, unclosed fence, negative index safety, and LeetCode fallbacks). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 21:05:30 +00:00