[PR #722] Decode evaluation JSONL consistently as UTF-8 #296

New issue

Open

opened 2026-10-26 21:05:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:05:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepSeek-Coder/pull/722
Author: @determine123
Created: 10/20/2026
Status: 🔄 Open

Base: main ← Head: fix/eval-gzip-utf8

📝 Commits (1)

e39cf5d fix(eval): decode compressed JSONL inputs as UTF-8

📊 Changes

4 files changed (+69 additions, -4 deletions)

View changed files

📝 Evaluation/HumanEval/human_eval/data.py (+1 -1)
📝 Evaluation/LeetCode/human_eval/data.py (+2 -2)
📝 Evaluation/MBPP/human_eval/data.py (+1 -1)
➕ tests/test_eval_jsonl_encoding.py (+65 -0)

📄 Description

The HumanEval, MBPP and LeetCode copies of stream_jsonl decode gzip input using the locale's default encoding, whereas their writers emit UTF-8 bytes. This can silently corrupt non-ASCII prompts or raise UnicodeDecodeError when reading externally produced UTF-8 JSONL with unescaped Unicode. LeetCode's plain-text reader also uses the default encoding.

Specify UTF-8 for the three gzip readers and the LeetCode plain reader, matching the existing HumanEval/MBPP plain-reader behavior. No generation or scoring behavior is changed.

Validation (Python 3.12.10, Windows, CPU only):

An actual subprocess with -X utf8=0 and CP936 default encoding read a UTF-8 gzip prompt containing Chinese and emoji as different characters before this fix.
python -m unittest discover -s tests -p test_eval_jsonl_encoding.py -v: 3 test methods / 9 subtest scenarios pass. Tests cover all three evaluation directories, UTF-8 gzip under an emulated ASCII default at the standard-library I/O boundary, plain UTF-8, and gzip write/append/read round trips. On the original readers, the ASCII-default gzip cases error in all three suites; the LeetCode plain-reader case also fails on this CP936 host.
python -m ruff check --select E9,F63,F7,F82 tests/test_eval_jsonl_encoding.py Evaluation/HumanEval/human_eval/data.py Evaluation/MBPP/human_eval/data.py Evaluation/LeetCode/human_eval/data.py
git diff --check

The tests execute the actual data-reader modules without model dependencies. Full model generation and execution-based benchmark scoring were not run. AI assistance was used for analysis, implementation, testing and drafting.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepSeek-Coder/pull/722 **Author:** [@determine123](https://github.com/determine123) **Created:** 10/20/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/eval-gzip-utf8` --- ### 📝 Commits (1) - [`e39cf5d`](https://github.com/deepseek-ai/DeepSeek-Coder/commit/e39cf5d61b421c9519c32471996b8d6b1c418cfe) fix(eval): decode compressed JSONL inputs as UTF-8 ### 📊 Changes **4 files changed** (+69 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `Evaluation/HumanEval/human_eval/data.py` (+1 -1) 📝 `Evaluation/LeetCode/human_eval/data.py` (+2 -2) 📝 `Evaluation/MBPP/human_eval/data.py` (+1 -1) ➕ `tests/test_eval_jsonl_encoding.py` (+65 -0) </details> ### 📄 Description The HumanEval, MBPP and LeetCode copies of `stream_jsonl` decode gzip input using the locale's default encoding, whereas their writers emit UTF-8 bytes. This can silently corrupt non-ASCII prompts or raise `UnicodeDecodeError` when reading externally produced UTF-8 JSONL with unescaped Unicode. LeetCode's plain-text reader also uses the default encoding. Specify UTF-8 for the three gzip readers and the LeetCode plain reader, matching the existing HumanEval/MBPP plain-reader behavior. No generation or scoring behavior is changed. Validation (Python 3.12.10, Windows, CPU only): - An actual subprocess with `-X utf8=0` and CP936 default encoding read a UTF-8 gzip prompt containing Chinese and emoji as different characters before this fix. - `python -m unittest discover -s tests -p test_eval_jsonl_encoding.py -v`: 3 test methods / 9 subtest scenarios pass. Tests cover all three evaluation directories, UTF-8 gzip under an emulated ASCII default at the standard-library I/O boundary, plain UTF-8, and gzip write/append/read round trips. On the original readers, the ASCII-default gzip cases error in all three suites; the LeetCode plain-reader case also fails on this CP936 host. - `python -m ruff check --select E9,F63,F7,F82 tests/test_eval_jsonl_encoding.py Evaluation/HumanEval/human_eval/data.py Evaluation/MBPP/human_eval/data.py Evaluation/LeetCode/human_eval/data.py` - `git diff --check` The tests execute the actual data-reader modules without model dependencies. Full model generation and execution-based benchmark scoring were not run. AI assistance was used for analysis, implementation, testing and drafting. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 21:05:30 +00:00