[PR #803] [MERGED] [Release] Release 0.2.3 #1485

New issue

Closed

opened 2026-10-29 14:52:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:52:50 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/803
Author: @Azure-Tang
Created: 3/5/2025
Status: ✅ Merged
Merged: 3/5/2025
Merged by: @undefined

Base: main ← Head: develop-0.2.3

📝 Commits (10+)

e6349eb iq1s
93c5b75 rem
9781d1e iq1s core
c513ae5 iq1s files
d24d369 iq1s files
216a63b Merge pull request #754 from moonshadow-25/dev023
5aee6c0 Merge branch 'main' into develop-0.2.3
dc10480 ⚡ add humaneval support
52fa300 Merge pull request #791 from KMSorSMS/develop-0.2.3
662c1e4 small fix about max new token

📊 Changes

21 files changed (+2240 additions, -105 deletions)

View changed files

📝 Makefile (+7 -0)
📝 README.md (+1 -0)
📝 doc/README.md (+1 -0)
📝 doc/en/DeepseekR1_V3_tutorial.md (+2 -2)
📝 doc/en/FAQ.md (+69 -1)
📝 doc/en/benchmark.md (+7 -3)
📝 doc/en/install.md (+9 -1)
📝 ktransformers/__init__.py (+1 -1)
📝 ktransformers/server/backend/interfaces/transformers.py (+5 -4)
➕ ktransformers/tests/.gitignore (+1 -0)
➕ ktransformers/tests/AIME_2024/eval_api.py (+133 -0)
➕ ktransformers/tests/AIME_2024/evaluation.py (+10 -0)
➕ ktransformers/tests/AIME_2024/prompts.py (+2 -0)
➕ ktransformers/tests/humaneval/eval_api.py (+89 -0)
➕ ktransformers/tests/humaneval/evaluation.py (+15 -0)
➕ ktransformers/tests/humaneval/prompts.py (+14 -0)
📝 ktransformers/tests/mmlu_pro_test.py (+2 -2)
📝 ktransformers/tests/mmlu_test.py (+2 -2)
📝 third_party/llamafile/iqk_mul_mat.inc (+1854 -75)
📝 third_party/llamafile/sgemm.h (+8 -3)

...and 1 more files

📄 Description

Key Updates:

Low-Precision Inference Optimization #754

Added IQ1_S/IQ2_XXS quantized matmul support, now compatible with Unsloth's DeepSeek-R1 1.58bit/2.51bit dynamic quantized weights
Released DeepSeek-R1 mixed-precision model (IQ1+FP8) achieving enhanced performance:
- 19GB VRAM usage & 140GB system memory consumption
- MMLU score of 83.6, slightly outperforming full-precision DeepSeek-V3
- Ongoing benchmarks: View Details
  (Special thanks to @moonshadow-25 and @godrosev for their contributions to v0.2.3)

Long Context Handling Enhancement #750

Implemented chunked prefill mechanism. Supports processing 139K-token contexts with DeepSeek-R1 on 24GB VRAM
Note: As DeepSeek's native context window only supports 128K tokens, we will pause further optimizations for extended context handling.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/803 **Author:** [@Azure-Tang](https://github.com/Azure-Tang) **Created:** 3/5/2025 **Status:** ✅ Merged **Merged:** 3/5/2025 **Merged by:** [@undefined](undefined) **Base:** `main` ← **Head:** `develop-0.2.3` --- ### 📝 Commits (10+) - [`e6349eb`](https://github.com/kvcache-ai/ktransformers/commit/e6349eb2404e4bb57a5f459ed9ecf151e1fb5ea6) iq1s - [`93c5b75`](https://github.com/kvcache-ai/ktransformers/commit/93c5b757160a8f613c17596f90e0ad94718ed952) rem - [`9781d1e`](https://github.com/kvcache-ai/ktransformers/commit/9781d1e6f4c83341810ba9165d59a7b1317becf4) iq1s core - [`c513ae5`](https://github.com/kvcache-ai/ktransformers/commit/c513ae59c32a2021aff404c6febcae1245df8069) iq1s files - [`d24d369`](https://github.com/kvcache-ai/ktransformers/commit/d24d3693329b274e99d541b86dc07662391ed38e) iq1s files - [`216a63b`](https://github.com/kvcache-ai/ktransformers/commit/216a63b858d04d8734352c5aba27f16e34489165) Merge pull request #754 from moonshadow-25/dev023 - [`5aee6c0`](https://github.com/kvcache-ai/ktransformers/commit/5aee6c0446b786e2ef3a7e6077a8456417e168ec) Merge branch 'main' into develop-0.2.3 - [`dc10480`](https://github.com/kvcache-ai/ktransformers/commit/dc10480ef606f8af752b110a8a5c3d487cf1738c) :zap: add humaneval support - [`52fa300`](https://github.com/kvcache-ai/ktransformers/commit/52fa3003d1448ff9cecdda61c11d1d95a38fcb8b) Merge pull request #791 from KMSorSMS/develop-0.2.3 - [`662c1e4`](https://github.com/kvcache-ai/ktransformers/commit/662c1e4c145f89dfb9de664e4d6525ca5081a8de) small fix about max new token ### 📊 Changes **21 files changed** (+2240 additions, -105 deletions) <details> <summary>View changed files</summary> 📝 `Makefile` (+7 -0) 📝 `README.md` (+1 -0) 📝 `doc/README.md` (+1 -0) 📝 `doc/en/DeepseekR1_V3_tutorial.md` (+2 -2) 📝 `doc/en/FAQ.md` (+69 -1) 📝 `doc/en/benchmark.md` (+7 -3) 📝 `doc/en/install.md` (+9 -1) 📝 `ktransformers/__init__.py` (+1 -1) 📝 `ktransformers/server/backend/interfaces/transformers.py` (+5 -4) ➕ `ktransformers/tests/.gitignore` (+1 -0) ➕ `ktransformers/tests/AIME_2024/eval_api.py` (+133 -0) ➕ `ktransformers/tests/AIME_2024/evaluation.py` (+10 -0) ➕ `ktransformers/tests/AIME_2024/prompts.py` (+2 -0) ➕ `ktransformers/tests/humaneval/eval_api.py` (+89 -0) ➕ `ktransformers/tests/humaneval/evaluation.py` (+15 -0) ➕ `ktransformers/tests/humaneval/prompts.py` (+14 -0) 📝 `ktransformers/tests/mmlu_pro_test.py` (+2 -2) 📝 `ktransformers/tests/mmlu_test.py` (+2 -2) 📝 `third_party/llamafile/iqk_mul_mat.inc` (+1854 -75) 📝 `third_party/llamafile/sgemm.h` (+8 -3) _...and 1 more files_ </details> ### 📄 Description Key Updates: 1. Low-Precision Inference Optimization [#754](https://github.com/kvcache-ai/ktransformers/pull/754) - Added IQ1_S/IQ2_XXS quantized matmul support, now compatible with Unsloth's DeepSeek-R1 1.58bit/2.51bit dynamic quantized weights - Released DeepSeek-R1 mixed-precision model (IQ1+FP8) achieving enhanced performance: - 19GB VRAM usage & 140GB system memory consumption - MMLU score of 83.6, slightly outperforming full-precision DeepSeek-V3 - Ongoing benchmarks: [View Details](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/benchmark.md) (Special thanks to @moonshadow-25 and @godrosev for their contributions to v0.2.3) 2. Long Context Handling Enhancement [#750](https://github.com/kvcache-ai/ktransformers/pull/750) - Implemented chunked prefill mechanism. Supports processing 139K-token contexts with DeepSeek-R1 on 24GB VRAM - Note: As DeepSeek's native context window only supports 128K tokens, we will pause further optimizations for extended context handling. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:52:50 +00:00