[PR #657] [MERGED] Feat absorb for long prefill #1447

New issue

Closed

opened 2026-10-29 14:52:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:52:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/657
Author: @Atream
Created: 2/25/2025
Status: ✅ Merged
Merged: 2/25/2025
Merged by: @Atream

Base: develop-0.2.2 ← Head: feat-absorb-for-long-prefill

📝 Commits (9)

006e8c6 remove causal mask
95d937c tmp
e8e02e5 support Moonlight
f5f6c6b update yaml
eb039b7 Merge pull request #621 from kvcache-ai/feat-moonlight
f327695 fix KExpertsMarlin on GPU with out CUDA Graph
4b5991e Merge pull request #638 from kvcache-ai/feat-moonlight
e9b1216 Merge branch 'main' into feat-absorb-for-long-prefill
f4c198b support absorb for prefill long context

📊 Changes

11 files changed (+193 additions, -43 deletions)

View changed files

📝 ktransformers/local_chat.py (+2 -3)
📝 ktransformers/operators/attention.py (+39 -19)
📝 ktransformers/operators/experts.py (+3 -3)
📝 ktransformers/operators/flashinfer_wrapper.py (+22 -8)
📝 ktransformers/operators/models.py (+9 -4)
📝 ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml (+1 -0)
➕ ktransformers/optimize/optimize_rules/Moonlight-16B-A3B.yaml (+86 -0)
📝 ktransformers/server/backend/interfaces/ktransformers.py (+5 -0)
📝 ktransformers/server/backend/interfaces/transformers.py (+1 -1)
📝 ktransformers/util/custom_gguf.py (+2 -0)
📝 ktransformers/util/utils.py (+23 -5)

📄 Description

No description provided

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/657 **Author:** [@Atream](https://github.com/Atream) **Created:** 2/25/2025 **Status:** ✅ Merged **Merged:** 2/25/2025 **Merged by:** [@Atream](https://github.com/Atream) **Base:** `develop-0.2.2` ← **Head:** `feat-absorb-for-long-prefill` --- ### 📝 Commits (9) - [`006e8c6`](https://github.com/kvcache-ai/ktransformers/commit/006e8c6abc6503921411db935efd80ad7f16032d) remove causal mask - [`95d937c`](https://github.com/kvcache-ai/ktransformers/commit/95d937c51d91630ec6d764500c4e668038ef8f22) tmp - [`e8e02e5`](https://github.com/kvcache-ai/ktransformers/commit/e8e02e5ccc9227055617247fad60e1a973885109) support Moonlight - [`f5f6c6b`](https://github.com/kvcache-ai/ktransformers/commit/f5f6c6b95d935e65fbc37d3245c2be064389cfa5) update yaml - [`eb039b7`](https://github.com/kvcache-ai/ktransformers/commit/eb039b723dafd4bb518fd5bf6ba0d3289cfa9e1a) Merge pull request #621 from kvcache-ai/feat-moonlight - [`f327695`](https://github.com/kvcache-ai/ktransformers/commit/f327695079298e703e24eb8a9f2493fe4e2bde80) fix KExpertsMarlin on GPU with out CUDA Graph - [`4b5991e`](https://github.com/kvcache-ai/ktransformers/commit/4b5991e77e5e005aa54e6ef1b9576dc4e96d7a60) Merge pull request #638 from kvcache-ai/feat-moonlight - [`e9b1216`](https://github.com/kvcache-ai/ktransformers/commit/e9b1216a9ad5797ab5c3c34d4e087134514692fe) Merge branch 'main' into feat-absorb-for-long-prefill - [`f4c198b`](https://github.com/kvcache-ai/ktransformers/commit/f4c198bd42f037ccc570eb8f1f1b4ab1ea9c7fa2) support absorb for prefill long context ### 📊 Changes **11 files changed** (+193 additions, -43 deletions) <details> <summary>View changed files</summary> 📝 `ktransformers/local_chat.py` (+2 -3) 📝 `ktransformers/operators/attention.py` (+39 -19) 📝 `ktransformers/operators/experts.py` (+3 -3) 📝 `ktransformers/operators/flashinfer_wrapper.py` (+22 -8) 📝 `ktransformers/operators/models.py` (+9 -4) 📝 `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` (+1 -0) ➕ `ktransformers/optimize/optimize_rules/Moonlight-16B-A3B.yaml` (+86 -0) 📝 `ktransformers/server/backend/interfaces/ktransformers.py` (+5 -0) 📝 `ktransformers/server/backend/interfaces/transformers.py` (+1 -1) 📝 `ktransformers/util/custom_gguf.py` (+2 -0) 📝 `ktransformers/util/utils.py` (+23 -5) </details> ### 📄 Description _No description provided_ --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:52:30 +00:00