kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-04-20 14:29:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,134 Commits 46 Branches 28 Tags
ac69ea891eb4ddb0a6f43c7760b686d47d126bec
Commit Graph

1134 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Oql
ac69ea891e Fix K2 MoE decode bug in buffer management (#1686) 2025-12-08 21:08:28 +08:00
Oql
8139c092bf Reduce CPU memory usage during large chunk prefill (Fixes #1676) (#1683) 
* fix(amx): add BufferASmallKGroupImpl to fix buffer overflow in from_mat

The original BufferAKGroupImpl::from_mat writes 64 bytes per K_STEP iteration
but when K_STEP=32 (for GemmKernel224Int4SmallKGroup), this causes buffer overflow.

BufferASmallKGroupImpl overrides from_mat to write only 32 bytes per iteration.

* perf(k2-moe): optimize memory allocation with pooled buffers

- Replace per-expert buffer allocation with shared memory pools
- Dynamically assign buffer slices based on activated experts
- Add group_size inference from scale tensor shape in amx.py

* delete kimi k2 forward test

* add TODO comment for pool_count_ calculation
 2025-12-08 20:19:07 +08:00
ErvinXie
eefc8cf98d 更新 Kimi-K2-Thinking-Native.md (#1684) 2025-12-08 19:58:20 +08:00
Jiaqi Liao
f20e5d1da5 Revise prefill strategy and performance metrics (#1675) 
Updated the prefill strategy descriptions and performance benchmarks in the documentation.
 2025-12-06 15:36:04 +08:00
Jiaqi Liao
1d62ac21f7 Update Kimi-K2-Thinking-Native.md (#1673) 2025-12-05 23:08:02 +08:00
Jiaqi Liao
69fa7b1a57 Revise installation steps in Kimi-K2 documentation (#1672) 
Updated installation instructions and added steps for cloning the repository.
 2025-12-05 23:05:24 +08:00
Jiaqi Liao
721b6c4c94 [docs] Update Native Kimi-K2-Thinking documentation and kt-kernel parameters (#1671) v0.4.3 2025-12-05 22:46:16 +08:00
Jiaqi Liao
47da806cde [doc](kt-kernel): add kimi-k2-thinking (#1670) 2025-12-05 21:53:59 +08:00
ErvinXie
71f683acec Support Native Kimi K2 Thinking (#1663) 
* [feat]: fix k2 prefill

* Update Kimi-K2-Thinking.md

* Create Kimi-K2-Thinking-Native.md

* Update Kimi-K2-Thinking.md

* Update Kimi-K2-Thinking.md

* Update Kimi-K2-Thinking-Native.md

* [perf] optimize K2 MoE weight loading with per-expert pointers

- Avoid expensive torch.stack().contiguous() in Python (was ~6.6s)
- Use per-expert pointer arrays (gate_projs) instead of contiguous memory
- C++ worker pool performs parallel memcpy for TP slicing
- Add LOAD_TIME_PROFILE for load_weights timing analysis

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: ouqingliang <1692110604@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-05 21:53:05 +08:00
ZiWei Yuan
4850424345 [docs]: add amd blis backend usage guide (#1669) 2025-12-05 16:52:26 +08:00
Jiaqi Liao
1ca3a2662e Add 9#AISoft to the list of contributors (#1668) 2025-12-05 15:44:04 +08:00
Jiaqi Liao
0698252484 [fix](kt-kernel): gate RAWINT4 behind AVX512 and avoid AVX2 build break (#1660) 2025-12-03 00:43:23 +08:00
Jianwei Dong
670c488155 [docs]: Add deepseek-v3.2 run tutorial (#1659) 2025-12-02 20:04:10 +08:00
Jiaqi Liao
fcf8882075 [Feature] Add avx-based kimi-k2 support (#1656) 
* support Kimi-K2-Thinking original weight
fix amx kernel bug

* update k2 avx kernel.

* feat: add CPUInfer write buffer task

* [feat]: add kimi k2 cpu write buffer support

- Implement write_weights_to_buffer function in k2-moe.hpp for extracting GPU expert weights
- Fix down (w2) weight column-wise slicing for different TP configurations
- Support three TP scenarios: cpu_tp == gpu_tp, cpu_tp > gpu_tp, cpu_tp < gpu_tp
- Add comprehensive test cases for weight extraction validation
- Ensure compatibility with Kimi model's MoE architecture

* [fix]: correct write_weight_scale_to_buffer expert offset calculation

Fixed the bug in write_weight_scale_to_buffer_task where expert offsets in GPU buffers were incorrectly calculated. Changed from using per_expert_gpu sizes to using full gpu_tp sizes, ensuring correct memory layout for multi-expert scenarios.

Also added benchmark scripts for k2 moe and write buffer operations, and cleaned up debug output in test files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [feat]: add write buffer wrapper

* [fix] fix comment

---------

Co-authored-by: ouqingliang <1692110604@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-02 16:01:07 +08:00
ZiWei Yuan
c2b8c60c4e [ci]: add int4_1 & int4_1k (#1653) 
* [feat]: init amd adaption

* [feat]: add blis support

* [fix]: fix setup and moe kernel warpper

* [fix](setup.py): support rebuild with cache and import kt_kernel works
fine

* [feat]: add moe_kernel converter for amd and implement the load
method(haven't tested yet)

* [feat](moe_kernel/moe.hpp): delete unused memory when using save

* [fix](moe_kernel): update PLAIN for pack

* [fix](moe_kernel): rm printf debug

* [fix](moe_kernel): skip gpu experts

* [fix](moe_kernel/moe.hpp): update include memory path

* [feat](moe_kernel/moe.hpp): support expert deferral

* [feat]: finish amd

* [ci]: add int4_1 & int4_1k

---------

Co-authored-by: mrhaoxx <mr.haoxx@gmail.com>
 2025-12-02 15:58:14 +08:00
Jianwei Dong
fd78fe520a fix(scripts): resolve OOM when converting gpu weights and update README (#1640) 2025-12-01 14:15:14 +08:00
Peilin Li
e637fedc65 [docs]: Add Full introduction of KT (#1636) 2025-11-29 15:46:55 +08:00
Peilin Li
7ee80bbc3d [docs]: Update README with Python 3.12 and dependency changes (#1634) 
Updated Python version in installation instructions and adjusted KTransformers and flash-attention wheel filenames accordingly.
 2025-11-29 15:46:05 +08:00
mrhaoxx
637c49c83f [feat](kt-kernel): support qwen3-vl weights convert (#1648) 2025-11-27 22:29:09 +08:00
Jianwei Dong
c256150e08 update ci test (#1647) 2025-11-27 16:39:48 +08:00
ZiWei Yuan
1374b98ee5 [feat](moe_kernel): add amd blis support (int8) (#1600) 
* [feat]: init amd adaption

* [feat]: add blis support

* [fix]: fix setup and moe kernel warpper

* [fix](setup.py): support rebuild with cache and import kt_kernel works
fine

* [feat]: add moe_kernel converter for amd and implement the load
method(haven't tested yet)

* [feat](moe_kernel/moe.hpp): delete unused memory when using save

* [fix](moe_kernel): update PLAIN for pack

* [fix](moe_kernel): rm printf debug

* [fix](moe_kernel): skip gpu experts

* [fix](moe_kernel/moe.hpp): update include memory path

* [feat](moe_kernel/moe.hpp): support expert deferral

* [feat]: finish amd

---------

Co-authored-by: mrhaoxx <mr.haoxx@gmail.com>
 2025-11-27 12:08:53 +08:00
Jianwei Dong
fef6dd98a8 add accuracy and performance test (#1643) 2025-11-27 10:56:39 +08:00
Jiaqi Liao
e7d1c1de09 fix(llamafile): resolve deferred experts data race and update README (#1646) 2025-11-26 23:19:37 +08:00
Jianwei Dong
51745a9ea1 add ci (#1642) 2025-11-25 20:52:08 +08:00
RICHARDNAN
2cffdf7033 [docs]: Update DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md (#1638) 2025-11-24 11:51:07 +08:00
DocShotgun
e72a4fb880 [feat](kt-kernel): Add resume arg to CPU weight conversion (#1630) 
* [feat]: kt-kernel: Add resume arg to CPU weight conversion

* [docs]: kt-kernel: Document resume arg for CPU weight conversion

* [fix]: kt-kernel: Only print resume layer if in use

* [fix]: kt-kernel: Don't log skipped layers when using resume_layer
 2025-11-22 12:00:15 +08:00
Jiaqi Liao
e69c67713f [refactor] fix third_party issue (#1632) 
* [refactor]: relocate third_party directory

* [fix]: fix custom_flashinfer for kt-sft
v0.4.2 		2025-11-20 13:55:55 +08:00
Jiaqi Liao
46af8fcab5 [doc] fix kt parameters (#1629) 2025-11-19 16:41:57 +08:00
Peilin Li
171578a7ec [refactor]: Change named 'KT-SFT' to 'kt-sft' (#1626) 
* Change named 'KT-SFT' to 'kt-sft'

* [docs]: update kt-sft name

---------

Co-authored-by: ZiWei Yuan <yzwliam@126.com>
 2025-11-17 11:48:42 +08:00
Pory
2887050ca1 [Feature] add Qwen3MoE models for KTransformers-FT (#1602) 
* add qwen3 attn

* fix KQwen3MoeSparseMoeBlock

* fix bug adapter for llamafactory

---------

Co-authored-by: unknown <xiongchenhui@hisense.ad>
 2025-11-16 16:39:19 +08:00
ZiWei Yuan
ab8ad0a110 [docs]: update web doc (#1625) 2025-11-16 14:40:22 +08:00
ZiWei Yuan
be6db6f46b [docs]: improve structure for kt-kernel (#1624) 
* [docs]: improve structure for kt-kernel

* Update doc/en/kt-kernel/README.md
 2025-11-16 13:21:41 +08:00
ZiWei Yuan
133eea037c [docs]: improve docs structure (#1623) 2025-11-16 12:40:59 +08:00
ZiWei Yuan
c2d2edbeef [docs]: update the web docs structure (#1622) 2025-11-16 12:09:44 +08:00
ZiWei Yuan
f914bc4581 [docs]: add issue config (#1621) 
* [docs]: add issue config

* [docs]: consider to force the issue template
 2025-11-16 11:50:43 +08:00
ZiWei Yuan
ac0e449980 [docs]: add code of conduct & pr template (#1619) 2025-11-15 22:45:53 +08:00
ZiWei Yuan
e0e2429748 [ci]: update issue template & security & license (#1617) 2025-11-15 22:29:01 +08:00
ZiWei Yuan
550e4986f5 [docs]: update README.md (#1616) 
* [docs]: update README.md
 2025-11-15 20:56:26 +08:00
ZiWei Yuan
7c2ad6dbca [docs]: update README.md (#1614) 
* [docs]: update README.md
 2025-11-15 18:34:27 +08:00
ZiWei Yuan
aef6672dd8 [docs]: add contribuing guide and add hooks install (#1613) 
* [feat]: update kt-kernel hooks and add contribution guide

* [docs]: add contributing guide
* [style]: format the python file and cpp file in kt-kernel
 2025-11-15 18:26:49 +08:00
ZiWei Yuan
c32fefb1cd [doc]: update web doc and kt-kernel doc (#1609) 
* [doc]: update web doc and kt-kernel doc

* [doc](book.toml): add book.toml for rust book compile
 2025-11-13 20:44:13 +08:00
Jiaqi Liao
4bd0fe812b docs(kt-kernel): improve SGLang integration documentation and fix syntax errors (#1607) 
- Clarified instructions for SGLang integration with kt-kernel
 2025-11-13 19:23:00 +08:00
Jiaqi Liao
13b8ddecd9 AMXMoEWrapper -> KTMoEWrapper (#1604) 
fix import KTMoEWrapper
 2025-11-12 16:34:54 +08:00
Jiaqi Liao
02801c3c4e fix kt-kernel installation issue (#1603) 
* update README for kt-kernel for installation issues

* update install.sh

* update install.sh

* update install.sh

* update install.sh

* update install.sh

* update README

* fix sudo issue

* fix install issue

* fix import issue

* fix import issue

* update install.sh

* fix import issue

* fix import issue

* fix import issue

* update README

* update install

* update install
 2025-11-12 15:56:02 +08:00
Peilin Li
26b86aa96b Add warning for CUDA installation step in README (#1599) 
Added a warning about skipping the CUDA installation step.
 2025-11-12 11:14:50 +08:00
Peilin Li
148a030026 upload hands-on tutorial with KTransformers-FT, especially in customize your KT-FT+LLaMA-Factory (#1597) 
* Add files via upload

* upload hands-on tutorial for KTransformers-FT
 2025-11-11 20:54:41 +08:00
Jiaqi Liao
d483147307 Fix kt-kernel compile issue (#1595) 
* update install.sh

* fix import issue

* update README
 2025-11-11 19:30:27 +08:00
ZiWei Yuan
a6bb7651f8 [build]: amd for llamafile right now (#1594) 2025-11-11 18:08:03 +08:00
Jiaqi Liao
bb1a667169 fix hwloc README (#1592) 2025-11-11 15:28:57 +08:00
Jiaqi Liao
743f6f719b update README for kt-kernel for installation issues (#1590) 2025-11-11 14:53:45 +08:00