kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-03-15 02:47:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,135 Commits 33 Branches 27 Tags
503295fc88dbb12537c8401d14f76b4c82b5506f
Commit Graph

50 Commits

Author SHA1 Message Date
mrhaoxx
503295fc88 [feat](kt-kernel): refactor convert_cpu_weights.py to support conversation for GLM-4.6V (#1687) 
Signed-off-by: mrhaoxx <mr.haoxx@gmail.com>
 2025-12-09 14:24:41 +08:00
Oql
ac69ea891e Fix K2 MoE decode bug in buffer management (#1686) 2025-12-08 21:08:28 +08:00
Oql
8139c092bf Reduce CPU memory usage during large chunk prefill (Fixes #1676) (#1683) 
* fix(amx): add BufferASmallKGroupImpl to fix buffer overflow in from_mat

The original BufferAKGroupImpl::from_mat writes 64 bytes per K_STEP iteration
but when K_STEP=32 (for GemmKernel224Int4SmallKGroup), this causes buffer overflow.

BufferASmallKGroupImpl overrides from_mat to write only 32 bytes per iteration.

* perf(k2-moe): optimize memory allocation with pooled buffers

- Replace per-expert buffer allocation with shared memory pools
- Dynamically assign buffer slices based on activated experts
- Add group_size inference from scale tensor shape in amx.py

* delete kimi k2 forward test

* add TODO comment for pool_count_ calculation
 2025-12-08 20:19:07 +08:00
Jiaqi Liao
721b6c4c94 [docs] Update Native Kimi-K2-Thinking documentation and kt-kernel parameters (#1671) 2025-12-05 22:46:16 +08:00
ErvinXie
71f683acec Support Native Kimi K2 Thinking (#1663) 
* [feat]: fix k2 prefill

* Update Kimi-K2-Thinking.md

* Create Kimi-K2-Thinking-Native.md

* Update Kimi-K2-Thinking.md

* Update Kimi-K2-Thinking.md

* Update Kimi-K2-Thinking-Native.md

* [perf] optimize K2 MoE weight loading with per-expert pointers

- Avoid expensive torch.stack().contiguous() in Python (was ~6.6s)
- Use per-expert pointer arrays (gate_projs) instead of contiguous memory
- C++ worker pool performs parallel memcpy for TP slicing
- Add LOAD_TIME_PROFILE for load_weights timing analysis

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: ouqingliang <1692110604@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-05 21:53:05 +08:00
ZiWei Yuan
4850424345 [docs]: add amd blis backend usage guide (#1669) 2025-12-05 16:52:26 +08:00
Jiaqi Liao
0698252484 [fix](kt-kernel): gate RAWINT4 behind AVX512 and avoid AVX2 build break (#1660) 2025-12-03 00:43:23 +08:00
Jiaqi Liao
fcf8882075 [Feature] Add avx-based kimi-k2 support (#1656) 
* support Kimi-K2-Thinking original weight
fix amx kernel bug

* update k2 avx kernel.

* feat: add CPUInfer write buffer task

* [feat]: add kimi k2 cpu write buffer support

- Implement write_weights_to_buffer function in k2-moe.hpp for extracting GPU expert weights
- Fix down (w2) weight column-wise slicing for different TP configurations
- Support three TP scenarios: cpu_tp == gpu_tp, cpu_tp > gpu_tp, cpu_tp < gpu_tp
- Add comprehensive test cases for weight extraction validation
- Ensure compatibility with Kimi model's MoE architecture

* [fix]: correct write_weight_scale_to_buffer expert offset calculation

Fixed the bug in write_weight_scale_to_buffer_task where expert offsets in GPU buffers were incorrectly calculated. Changed from using per_expert_gpu sizes to using full gpu_tp sizes, ensuring correct memory layout for multi-expert scenarios.

Also added benchmark scripts for k2 moe and write buffer operations, and cleaned up debug output in test files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [feat]: add write buffer wrapper

* [fix] fix comment

---------

Co-authored-by: ouqingliang <1692110604@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-02 16:01:07 +08:00
ZiWei Yuan
c2b8c60c4e [ci]: add int4_1 & int4_1k (#1653) 
* [feat]: init amd adaption

* [feat]: add blis support

* [fix]: fix setup and moe kernel warpper

* [fix](setup.py): support rebuild with cache and import kt_kernel works
fine

* [feat]: add moe_kernel converter for amd and implement the load
method(haven't tested yet)

* [feat](moe_kernel/moe.hpp): delete unused memory when using save

* [fix](moe_kernel): update PLAIN for pack

* [fix](moe_kernel): rm printf debug

* [fix](moe_kernel): skip gpu experts

* [fix](moe_kernel/moe.hpp): update include memory path

* [feat](moe_kernel/moe.hpp): support expert deferral

* [feat]: finish amd

* [ci]: add int4_1 & int4_1k

---------

Co-authored-by: mrhaoxx <mr.haoxx@gmail.com>
 2025-12-02 15:58:14 +08:00
Jianwei Dong
fd78fe520a fix(scripts): resolve OOM when converting gpu weights and update README (#1640) 2025-12-01 14:15:14 +08:00
mrhaoxx
637c49c83f [feat](kt-kernel): support qwen3-vl weights convert (#1648) 2025-11-27 22:29:09 +08:00
Jianwei Dong
c256150e08 update ci test (#1647) 2025-11-27 16:39:48 +08:00
ZiWei Yuan
1374b98ee5 [feat](moe_kernel): add amd blis support (int8) (#1600) 
* [feat]: init amd adaption

* [feat]: add blis support

* [fix]: fix setup and moe kernel warpper

* [fix](setup.py): support rebuild with cache and import kt_kernel works
fine

* [feat]: add moe_kernel converter for amd and implement the load
method(haven't tested yet)

* [feat](moe_kernel/moe.hpp): delete unused memory when using save

* [fix](moe_kernel): update PLAIN for pack

* [fix](moe_kernel): rm printf debug

* [fix](moe_kernel): skip gpu experts

* [fix](moe_kernel/moe.hpp): update include memory path

* [feat](moe_kernel/moe.hpp): support expert deferral

* [feat]: finish amd

---------

Co-authored-by: mrhaoxx <mr.haoxx@gmail.com>
 2025-11-27 12:08:53 +08:00
Jianwei Dong
fef6dd98a8 add accuracy and performance test (#1643) 2025-11-27 10:56:39 +08:00
Jiaqi Liao
e7d1c1de09 fix(llamafile): resolve deferred experts data race and update README (#1646) 2025-11-26 23:19:37 +08:00
Jianwei Dong
51745a9ea1 add ci (#1642) 2025-11-25 20:52:08 +08:00
DocShotgun
e72a4fb880 [feat](kt-kernel): Add resume arg to CPU weight conversion (#1630) 
* [feat]: kt-kernel: Add resume arg to CPU weight conversion

* [docs]: kt-kernel: Document resume arg for CPU weight conversion

* [fix]: kt-kernel: Only print resume layer if in use

* [fix]: kt-kernel: Don't log skipped layers when using resume_layer
 2025-11-22 12:00:15 +08:00
Jiaqi Liao
e69c67713f [refactor] fix third_party issue (#1632) 
* [refactor]: relocate third_party directory

* [fix]: fix custom_flashinfer for kt-sft
 2025-11-20 13:55:55 +08:00
ZiWei Yuan
aef6672dd8 [docs]: add contribuing guide and add hooks install (#1613) 
* [feat]: update kt-kernel hooks and add contribution guide

* [docs]: add contributing guide
* [style]: format the python file and cpp file in kt-kernel
 2025-11-15 18:26:49 +08:00
ZiWei Yuan
c32fefb1cd [doc]: update web doc and kt-kernel doc (#1609) 
* [doc]: update web doc and kt-kernel doc

* [doc](book.toml): add book.toml for rust book compile
 2025-11-13 20:44:13 +08:00
Jiaqi Liao
4bd0fe812b docs(kt-kernel): improve SGLang integration documentation and fix syntax errors (#1607) 
- Clarified instructions for SGLang integration with kt-kernel
 2025-11-13 19:23:00 +08:00
Jiaqi Liao
13b8ddecd9 AMXMoEWrapper -> KTMoEWrapper (#1604) 
fix import KTMoEWrapper
 2025-11-12 16:34:54 +08:00
Jiaqi Liao
02801c3c4e fix kt-kernel installation issue (#1603) 
* update README for kt-kernel for installation issues

* update install.sh

* update install.sh

* update install.sh

* update install.sh

* update install.sh

* update README

* fix sudo issue

* fix install issue

* fix import issue

* fix import issue

* update install.sh

* fix import issue

* fix import issue

* fix import issue

* update README

* update install

* update install
 2025-11-12 15:56:02 +08:00
Jiaqi Liao
d483147307 Fix kt-kernel compile issue (#1595) 
* update install.sh

* fix import issue

* update README
 2025-11-11 19:30:27 +08:00
ZiWei Yuan
a6bb7651f8 [build]: amd for llamafile right now (#1594) 2025-11-11 18:08:03 +08:00
Jiaqi Liao
bb1a667169 fix hwloc README (#1592) 2025-11-11 15:28:57 +08:00
Jiaqi Liao
743f6f719b update README for kt-kernel for installation issues (#1590) 2025-11-11 14:53:45 +08:00
Jiaqi Liao
94c25626dc Fix kt-kernel for new wrapper (#1588) 
* update README for kt-kernel

* style: format C++ and Python code in kt-kernel

  - Format C++ files: task_queue, ext_bindings, and MoE operators
  - Format Python utility modules: amx, llamafile, and loader
  - Improve code readability and consistency
 2025-11-10 21:47:34 +08:00
Jiaqi Liao
9bc00e587b Refactor KTMoEWrapper backend (#1587) 
* universal backend for cpu inference
* expert defer
 2025-11-10 20:26:15 +08:00
KMSorSMS
62d8685699 [fix]: update moe's physical to logical map 2025-11-07 16:29:04 +00:00
KMSorSMS
2641d15383 [fix]: add amx optimization 2025-11-07 15:51:56 +00:00
KMSorSMS
9509220b64 [feat]: Merge branch 'main' of https://github.com/kvcache-ai/ktransformers 2025-11-07 14:56:43 +00:00
KMSorSMS
7b88bb3d39 [fix]: fix precision 2025-11-07 14:56:05 +00:00
Oql
34c71ba8bf Merge pull request #1568 from kvcache-ai/add_bf16_scripts 
add convert_moe_to_bf16.py
 2025-11-07 17:55:38 +08:00
ouqingliang
a18f007d45 add convert_moe_to_bf16.py 2025-11-07 09:53:19 +00:00
Peilin Li
d939e56646 add the convert from fp8 to bf16 for Kimi-K2 model 2025-11-06 17:20:28 +08:00
KMSorSMS
85abac27c8 [build](cmake): fix target include bug 2025-11-05 08:04:12 +00:00
KMSorSMS
b70c44a959 [build](cmake): not error if blis not found 2025-11-05 05:05:39 +00:00
KMSorSMS
0c15da437f [feat](cmake & doc): fix bug with cmake arch detect & update doc for sft 2025-11-04 08:46:26 +00:00
chenht2022
6fe30af50d Merge branch 'main' into develop-cht 2025-11-03 14:35:44 +00:00
Jianwei Dong
9f2cb4787c Merge pull request #1542 from KMSorSMS/main 
[build]: fix amx cmake build support
 2025-11-03 20:01:32 +08:00
ovowei
f854d03bd7 update kt-kernel 2025-11-03 15:19:52 +08:00
KMSorSMS
b8f099c8b3 [build]: in case of missing, adding two more flags: -mamx-bf16 -mamx-int8s 2025-11-03 04:07:53 +00:00
KMSorSMS
49a49ade66 [build]: fix amx cmake build support 2025-11-03 03:58:36 +00:00
KMSorSMS
164b13adac [build]: fix cmake env settings bug 2025-11-02 04:49:27 +00:00
chenht2022
dd4377b60b feat: add deferred expert scheduling support 2025-10-31 08:03:37 +00:00
chenht2022
4a9b6cd99e Rename sync method parameters 2025-10-28 03:28:41 +00:00
ovowei
da4c68c7a0 update kt-kernel 2025-10-24 14:32:45 +08:00
ovowei
28d8663374 fix 2025-10-22 18:14:34 +08:00
Atream
4c5fcf9774 add kt-kernel 2025-10-12 05:13:00 +00:00