[GH-ISSUE #860] why not use KExpertsTorch device=cpu for `KExpertsCPU`, but use c++ to compute? #621

New issue

Closed

opened 2026-10-29 14:42:02 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:42:02 +00:00

Owner

Originally created by @dragen1860 on GitHub (Mar 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/860

Dear all:
since pytorch can also compute with device=cpu for function KExpertsTorch, why the author use C++ to implement KExpertsCPU function? thanks.

Originally created by @dragen1860 on GitHub (Mar 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/860 Dear all: since pytorch can also compute with device=cpu for function `KExpertsTorch`, why the author use C++ to implement `KExpertsCPU` function? thanks.

bsakaguchi closed this issue

2026-10-29 14:42:02 +00:00

bsakaguchi commented

2026-10-29 14:42:03 +00:00

Author

Owner

@ovowei commented on GitHub (Mar 11, 2025):

Our implementation is based on llamafile, which provides better performance compared to Torch CPU computation. I don't remember the exact benchmark results, but you can test it yourself using the benchmarks available in the following directory:
bench

@ovowei commented on GitHub (Mar 11, 2025): Our implementation is based on llamafile, which provides better performance compared to Torch CPU computation. I don't remember the exact benchmark results, but you can test it yourself using the benchmarks available in the following directory: [bench](https://github.com/kvcache-ai/ktransformers/tree/main/ktransformers/ktransformers_ext/bench)

bsakaguchi referenced this issue

2026-10-29 14:52:24 +00:00

[PR #621] [MERGED] Feat moonlight #1436

bsakaguchi referenced this issue

2026-10-29 14:52:30 +00:00

[PR #657] [MERGED] Feat absorb for long prefill #1447