[GH-ISSUE #1657] 能不能基于sglang官方非ascend镜像提供一个使用ktransformers运行llamafile格式模型的demo #1066

New issue

Open

opened 2026-10-29 14:48:10 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:48:10 +00:00

Owner

Originally created by @fanzetian on GitHub (Dec 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1657

Originally assigned to: @SkqLiao on GitHub.

Reminder

I have read the above rules and searched the existing issues.

Description

能不能基于sglang官方非ascend镜像提供集成使用ktransformers运行llamafile格式模型的demo
比如：

Pull Request

No response

Originally created by @fanzetian on GitHub (Dec 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1657 Originally assigned to: @SkqLiao on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description 能不能基于sglang官方非ascend镜像提供集成使用ktransformers运行llamafile格式模型的demo 比如： ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:48:10 +00:00

bsakaguchi commented

2026-10-29 14:48:10 +00:00

Author

Owner

@SkqLiao commented on GitHub (Dec 3, 2025):

Hi, we will be providing Docker images for KT-Kernel and KT-SFT in the upcoming next release.

Currently, you can try using the image approachingai/sglang-kt:latest, which uses the AMX instruction set by default. To support AVX, you can re-run ./install.sh within Docker to install KT-Kernel compatible with your CPU's instruction set (the installation script will automatically detect the cpu instruction set; you can also use git pull to fetch the latest code first).

For the startup process, you can refer to README_zh, which demonstrates how to start the Qwen3-30B-A3B model using BF16 (GPU) + Q4_K_M (GGUF).

@SkqLiao commented on GitHub (Dec 3, 2025): Hi, we will be providing Docker images for KT-Kernel and KT-SFT in the upcoming next release. Currently, you can try using the image **approachingai/sglang-kt:latest**, which uses the AMX instruction set by default. To support AVX, you can re-run `./install.sh` within Docker to install KT-Kernel compatible with your CPU's instruction set (the installation script will automatically detect the cpu instruction set; you can also use `git pull` to fetch the latest code first). For the startup process, you can refer to [README_zh](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README_zh.md#%E5%AE%8C%E6%95%B4%E7%A4%BA%E4%BE%8Bqwen3-30b-a3b), which demonstrates how to start the Qwen3-30B-A3B model using BF16 (GPU) + Q4_K_M (GGUF).