kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-04-26 17:28:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix/sglang kt detection (#1875)

Browse Source 
* [feat]: simplify sglang installation with submodule, auto-sync CI, and version alignment

- Add kvcache-ai/sglang as git submodule at third_party/sglang (branch = main)
- Add top-level install.sh for one-click source installation (sglang + kt-kernel)
- Add sglang-kt as hard dependency in kt-kernel/pyproject.toml
- Add CI workflow to auto-sync sglang submodule daily and create PR
- Add CI workflow to build and publish sglang-kt to PyPI
- Integrate sglang-kt build into release-pypi.yml (version.py bump publishes both packages)
- Align sglang-kt version with ktransformers via SGLANG_KT_VERSION env var injection
- Update Dockerfile to use submodule and inject aligned version
- Update all 13 doc files, CLI hints, and i18n strings to reference new install methods

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [build]: bump version to 0.5.2

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [build]: rename PyPI package from kt-kernel to ktransformers

Users can now `pip install ktransformers` to get everything
(sglang-kt is auto-installed as a dependency).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "[build]: rename PyPI package from kt-kernel to ktransformers"

This reverts commit e0cbbf6364.

* [build]: add ktransformers meta-package for PyPI

`pip install ktransformers` now works as a single install command.
It pulls kt-kernel (which in turn pulls sglang-kt).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [fix]: show sglang-kt package version in kt version command

- Prioritize sglang-kt package version (aligned with ktransformers)
  over sglang internal __version__
- Update display name from "sglang" to "sglang-kt"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [fix]: improve sglang-kt detection in kt doctor and kt version

Recognize sglang-kt package name as proof of kvcache-ai fork installation.
Previously both commands fell through to "PyPI (not recommended)" for
non-editable local source installs. Now version.py reuses the centralized
check_sglang_installation() logic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [build]: bump version to 0.5.2.post1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Jianwei Dong
2026-03-04 16:54:48 +08:00
committed by GitHub
parent 9e69fccb02
commit 15c624dcae
29 changed files with 787 additions and 179 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
kt-kernel/README.md
View File

@@ -262,12 +262,23 @@ KT-Kernel can be used standalone via [Direct Python API](#direct-python-api-usag
#### 1. Install SGLang
Install the kvcache-ai fork of SGLang (required for kt-kernel support):
```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"
# Option A: One-click install (from ktransformers root, installs sglang + kt-kernel)
./install.sh
# Option B: pip install
pip install sglang-kt
# Option C: From source (editable mode)
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install -e "third_party/sglang/python[all]"
```
> **Important:** Use `sglang-kt` (kvcache-ai fork), not the official `sglang` package. If you have the official version installed, uninstall it first: `pip uninstall sglang -y`
#### 2. Prepare Weights
You need both GPU weights and CPU-side expert weights for heterogeneous inference. The exact format depends on the backend:

17
kt-kernel/README_zh.md
View File

@@ -115,12 +115,23 @@ KT-Kernel 可以单独通过 [Python API](#直接使用-python-api) 使用，也
#### 1. 安装 SGLang
安装 kvcache-ai 分支的 SGLang（kt-kernel 需要此分支）：
```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"
# 方式 A: 一键安装（从 ktransformers 根目录，同时安装 sglang + kt-kernel）
./install.sh
# 方式 B: pip 安装
pip install sglang-kt
# 方式 C: 从源码安装（可编辑模式）
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install -e "third_party/sglang/python[all]"
```
> **重要:** 请使用 `sglang-kt`（kvcache-ai 分支），而非官方 `sglang` 包。如已安装官方版本，请先卸载：`pip uninstall sglang -y`
#### 2. 准备权重
要进行异构推理，需要同时准备 GPU 权重和 CPU 侧 experts 对应的权重，具体格式取决于后端类型：

2
kt-kernel/pyproject.toml
View File

@@ -33,6 +33,8 @@ dependencies = [
"pyyaml>=6.0",
"httpx>=0.25.0",
"packaging>=23.0",
# SGLang (kvcache-ai fork)
"sglang-kt",
# Development dependencies
"black>=25.9.0",
]

18
kt-kernel/python/cli/commands/doctor.py
View File

@@ -369,7 +369,19 @@ def doctor(
sglang_info = check_sglang_installation()
if sglang_info["installed"]:
if sglang_info["from_source"]:
if sglang_info.get("is_kvcache_fork"):
# Package name is sglang-kt — this is definitively the kvcache-ai fork
if sglang_info["from_source"] and sglang_info["git_info"]:
git_remote = sglang_info["git_info"].get("remote", "unknown")
git_branch = sglang_info["git_info"].get("branch", "unknown")
sglang_source_value = f"sglang-kt (Source: {git_remote}, branch: {git_branch})"
elif sglang_info["editable"]:
sglang_source_value = "sglang-kt (editable)"
else:
sglang_source_value = "sglang-kt"
sglang_source_status = "ok"
sglang_source_hint = None
elif sglang_info["from_source"]:
if sglang_info["git_info"]:
git_remote = sglang_info["git_info"].get("remote", "unknown")
git_branch = sglang_info["git_info"].get("branch", "unknown")
@@ -381,7 +393,7 @@ def doctor(
sglang_source_status = "ok"
sglang_source_hint = None
else:
sglang_source_value = "PyPI (not recommended)"
sglang_source_value = "PyPI sglang (not kvcache-ai fork)"
sglang_source_status = "warning"
sglang_source_hint = t("sglang_pypi_hint")
else:
@@ -411,7 +423,7 @@ def doctor(
else:
kt_kernel_value = t("sglang_kt_kernel_not_supported")
kt_kernel_status = "error"
kt_kernel_hint = 'Reinstall SGLang from: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"'
kt_kernel_hint = "Reinstall SGLang: pip uninstall sglang -y && pip install sglang-kt (or run ./install.sh from ktransformers root)"
issues_found = True
checks.append(

72
kt-kernel/python/cli/commands/version.py
View File

@@ -16,54 +16,38 @@ from kt_kernel.cli.utils.environment import detect_cuda_version, get_installed_p
def _get_sglang_info() -> str:
"""Get sglang version and installation source information."""
try:
import sglang
"""Get sglang-kt version and installation source information."""
from kt_kernel.cli.utils.sglang_checker import check_sglang_installation
version = getattr(sglang, "__version__", None)
info = check_sglang_installation()
if not version:
version = get_installed_package_version("sglang")
if not version:
return t("version_not_installed")
# Try to detect installation source
from pathlib import Path
import subprocess
if hasattr(sglang, "__file__") and sglang.__file__:
location = Path(sglang.__file__).parent.parent
git_dir = location / ".git"
if git_dir.exists():
# Installed from git (editable install)
try:
# Get remote URL
result = subprocess.run(
["git", "remote", "get-url", "origin"],
cwd=location,
capture_output=True,
text=True,
timeout=2,
)
if result.returncode == 0:
remote_url = result.stdout.strip()
# Simplify GitHub URLs
if "github.com" in remote_url:
repo_name = remote_url.split("/")[-1].replace(".git", "")
owner = remote_url.split("/")[-2]
return f"{version} [dim](GitHub: {owner}/{repo_name})[/dim]"
return f"{version} [dim](Git: {remote_url})[/dim]"
except (subprocess.TimeoutExpired, FileNotFoundError, OSError):
pass
# Default: installed from PyPI
return f"{version} [dim](PyPI)[/dim]"
except ImportError:
if not info["installed"]:
return t("version_not_installed")
# Get version from package metadata (prefer sglang-kt)
version = get_installed_package_version("sglang-kt")
if not version:
version = get_installed_package_version("sglang")
if not version:
version = info.get("version") or "unknown"
# Determine source label
if info.get("is_kvcache_fork"):
if info["from_source"] and info.get("git_info"):
git_remote = info["git_info"].get("remote", "")
return f"{version} [dim](Source: {git_remote})[/dim]"
elif info["editable"]:
return f"{version} [dim](editable)[/dim]"
else:
return f"{version} [dim](sglang-kt)[/dim]"
elif info["from_source"]:
if info.get("git_info"):
git_remote = info["git_info"].get("remote", "")
return f"{version} [dim](Source: {git_remote})[/dim]"
return f"{version} [dim](source)[/dim]"
else:
return f"{version} [dim](PyPI)[/dim]"
def version(
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed version info"),

20
kt-kernel/python/cli/i18n.py
View File

@@ -37,7 +37,7 @@ MESSAGES: dict[str, dict[str, str]] = {
"version_cuda_not_found": "Not found",
"version_kt_kernel": "kt-kernel",
"version_ktransformers": "ktransformers",
"version_sglang": "sglang",
"version_sglang": "sglang-kt",
"version_llamafactory": "llamafactory",
"version_not_installed": "Not installed",
# Install command
@@ -300,10 +300,10 @@ MESSAGES: dict[str, dict[str, str]] = {
"completion_next_session": "Completion will be automatically enabled in new terminal sessions.",
# SGLang
"sglang_not_found": "SGLang not found",
"sglang_pypi_warning": "SGLang from PyPI may not be compatible with kt-kernel",
"sglang_pypi_hint": 'SGLang from PyPI may not be compatible. Install from source: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_install_hint": 'Install SGLang: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_recommend_source": 'Recommend reinstalling from source: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_pypi_warning": "SGLang from PyPI may not be compatible with kt-kernel. Use sglang-kt instead: pip install sglang-kt",
"sglang_pypi_hint": "SGLang from PyPI may not be compatible. Install the kvcache-ai fork: pip install sglang-kt (or run ./install.sh from ktransformers root)",
"sglang_install_hint": "Install SGLang: pip install sglang-kt (or run ./install.sh from ktransformers root)",
"sglang_recommend_source": "Recommend reinstalling with the kvcache-ai fork: pip uninstall sglang -y && pip install sglang-kt",
"sglang_kt_kernel_not_supported": "SGLang does not support kt-kernel (missing --kt-gpu-prefill-token-threshold parameter)",
"sglang_checking_kt_kernel_support": "Checking SGLang kt-kernel support...",
"sglang_kt_kernel_supported": "SGLang kt-kernel support verified",
@@ -657,7 +657,7 @@ MESSAGES: dict[str, dict[str, str]] = {
"version_cuda_not_found": "未找到",
"version_kt_kernel": "kt-kernel",
"version_ktransformers": "ktransformers",
"version_sglang": "sglang",
"version_sglang": "sglang-kt",
"version_llamafactory": "llamafactory",
"version_not_installed": "未安装",
# Install command
@@ -920,10 +920,10 @@ MESSAGES: dict[str, dict[str, str]] = {
"completion_next_session": "新的终端会话将自动启用补全。",
# SGLang
"sglang_not_found": "未找到 SGLang",
"sglang_pypi_warning": "PyPI 版本的 SGLang 可能与 kt-kernel 不兼容",
"sglang_pypi_hint": 'PyPI 版本可能不兼容。从源码安装: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_install_hint": '安装 SGLang: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_recommend_source": '建议从源码重新安装: git clone https://github.com/kvcache-ai/sglang && cd sglang && pip install -e "python[all]"',
"sglang_pypi_warning": "PyPI 版本的 SGLang 可能与 kt-kernel 不兼容。请使用 sglang-kt: pip install sglang-kt",
"sglang_pypi_hint": "PyPI 版本可能不兼容。安装 kvcache-ai 分支: pip install sglang-kt (或在 ktransformers 根目录运行 ./install.sh)",
"sglang_install_hint": "安装 SGLang: pip install sglang-kt (或在 ktransformers 根目录运行 ./install.sh)",
"sglang_recommend_source": "建议重新安装 kvcache-ai 分支: pip uninstall sglang -y && pip install sglang-kt",
"sglang_kt_kernel_not_supported": "SGLang 不支持 kt-kernel (缺少 --kt-gpu-prefill-token-threshold 参数)",
"sglang_checking_kt_kernel_support": "正在检查 SGLang kt-kernel 支持...",
"sglang_kt_kernel_supported": "SGLang kt-kernel 支持已验证",

84
kt-kernel/python/cli/utils/sglang_checker.py
View File

@@ -38,15 +38,25 @@ def check_sglang_installation() -> dict:
editable = False
git_info = None
from_source = False
is_kvcache_fork = False # True if installed as sglang-kt package
try:
# Get pip show output
# Get pip show output (try sglang-kt first, then sglang)
result = subprocess.run(
[sys.executable, "-m", "pip", "show", "sglang"],
[sys.executable, "-m", "pip", "show", "sglang-kt"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0:
is_kvcache_fork = True # sglang-kt package name proves it's the fork
else:
result = subprocess.run(
[sys.executable, "-m", "pip", "show", "sglang"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0:
pip_info = {}
@@ -128,6 +138,7 @@ def check_sglang_installation() -> dict:
"editable": editable,
"git_info": git_info,
"from_source": from_source,
"is_kvcache_fork": is_kvcache_fork,
}
except ImportError:
return {
@@ -137,6 +148,7 @@ def check_sglang_installation() -> dict:
"editable": False,
"git_info": None,
"from_source": False,
"is_kvcache_fork": False,
}
@@ -158,20 +170,19 @@ def get_sglang_install_instructions(lang: Optional[str] = None) -> str:
return """
[bold yellow]SGLang \u672a\u5b89\u88c5[/bold yellow]
\u8bf7\u6309\u7167\u4ee5\u4e0b\u6b65\u9aa4\u5b89\u88c5 SGLang:
\u8bf7\u9009\u62e9\u4ee5\u4e0b\u65b9\u5f0f\u4e4b\u4e00\u5b89\u88c5 SGLang (kvcache-ai \u5206\u652f):
[bold]1. \u514b\u9686\u4ed3\u5e93:[/bold]
git clone https://github.com/kvcache-ai/sglang.git
cd sglang
[bold]\u65b9\u5f0f A - \u4e00\u952e\u5b89\u88c5 (\u63a8\u8350):[/bold]
\u4ece ktransformers \u6839\u76ee\u5f55\u8fd0\u884c:
[cyan]./install.sh[/cyan]
[bold]2. \u5b89\u88c5 (\u4e8c\u9009\u4e00):[/bold]
[bold]\u65b9\u5f0f B - pip \u5b89\u88c5:[/bold]
[cyan]pip install sglang-kt[/cyan]
[cyan]\u65b9\u5f0f A - pip \u5b89\u88c5 (\u63a8\u8350):[/cyan]
pip install -e "python[all]"
[cyan]\u65b9\u5f0f B - uv \u5b89\u88c5 (\u66f4\u5feb):[/cyan]
pip install uv
uv pip install -e "python[all]"
[bold]\u65b9\u5f0f C - \u4ece\u6e90\u7801\u5b89\u88c5:[/bold]
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install "third_party/sglang/python[all]"
[dim]\u6ce8\u610f: \u8bf7\u786e\u4fdd\u5728\u6b63\u786e\u7684 Python \u73af\u5883\u4e2d\u6267\u884c\u4ee5\u4e0a\u547d\u4ee4[/dim]
"""
@@ -179,20 +190,19 @@ def get_sglang_install_instructions(lang: Optional[str] = None) -> str:
return """
[bold yellow]SGLang is not installed[/bold yellow]
Please follow these steps to install SGLang:
Install SGLang (kvcache-ai fork) using one of these methods:
[bold]1. Clone the repository:[/bold]
git clone https://github.com/kvcache-ai/sglang.git
cd sglang
[bold]Option A - One-click install (recommended):[/bold]
From the ktransformers root directory, run:
[cyan]./install.sh[/cyan]
[bold]2. Install (choose one):[/bold]
[bold]Option B - pip install:[/bold]
[cyan]pip install sglang-kt[/cyan]
[cyan]Option A - pip install (recommended):[/cyan]
pip install -e "python[all]"
[cyan]Option B - uv install (faster):[/cyan]
pip install uv
uv pip install -e "python[all]"
[bold]Option C - From source:[/bold]
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install "third_party/sglang/python[all]"
[dim]Note: Make sure to run these commands in the correct Python environment[/dim]
"""
@@ -369,17 +379,18 @@ def print_sglang_kt_kernel_instructions() -> None:
您当前安装的 SGLang 不包含 kt-kernel 支持。
kt-kernel 需要使用 kvcache-ai 维护的 SGLang 分支。
[bold]请按以下步骤重新安装 SGLang:[/bold]
[bold]请按以下步骤重新安装:[/bold]
[cyan]1. 卸载当前的 SGLang:[/cyan]
pip uninstall sglang -y
[cyan]2. 克隆 kvcache-ai 的 SGLang 仓库:[/cyan]
git clone https://github.com/kvcache-ai/sglang.git
cd sglang
[cyan]2. 安装 kvcache-ai 版本 (选择一种方式):[/cyan]
[cyan]3. 安装 SGLang:[/cyan]
pip install -e "python[all]"
[bold]方式 A - 一键安装 (推荐):[/bold]
从 ktransformers 根目录运行: ./install.sh
[bold]方式 B - pip 安装:[/bold]
pip install sglang-kt
[dim]注意: 请确保在正确的 Python 环境中执行以上命令[/dim]
"""
@@ -390,17 +401,18 @@ kt-kernel 需要使用 kvcache-ai 维护的 SGLang 分支。
Your current SGLang installation does not include kt-kernel support.
kt-kernel requires the kvcache-ai maintained fork of SGLang.
[bold]Please reinstall SGLang with the following steps:[/bold]
[bold]Please reinstall SGLang:[/bold]
[cyan]1. Uninstall current SGLang:[/cyan]
pip uninstall sglang -y
[cyan]2. Clone the kvcache-ai SGLang repository:[/cyan]
git clone https://github.com/kvcache-ai/sglang.git
cd sglang
[cyan]2. Install the kvcache-ai fork (choose one):[/cyan]
[cyan]3. Install SGLang:[/cyan]
pip install -e "python[all]"
[bold]Option A - One-click install (recommended):[/bold]
From the ktransformers root directory, run: ./install.sh
[bold]Option B - pip install:[/bold]
pip install sglang-kt
[dim]Note: Make sure to run these commands in the correct Python environment[/dim]
"""