mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1658] ktransformers cannot be used in SGLang #1067
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1067
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fanzetian on GitHub (Dec 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1658
Originally assigned to: @SkqLiao on GitHub.
Reminder
System Info
docker image:
nvidia info in docker instance:
ktransformers version:
master code for commit:
637c49c83fI just use ktransformers/kt-kernel/install.sh to build kt-kernel
model info:
Qwen3-30B-A3B-Thinking-2507
gguf model info:
Qwen3-30B-A3B-Thinking-2507-F16.gguf
cpu info:
cpu just support avx ;128 core; 2 numa node
memory info: 2015G
Reproduction
Others
I want to run qwen3-30b-a3b model by gpu and cpu; ktransformers model weight is qwen3-30b-a3b-f16; the sglang program crash and stack is below; if I do not use ktransformers, everything is ok;
ang when sglang program hang, I find many progress is 'defunct';such as:
sglang error info:
@SkqLiao commented on GitHub (Dec 3, 2025):
Hi, We recently encountered a compatibility issue with the latest mainline of KT-kernel and SGLang. We are working on a solution, and currently it is possible to switch to the [ddd1440d0] commit of SGLang.
@fanzetian commented on GitHub (Dec 5, 2025):
The [ddd1440d0] is same error; my cpu is Intel Xeon Platinum 8358 , this is a third-generation cpu; but people for kt can run successfully in fourth-generation cpu [8581c];
Does it support third-generation cpus? How can I resolve this?
@Diffizle commented on GitHub (Dec 10, 2025):
same 8358 cpu, same error
@Diffizle commented on GitHub (Dec 12, 2025):
comment the lines ralated to
__AVX512BF16__inggml.cworks for me.void ggml_fp32_to_bf16_row(const float * x, ggml_bf16_t * y, int64_t n) {
int i = 0;
// #if defined(AVX512BF16)
// for (; i + 32 <= n; i += 32) {
// _mm512_storeu_si512(
// (__m512i *)(y + i),
// m512i(_mm512_cvtne2ps_pbh(_mm512_loadu_ps(x + i + 16),
// _mm512_loadu_ps(x + i))));
// }
// #endif
for (; i < n; i++) {
y[i] = GGML_FP32_TO_BF16(x[i]);
}
}
It seems that some versions of
gcccan successfully compile AVX512BF16, even if the cpu does not support it.@SkqLiao commented on GitHub (Dec 12, 2025):
Thank you for your feedback, that's interesting... It appears that this CPU instruction set problem may be the cause of similar problems for many users.
We will study this issue carefully. thx.
@SkqLiao commented on GitHub (Dec 16, 2025):
@Diffizle @fanzetian Hi, I rented a machine with the same CPU from autodl and reproduced your issue. The root cause is that CMake is not correctly detecting the AVX512 instruction set (especially AVX512-VNNI and AVX512-BF16). The AVX512-BF16 instruction set requires CPUs with the Cooper Lake architecture or later, while Ice Lake only supports AVX512-VNNI. We are working on fixing this issue and implementing some fallbacks for lower instruction sets. You can try using the cpu-fix branch; it should fix the issue on this cpu.