[GH-ISSUE #1248] [Bug] Windows built without balance_serve successfully but failed while loading #844

New issue

Closed

opened 2026-10-29 14:45:14 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:45:14 +00:00

Owner

Originally created by @Ryuukeisyou on GitHub (May 1, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1248

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Hi, I am trying to run ktransformers natively in windows. With some changes, it build sucessfully, but failed at loading model.

Reproduction

I have done the following to try to start, without balance_serve, since balance_serve seem to require a lot more fixes to work in windows.

change triton to triton-windows in pyproject.toml
change use of aligned_alloc in csrc/ktransformers_ext/cpu_backend/shared_mem_buffer.cpp

from

buffer_ = std::aligned_alloc(64, size);

#ifdef _WIN32
#include <malloc.h>
buffer_ = _aligned_malloc(size, 64);
#else
buffer_ = std::aligned_alloc(64, size);
#endif

use a mockup for sched_ext to bypass error
build with USE_BALANCE_SERVE=0

the build and install process was successful without errors, and then I try to run DeepSeek-R1-UD-IQ1_S with

python -m ktransformers.local_chat /
--model_path \DeepSeek-R1 /
--gguf_path DeepSeek-R1-UD-IQ1_S /
--optimize_config_path \optimize_rules\DeepSeek-V3-Chat.yaml /
--cpu_infer 28 /
--backend_type ktransformers

which used to work in version 0.2.3post2.

However, the process exited without showing any exceptions, even if I use debug mode with vscode. here is the last console output when it exited:

loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_q_a_norm.weight to cuda:0
loading blk.1.attn_kv_a_norm.weight to cuda:0
loading blk.1.attn_kv_b.weight to cuda:0
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_q_a_norm.weight to cuda:0
loading blk.2.attn_kv_a_norm.weight to cuda:0
loading blk.2.attn_kv_b.weight to cuda:0
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_q_a_norm.weight to cuda:0
loading blk.3.attn_kv_a_norm.weight to cuda:0
loading blk.3.attn_kv_b.weight to cuda:0

Any help or advice on how to debug would be appreciated!

Environment

sys windows 11
pytorch 2.6+cu126
gpu nvidia4090

Originally created by @Ryuukeisyou on GitHub (May 1, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1248 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug Hi, I am trying to run ktransformers natively in windows. With some changes, it build sucessfully, but failed at loading model. ### Reproduction I have done the following to try to start, without balance_serve, since balance_serve seem to require a lot more fixes to work in windows. + change triton to triton-windows in pyproject.toml + change use of aligned_alloc in csrc/ktransformers_ext/cpu_backend/shared_mem_buffer.cpp from ``` buffer_ = std::aligned_alloc(64, size); ``` to ``` #ifdef _WIN32 #include <malloc.h> buffer_ = _aligned_malloc(size, 64); #else buffer_ = std::aligned_alloc(64, size); #endif ``` + use a mockup for sched_ext to bypass error + build with USE_BALANCE_SERVE=0 the build and install process was successful without errors, and then I try to run DeepSeek-R1-UD-IQ1_S with ``` python -m ktransformers.local_chat / --model_path \DeepSeek-R1 / --gguf_path DeepSeek-R1-UD-IQ1_S / --optimize_config_path \optimize_rules\DeepSeek-V3-Chat.yaml / --cpu_infer 28 / --backend_type ktransformers ``` which used to work in version 0.2.3post2. However, the process exited **without showing any exceptions**, even if I use debug mode with vscode. here is the last console output when it exited: ``` loading token_embd.weight to cpu loading token_embd.weight with CPU loading blk.0.attn_q_a_norm.weight to cuda:0 loading blk.0.attn_kv_a_norm.weight to cuda:0 loading blk.0.attn_kv_b.weight to cuda:0 loading blk.0.attn_norm.weight to cuda:0 loading blk.0.ffn_norm.weight to cuda:0 loading blk.1.attn_q_a_norm.weight to cuda:0 loading blk.1.attn_kv_a_norm.weight to cuda:0 loading blk.1.attn_kv_b.weight to cuda:0 loading blk.1.attn_norm.weight to cuda:0 loading blk.1.ffn_norm.weight to cuda:0 loading blk.2.attn_q_a_norm.weight to cuda:0 loading blk.2.attn_kv_a_norm.weight to cuda:0 loading blk.2.attn_kv_b.weight to cuda:0 loading blk.2.attn_norm.weight to cuda:0 loading blk.2.ffn_norm.weight to cuda:0 loading blk.3.attn_q_a_norm.weight to cuda:0 loading blk.3.attn_kv_a_norm.weight to cuda:0 loading blk.3.attn_kv_b.weight to cuda:0 ``` Any help or advice on how to debug would be appreciated! ### Environment sys windows 11 pytorch 2.6+cu126 gpu nvidia4090

bsakaguchi closed this issue

2026-10-29 14:45:14 +00:00

bsakaguchi commented

2026-10-29 14:45:15 +00:00

Author

Owner

@Ryuukeisyou commented on GitHub (May 2, 2025):

The problem is solved. For anyone who is interested, in fact I forgot to change also the free method from free() to _aligned_free().

void aligned_free(void* ptr) {
    #ifdef _WIN32
        _aligned_free(ptr);
    #else
        std::free(ptr);
    #endif
}

and then change all free() call to this method.

Now ktransformers is running normally on Windows, without balance_serve. The balance serve part still seems to require a lot work, which I have not delt with yet.

@Ryuukeisyou commented on GitHub (May 2, 2025): The problem is solved. For anyone who is interested, in fact I forgot to change also the free method from ```free()``` to ```_aligned_free()```. ``` void aligned_free(void* ptr) { #ifdef _WIN32 _aligned_free(ptr); #else std::free(ptr); #endif } ``` and then change all free() call to this method. Now ktransformers is running normally on Windows, without balance_serve. The balance serve part still seems to require a lot work, which I have not delt with yet.