[PR #1031] [MERGED] 文档更新：model_path名字要求以及在示例中添加force_think #1540

New issue

Closed

opened 2026-10-29 14:53:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:53:19 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/1031
Author: @wangkuigang-yewu-cmss
Created: 4/3/2025
Status: ✅ Merged
Merged: 4/3/2025
Merged by: @qiyuxinlin

Base: main ← Head: doc-update

📝 Commits (1)

c590583 doc upgrade: model_path requirements and reasoning

📊 Changes

1 file changed (+5 additions, -1 deletions)

View changed files

📝 doc/en/balance-serve.md (+5 -1)

📄 Description

在 issue #1018 中，有多个用户遇到了这样的问题：

在推理开始时，rpc进程退出，并且在rpc.log中可以看到如下日志：

terminate called after throwing an instance of 'std::out_of_range'

经过gdb排查，发现异常发生在:

#11 kvc2::CacheInfo::hidden_layer_count (this=this@entry=0x7696ba9ff620)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:50

相关代码如下：

size_t CacheInfo::hidden_layer_count() {
  return model_configs.at(model_name).num_hidden_layers;
}

gdb中查看：model_name 为用户在--model_path参数中指定的路径名的末尾路径，而model_configs看起来是./ktransformers/configs/model_configs.json中配置的模型名。

如果用户使用的路径并非标准模型名，那么这里肯定会崩掉。

目前最简单的解决办法，就是在文档中说明一下，让用户明确使用标准的模型名作为路径名称。

此外关于--force_think，相信大多数用户本地部署时都会优先考虑R1，并且也会想要R1输出深度思考过程，所以这个选项对于大多数用户都非常有用。建议在文档中也带上这个选项。

附录：

gdb backtrace

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x000076983b64527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x000076983b6288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x000076983aca5ff5 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x000076983acbb0da in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x000076983aca5a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x000076983acbb391 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x000076983aca932d in std::__throw_out_of_range(char const*) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x0000769716861f1c in std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ModelConfig, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ModelConfig> > >::at (
    __k=<error reading variable: Cannot access memory at address 0x6b6565737065654c>, this=0x7697168ff0a0 <model_configs>)
    at /usr/include/c++/13/bits/stl_map.h:553
#11 kvc2::CacheInfo::hidden_layer_count (this=this@entry=0x7696ba9ff620)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:50
#12 0x000076971686aa90 in kvc2::DoubleCacheHandle::set_cache_info (this=this@entry=0x769540013770, quant_type="BF16",
    turn_on_k_cache=turn_on_k_cache@entry=true, turn_on_v_cache=turn_on_v_cache@entry=false, model_name=...)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:743
#13 0x00007697168c9b16 in kvc2::KVC2::lookup (this=this@entry=0x1ed10500, model_name="deepseek-r1-info", quant_type="BF16",
    id=id@entry=0x76954000b480, length=<optimized out>, length@entry=5, estimated_length=8192)
    at /usr/include/c++/13/bits/allocator.h:184
#14 0x00007697168c9339 in kvc2::KVC2::lookup_to_gpu_async(std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned long, unsigned long, std::function<void (std::shared_ptr<kvc2::DoubleCacheHandleInterface>)>) (this=this@entry=0x1ed10500, model_name="deepseek-r1-info",
    quant_type="BF16", id=id@entry=0x76954000b480, length=length@entry=5, estimated_length=estimated_length@entry=8192,
    call_back=...) at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:1279
#15 0x00007697e66a6478 in scheduler::Query::to_status (this=0x769540006320, to=to@entry=scheduler::Query::Preparing)
    at /usr/include/c++/13/bits/allocator.h:184
#16 0x00007697e66f8a4b in scheduler::FCFS_single_prefill::strategy_prepare (prepare=..., this=0x1ef08990)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:801
#17 scheduler::FCFS_single_prefill::strategy_prepare (prepare=..., this=0x1ef08990)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:798
#18 scheduler::QueryMaintainer::tackle_event (event=..., this=0x1ef08990)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:607
#19 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}::operator()<scheduler::EventPrepare>(scheduler::EventPrepare) const (event=..., __closure=<optimized out>)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:650
#20 std::__invoke_impl<void, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, scheduler::EventPrepare&>(std::__invoke_other, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, scheduler::EventPrepare&) (__f=...) at /usr/include/c++/13/bits/invoke.h:61
#21 std::__invoke<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, scheduler::EventPrepare&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, scheduler::EventPrepare&) (
    __fn=...) at /usr/include/c++/13/bits/invoke.h:96
#22 std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&)>, std::integer_sequence<unsigned long, 3ul> >::__visit_invoke(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (
    __vars#0=std::variant [index 3] = {...}, __visitor=...) at /usr/include/c++/13/variant:1060
#23 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, sched--Type <RET> for more, q to quit, c to continue without paging--
uler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...)
    at /usr/include/c++/13/variant:1818
#24 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...)
    at /usr/include/c++/13/variant:1756
#25 std::visit<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1878
#26 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const (__closure=<optimized out>)
    at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:639
#27 0x000076983a7549c0 in execute_native_thread_routine ()
   from /home/wkgcass/.virtualenvs/ktnew/lib/python3.12/site-packages/torch/lib/libc10.so
#28 0x000076983b69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#29 0x000076983b729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

gdb print

frame 11
print model_name
$1 = "deepseek-r1-info"

print model_configs
$2 = std::map with 12 elements = {["DeepSeek-Coder-V2-Instruct"] = {hidden_size = 5120, intermediate_size = 12288,
    max_position_embeddings = 163840, model_type = "deepseek_v2", num_attention_heads = 128, num_hidden_layers = 60,
    num_key_value_heads = 128, vocab_size = 102400}, ["DeepSeek-R1"] = {hidden_size = 7168, intermediate_size = 18432,
    max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128, num_hidden_layers = 61,
    num_key_value_heads = 128, vocab_size = 129280}, ["DeepSeek-V2-Lite-Chat"] = {hidden_size = 2048,
    intermediate_size = 10944, max_position_embeddings = 163840, model_type = "deepseek_v2", num_attention_heads = 16,
    num_hidden_layers = 27, num_key_value_heads = 16, vocab_size = 102400}, ["DeepSeek-V3"] = {hidden_size = 7168,
    intermediate_size = 18432, max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128,
    num_hidden_layers = 3, num_key_value_heads = 128, vocab_size = 129280}, ["DeepSeek-V3-bf16"] = {hidden_size = 7168,
    intermediate_size = 18432, max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128,
    num_hidden_layers = 61, num_key_value_heads = 128, vocab_size = 129280}, ["LLaMA-2-7B-32K"] = {hidden_size = 4096,
    intermediate_size = 11008, max_position_embeddings = 32768, model_type = "llama", num_attention_heads = 32,
    num_hidden_layers = 32, num_key_value_heads = 32, vocab_size = 32000}, ["Moonlight-16B-A3B-Instruct"] = {
    hidden_size = 2048, intermediate_size = 11264, max_position_embeddings = 8192, model_type = "deepseek_v3",
    num_attention_heads = 16, num_hidden_layers = 27, num_key_value_heads = 16, vocab_size = 163840},
  ["Qwen2.5-32B-Instruct"] = {hidden_size = 5120, intermediate_size = 27648, max_position_embeddings = 32768,
    model_type = "qwen2", num_attention_heads = 40, num_hidden_layers = 64, num_key_value_heads = 8, vocab_size = 152064},
  ["Qwen2.5-32B-Instruct-GPTQ-Int4"] = {hidden_size = 5120, intermediate_size = 27648, max_position_embeddings = 32768,
    model_type = "qwen2", num_attention_heads = 40, num_hidden_layers = 64, num_key_value_heads = 8, vocab_size = 152064},
  ["Qwen2.5-7B-Instruct"] = {hidden_size = 3584, intermediate_size = 18944, max_position_embeddings = 32768,
    model_type = "qwen2", num_attention_heads = 28, num_hidden_layers = 28, num_key_value_heads = 4, vocab_size = 152064},
  ["Qwen2.5-7B-Instruct-GPTQ-Int4"] = {hidden_size = 3584, intermediate_size = 18944, max_position_embeddings = 32768,
    model_type = "qwen2", num_attention_heads = 28, num_hidden_layers = 28, num_key_value_heads = 4, vocab_size = 152064},
  ["qwen2-72b-instruct"] = {hidden_size = 8192, intermediate_size = 29568, max_position_embeddings = 32768,
    model_type = "qwen2", num_attention_heads = 64, num_hidden_layers = 80, num_key_value_heads = 8, vocab_size = 152064}}

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/1031 **Author:** [@wangkuigang-yewu-cmss](https://github.com/wangkuigang-yewu-cmss) **Created:** 4/3/2025 **Status:** ✅ Merged **Merged:** 4/3/2025 **Merged by:** [@qiyuxinlin](https://github.com/qiyuxinlin) **Base:** `main` ← **Head:** `doc-update` --- ### 📝 Commits (1) - [`c590583`](https://github.com/kvcache-ai/ktransformers/commit/c590583262cc586e452176fceac7eca1885e4d43) doc upgrade: model_path requirements and reasoning ### 📊 Changes **1 file changed** (+5 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/balance-serve.md` (+5 -1) </details> ### 📄 Description 在 issue #1018 中，有多个用户遇到了这样的问题：在推理开始时，rpc进程退出，并且在rpc.log中可以看到如下日志： ``` terminate called after throwing an instance of 'std::out_of_range' ``` 经过gdb排查，发现异常发生在: ``` #11 kvc2::CacheInfo::hidden_layer_count (this=this@entry=0x7696ba9ff620) at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:50 ``` 相关代码如下： ```cpp size_t CacheInfo::hidden_layer_count() { return model_configs.at(model_name).num_hidden_layers; } ``` gdb中查看：`model_name` 为用户在`--model_path`参数中指定的路径名的末尾路径，而`model_configs`看起来是`./ktransformers/configs/model_configs.json`中配置的模型名。如果用户使用的路径并非标准模型名，那么这里肯定会崩掉。目前最简单的解决办法，就是在文档中说明一下，让用户明确使用标准的模型名作为路径名称。 --- 此外关于`--force_think`，相信大多数用户本地部署时都会优先考虑R1，并且也会想要R1输出深度思考过程，所以这个选项对于大多数用户都非常有用。建议在文档中也带上这个选项。 --- 附录： <details><summary>gdb backtrace</summary> ``` #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x000076983b64527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x000076983b6288ff in __GI_abort () at ./stdlib/abort.c:79 #5 0x000076983aca5ff5 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x000076983acbb0da in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x000076983aca5a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x000076983acbb391 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x000076983aca932d in std::__throw_out_of_range(char const*) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x0000769716861f1c in std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ModelConfig, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ModelConfig> > >::at ( __k=<error reading variable: Cannot access memory at address 0x6b6565737065654c>, this=0x7697168ff0a0 <model_configs>) at /usr/include/c++/13/bits/stl_map.h:553 #11 kvc2::CacheInfo::hidden_layer_count (this=this@entry=0x7696ba9ff620) at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:50 #12 0x000076971686aa90 in kvc2::DoubleCacheHandle::set_cache_info (this=this@entry=0x769540013770, quant_type="BF16", turn_on_k_cache=turn_on_k_cache@entry=true, turn_on_v_cache=turn_on_v_cache@entry=false, model_name=...) at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:743 #13 0x00007697168c9b16 in kvc2::KVC2::lookup (this=this@entry=0x1ed10500, model_name="deepseek-r1-info", quant_type="BF16", id=id@entry=0x76954000b480, length=<optimized out>, length@entry=5, estimated_length=8192) at /usr/include/c++/13/bits/allocator.h:184 #14 0x00007697168c9339 in kvc2::KVC2::lookup_to_gpu_async(std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned long, unsigned long, std::function<void (std::shared_ptr<kvc2::DoubleCacheHandleInterface>)>) (this=this@entry=0x1ed10500, model_name="deepseek-r1-info", quant_type="BF16", id=id@entry=0x76954000b480, length=length@entry=5, estimated_length=estimated_length@entry=8192, call_back=...) at /home/wkgcass/ktransformers-new/csrc/balance_serve/kvc2/src/prefix.cpp:1279 #15 0x00007697e66a6478 in scheduler::Query::to_status (this=0x769540006320, to=to@entry=scheduler::Query::Preparing) at /usr/include/c++/13/bits/allocator.h:184 #16 0x00007697e66f8a4b in scheduler::FCFS_single_prefill::strategy_prepare (prepare=..., this=0x1ef08990) at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:801 #17 scheduler::FCFS_single_prefill::strategy_prepare (prepare=..., this=0x1ef08990) at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:798 #18 scheduler::QueryMaintainer::tackle_event (event=..., this=0x1ef08990) at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:607 #19 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}::operator()<scheduler::EventPrepare>(scheduler::EventPrepare) const (event=..., __closure=<optimized out>) at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:650 #20 std::__invoke_impl<void, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, scheduler::EventPrepare&>(std::__invoke_other, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, scheduler::EventPrepare&) (__f=...) at /usr/include/c++/13/bits/invoke.h:61 #21 std::__invoke<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, scheduler::EventPrepare&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, scheduler::EventPrepare&) ( __fn=...) at /usr/include/c++/13/bits/invoke.h:96 #22 std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&)>, std::integer_sequence<unsigned long, 3ul> >::__visit_invoke(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) ( __vars#0=std::variant [index 3] = {...}, __visitor=...) at /usr/include/c++/13/variant:1060 #23 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, sched--Type <RET> for more, q to quit, c to continue without paging-- uler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1818 #24 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1756 #25 std::visit<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1878 #26 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const (__closure=<optimized out>) at /home/wkgcass/ktransformers-new/csrc/balance_serve/sched/scheduler.cpp:639 #27 0x000076983a7549c0 in execute_native_thread_routine () from /home/wkgcass/.virtualenvs/ktnew/lib/python3.12/site-packages/torch/lib/libc10.so #28 0x000076983b69caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #29 0x000076983b729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 ``` </details> <details><summary>gdb print</summary> ``` frame 11 print model_name $1 = "deepseek-r1-info" print model_configs $2 = std::map with 12 elements = {["DeepSeek-Coder-V2-Instruct"] = {hidden_size = 5120, intermediate_size = 12288, max_position_embeddings = 163840, model_type = "deepseek_v2", num_attention_heads = 128, num_hidden_layers = 60, num_key_value_heads = 128, vocab_size = 102400}, ["DeepSeek-R1"] = {hidden_size = 7168, intermediate_size = 18432, max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128, num_hidden_layers = 61, num_key_value_heads = 128, vocab_size = 129280}, ["DeepSeek-V2-Lite-Chat"] = {hidden_size = 2048, intermediate_size = 10944, max_position_embeddings = 163840, model_type = "deepseek_v2", num_attention_heads = 16, num_hidden_layers = 27, num_key_value_heads = 16, vocab_size = 102400}, ["DeepSeek-V3"] = {hidden_size = 7168, intermediate_size = 18432, max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128, num_hidden_layers = 3, num_key_value_heads = 128, vocab_size = 129280}, ["DeepSeek-V3-bf16"] = {hidden_size = 7168, intermediate_size = 18432, max_position_embeddings = 163840, model_type = "deepseek_v3", num_attention_heads = 128, num_hidden_layers = 61, num_key_value_heads = 128, vocab_size = 129280}, ["LLaMA-2-7B-32K"] = {hidden_size = 4096, intermediate_size = 11008, max_position_embeddings = 32768, model_type = "llama", num_attention_heads = 32, num_hidden_layers = 32, num_key_value_heads = 32, vocab_size = 32000}, ["Moonlight-16B-A3B-Instruct"] = { hidden_size = 2048, intermediate_size = 11264, max_position_embeddings = 8192, model_type = "deepseek_v3", num_attention_heads = 16, num_hidden_layers = 27, num_key_value_heads = 16, vocab_size = 163840}, ["Qwen2.5-32B-Instruct"] = {hidden_size = 5120, intermediate_size = 27648, max_position_embeddings = 32768, model_type = "qwen2", num_attention_heads = 40, num_hidden_layers = 64, num_key_value_heads = 8, vocab_size = 152064}, ["Qwen2.5-32B-Instruct-GPTQ-Int4"] = {hidden_size = 5120, intermediate_size = 27648, max_position_embeddings = 32768, model_type = "qwen2", num_attention_heads = 40, num_hidden_layers = 64, num_key_value_heads = 8, vocab_size = 152064}, ["Qwen2.5-7B-Instruct"] = {hidden_size = 3584, intermediate_size = 18944, max_position_embeddings = 32768, model_type = "qwen2", num_attention_heads = 28, num_hidden_layers = 28, num_key_value_heads = 4, vocab_size = 152064}, ["Qwen2.5-7B-Instruct-GPTQ-Int4"] = {hidden_size = 3584, intermediate_size = 18944, max_position_embeddings = 32768, model_type = "qwen2", num_attention_heads = 28, num_hidden_layers = 28, num_key_value_heads = 4, vocab_size = 152064}, ["qwen2-72b-instruct"] = {hidden_size = 8192, intermediate_size = 29568, max_position_embeddings = 32768, model_type = "qwen2", num_attention_heads = 64, num_hidden_layers = 80, num_key_value_heads = 8, vocab_size = 152064}} ``` </details> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:53:19 +00:00