[PR #1132] [MERGED] 使用长prompt时，避免rpc进程挂掉 #1565

New issue

Closed

opened 2026-10-29 14:53:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:53:32 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/1132
Author: @wangkuigang-yewu-cmss
Created: 4/13/2025
Status: ✅ Merged
Merged: 4/13/2025
Merged by: @qiyuxinlin

Base: main ← Head: long-prompt-crash

📝 Commits (1)

4538bda prevent rpc process from crashing on long prompt

📊 Changes

1 file changed (+4 additions, -0 deletions)

View changed files

📝 ktransformers/server/backend/interfaces/balance_serve.py (+4 -0)

📄 Description

当prompt超过cache_len的时候，rpc进程会crash掉，导致整体不可用。
这里增加一个检查，让过长的prompt在请求早期就被提前过滤掉。

在使用超过cache_len的长prompt时，rpc进程会在这里挂掉：

gdb输出

Thread 2 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x75f036a006c0 (LWP 152917)]
__memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660
warning: 660	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory
(gdb) bt
#0  __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660
#1  0x000075f1b8436b91 in memcpy (__len=204028, __src=0x75f0140337c0, __dest=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#2  scheduler::Query::Query (this=0x75f0140336f0, id=<optimized out>, query_add=..., context=...)
    at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:163
#3  0x000075f1b84378a6 in scheduler::QueryMaintainer::tackle_event (this=this@entry=0x26155850, event={...})
    at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:567
#4  0x000075f1b843979c in scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}::operator()<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*> >(std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>) const (event={...}, __closure=<optimized out>) at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:640
#5  std::__invoke_impl<void, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&>(std::__invoke_other, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&) (__f=...)
    at /usr/include/c++/13/bits/invoke.h:61
#6  std::__invoke<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&) (__fn=...) at /usr/include/c++/13/bits/invoke.h:96
#7  std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&)>, std::integer_sequence<unsigned long, 0ul> >::__visit_invoke(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (
    __vars#0=std::variant [index 0] containing None = {...}, __visitor=...) at /usr/include/c++/13/variant:1060
#8  std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...)
    at /usr/include/c++/13/variant:1815
#9  std::visit<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1878
#10 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const (__closure=<optimized out>)
    at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:639
#11 0x000075f1bd0989c0 in execute_native_thread_routine ()
   from /home/wkgcass/.virtualenvs/ktnew-node-1/lib/python3.12/site-packages/torch/lib/libc10.so
#12 0x000075f1be09caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#13 0x000075f1be129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb)

挂掉位置的代码，以及相关变量如截图所示：

用torch申请的内存使用estimated_length，而实际prompt为query_length，比申请到的更大，所以会直接挂掉。

这个提交做了一个简单的规避。抛出异常后，连接会被立即关闭（具体行为是，响应了200+headers后，连接被立即关闭）
并且由于抛出异常的位置query还没开始处理，所以不会影响后续请求。

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/1132 **Author:** [@wangkuigang-yewu-cmss](https://github.com/wangkuigang-yewu-cmss) **Created:** 4/13/2025 **Status:** ✅ Merged **Merged:** 4/13/2025 **Merged by:** [@qiyuxinlin](https://github.com/qiyuxinlin) **Base:** `main` ← **Head:** `long-prompt-crash` --- ### 📝 Commits (1) - [`4538bda`](https://github.com/kvcache-ai/ktransformers/commit/4538bdae977a83f29c0c16d3e4b372efe0cdbb96) prevent rpc process from crashing on long prompt ### 📊 Changes **1 file changed** (+4 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `ktransformers/server/backend/interfaces/balance_serve.py` (+4 -0) </details> ### 📄 Description 当prompt超过cache_len的时候，rpc进程会crash掉，导致整体不可用。这里增加一个检查，让过长的prompt在请求早期就被提前过滤掉。 --- 在使用超过cache_len的长prompt时，rpc进程会在这里挂掉： <details><summary>gdb输出</summary> <img width="898" alt="Xnip Helper 2025-04-13 15 41 46" src="https://github.com/user-attachments/assets/56348a28-77bd-43f7-90db-4e1097823e9f" /> ``` Thread 2 "python3" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x75f036a006c0 (LWP 152917)] __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660 warning: 660 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory (gdb) bt #0 __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660 #1 0x000075f1b8436b91 in memcpy (__len=204028, __src=0x75f0140337c0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29 #2 scheduler::Query::Query (this=0x75f0140336f0, id=<optimized out>, query_add=..., context=...) at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:163 #3 0x000075f1b84378a6 in scheduler::QueryMaintainer::tackle_event (this=this@entry=0x26155850, event={...}) at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:567 #4 0x000075f1b843979c in scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}::operator()<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*> >(std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>) const (event={...}, __closure=<optimized out>) at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:640 #5 std::__invoke_impl<void, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&>(std::__invoke_other, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&) (__f=...) at /usr/include/c++/13/bits/invoke.h:61 #6 std::__invoke<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>&) (__fn=...) at /usr/include/c++/13/bits/invoke.h:96 #7 std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&)>, std::integer_sequence<unsigned long, 0ul> >::__visit_invoke(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) ( __vars#0=std::variant [index 0] containing None = {...}, __visitor=...) at /usr/include/c++/13/variant:1060 #8 std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1815 #9 std::visit<scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&>(scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const::{lambda(auto:1)#1}&&, std::variant<std::pair<scheduler::QueryAdd, std::promise<unsigned long>*>, std::vector<scheduler::QueryUpdate, std::allocator<scheduler::QueryUpdate> >, std::shared_ptr<scheduler::BatchQueryTodo>, scheduler::EventPrepare, scheduler::EventPrepared, scheduler::EventQueryStatus, scheduler::EventSchedule>&) (__visitor=...) at /usr/include/c++/13/variant:1878 #10 scheduler::QueryMaintainer::run()::{lambda()#1}::operator()() const (__closure=<optimized out>) at /home/wkgcass/ktransformers-node-1/csrc/balance_serve/sched/scheduler.cpp:639 #11 0x000075f1bd0989c0 in execute_native_thread_routine () from /home/wkgcass/.virtualenvs/ktnew-node-1/lib/python3.12/site-packages/torch/lib/libc10.so #12 0x000075f1be09caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #13 0x000075f1be129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 (gdb) ``` </details> 挂掉位置的代码，以及相关变量如截图所示： <img width="730" alt="Xnip Helper 2025-04-13 15 50 01" src="https://github.com/user-attachments/assets/d67aea98-0919-4632-bbff-bc584a5e836e" /> <img width="359" alt="Xnip Helper 2025-04-13 15 49 32" src="https://github.com/user-attachments/assets/59354a7a-c096-4828-985f-65efbc792234" /> 用torch申请的内存使用`estimated_length`，而实际prompt为`query_length`，比申请到的更大，所以会直接挂掉。这个提交做了一个简单的规避。抛出异常后，连接会被立即关闭（具体行为是，响应了200+headers后，连接被立即关闭）并且由于抛出异常的位置query还没开始处理，所以不会影响后续请求。 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:53:32 +00:00