theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-29 10:42:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Model: Add proper jobs cleanup and fix var calls

Browse Source 
Jobs should be started and immediately cleaned up when calling the
generation stream. Expose a stream_generate function and append
this to the base class since it's more idiomatic than generate_gen.

The exl2 container's generate_gen function is now internal.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri
2025-04-24 21:30:55 -04:00
parent 7e007f0761
commit f070587e9f
6 changed files with 45 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
endpoints/Kobold/utils/generation.py
View File

@@ -52,7 +52,7 @@ async def _stream_collector(data: GenerateRequest, request: Request):
try:
logger.info(f"Received Kobold generation request {data.genkey}")
generator = model.container.generate_gen(
generator = model.container.stream_generate(
request_id=data.genkey, abort_event=abort_event, **data.model_dump()
)
async for generation in generator: