mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-07-17 00:57:53 +00:00
e0f24264bb
* Fix ValidationError when a client disconnects during prefill
_chat_stream_collector initialized `generation = {}` and only set the
"index" key inside the `async for` loop body. If the client disconnects
before the first token is produced (still in prefill), the loop never
runs, the empty dict is returned, and _compose_response constructs
ChatCompletionRespChoice(index=None), raising a pydantic ValidationError.
Initialize `generation = {"index": task_idx}` so a valid index is always
present even when no tokens were generated.
* Guard against division by zero in draft accept-rate logging
When draft_accept and draft_reject are both 0 (a request that produced
no draft tokens), total_draft is 0 and `accept / total_draft` raises
ZeroDivisionError inside log_metrics. Guard the division and report
0.0% in that case.