kingbri 660f9b8432 OAI: Fix request cancellation behavior ...

Depending on the day of the week, Starlette can work with a CancelledError
or using await request.is_disconnected(). Run the same behavior for both
cases and allow cancellation.

Streaming requests now set an event to cancel the batched job and break
out of the generation loop.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-05-26 13:00:33 -04:00

..

chat_completion.py

OAI: Fix request cancellation behavior

2024-05-26 13:00:33 -04:00

completion.py

OAI: Fix request cancellation behavior

2024-05-26 13:00:33 -04:00

lora.py

Tree: Switch to async generators

2024-03-16 23:23:31 -04:00

model.py

API + Model: Add blocks and checks for various load requests

2024-05-25 21:16:14 -04:00