turboderp 79d581e1f5 OAI endpoints: More rework ...

- remove disconnect_task
- move disconnect logic to a per-request handler that wraps cleanup operation and directly polls the request state with throttling
- exclusively signal disconnect with CancelledError
- rework completions endpoint to follow same approach as chat completions, share some code
- refactor OAI endpoints a bit
- correct behavior for batched completion requests
- make sure logprobs work for completion and streaming completion requests
- more tests

2026-04-02 01:26:44 +02:00

8.4 KiB

Raw Permalink Blame History

View Raw