kingbri e2a8b6e8ae OAI: Add "n" support for streaming generations ...

Use a queue-based system to get choices independently and send them
in the overall streaming payload. This method allows for unordered
streaming of generations.

The system is a bit redundant, so maybe make the code more optimized
in the future.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-05-28 00:52:30 -04:00

9.4 KiB

Raw Blame History

View Raw