mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-03-14 15:57:27 +00:00
ed6c962aad
FastAPI is kinda weird with queueing. If an await is used within an async def, requests aren't executed sequentially. Get the sequential requests back by using a semaphore to limit concurrent execution from generator functions. Also scaffold the framework to move generator functions to their own file. Signed-off-by: kingbri <bdashore3@proton.me>
11 lines
309 B
Python
11 lines
309 B
Python
from asyncio import Semaphore
|
|
from typing import AsyncGenerator
|
|
|
|
generate_semaphore = Semaphore(1)
|
|
|
|
# Async generation that blocks on a semaphore
|
|
async def generate_with_semaphore(generator: AsyncGenerator):
|
|
async with generate_semaphore:
|
|
async for result in generator():
|
|
yield result
|