mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-03-14 15:57:27 +00:00
ae69b18583
sse_starlette kept firing a ping response if it was taking too long to set an event. Rather than using a hacky workaround, switch to FastAPI's inbuilt streaming response and construct SSE requests with a utility function. This helps the API become more robust and removes an extra requirement. Signed-off-by: kingbri <bdashore3@proton.me>
26 lines
2.2 KiB
Plaintext
26 lines
2.2 KiB
Plaintext
fastapi
|
|
pydantic < 2,>= 1
|
|
PyYAML
|
|
progress
|
|
uvicorn
|
|
|
|
# Wheels
|
|
|
|
# Flash Attention 2. If the wheels don't work, comment these out, uncomment the old wheels, and run `pip install -r requirements.txt`
|
|
|
|
# Windows FA2 from https://github.com/jllllll/flash-attention/releases
|
|
https://github.com/jllllll/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu121torch2.1cxx11abiFALSE-cp311-cp311-win_amd64.whl; platform_system == "Windows" and python_version == "3.11"
|
|
https://github.com/jllllll/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu121torch2.1cxx11abiFALSE-cp310-cp310-win_amd64.whl; platform_system == "Windows" and python_version == "3.10"
|
|
|
|
# Linux FA2 from https://github.com/Dao-AILab/flash-attention/releases
|
|
https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu122torch2.1cxx11abiFALSE-cp311-cp311-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.11"
|
|
https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.10"
|
|
|
|
# Old flash attention 2 wheels. Uncomment these and if the new ones don't work!
|
|
# IMPORTANT: Make sure to re-comment the new wheels before re-running `pip install -r requirements.txt`
|
|
|
|
# https://github.com/bdashore3/flash-attention/releases/download/2.3.3/flash_attn-2.3.3+cu122-cp311-cp311-win_amd64.whl; platform_system == "Windows" and python_version == "3.11"
|
|
# https://github.com/bdashore3/flash-attention/releases/download/2.3.3/flash_attn-2.3.3+cu122-cp310-cp310-win_amd64.whl; platform_system == "Windows" and python_version == "3.10"
|
|
# https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.3/flash_attn-2.3.3+cu122torch2.1cxx11abiFALSE-cp311-cp311-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.11"
|
|
# https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.3/flash_attn-2.3.3+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.10"
|