kingbri 871c89063d Model: Add Tensor Parallel support ...

Use the tensor parallel loader when the flag is enabled. The new loader
has its own autosplit implementation, so gpu_split_auto isn't valid
here.

Also make it easier to determine which cache type to use rather than
multiple if/else statements.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-08-22 14:15:19 -04:00

..

args.py

Model: Add Tensor Parallel support

2024-08-22 14:15:19 -04:00

auth.py

Auth: Fix disable auth when checking for key permissions

2024-07-26 15:04:29 -04:00

concurrency.py

API + Model: Add blocks and checks for various load requests

2024-05-25 21:16:14 -04:00

config.py

Embeddings: Update config, args, and parameter names

2024-07-30 15:32:26 -04:00

downloader.py

Downloader: Make timeout configurable

2024-07-23 21:42:38 -04:00

gen_logging.py

Model: Attach request ID to logs

2024-08-01 00:25:54 -04:00

logger.py

API: Add HuggingFace downloader

2024-04-29 01:15:02 -04:00

model.py

Model: Bypass lock checks when shutting down

2024-08-03 16:05:34 -04:00

networking.py

API: Add request logging

2024-07-22 21:40:00 -04:00

sampling.py

[WIP] OpenAI Tools Support/Function calling (#154)

2024-08-17 00:16:25 -04:00

signals.py

Model: Bypass lock checks when shutting down

2024-08-03 16:05:34 -04:00

templating.py

Templates: Switch to async jinja engine

2024-08-17 12:03:41 -04:00

transformers_utils.py

Tree: Format

2024-07-26 18:33:04 -04:00

utils.py

Model: Add support for HuggingFace config and bad_words_ids

2024-07-26 18:23:22 -04:00