kingbri 46304ce875 Model: Properly pass in max_batch_size from config ...

The override wasn't being passed in before. Also, the default is now
none since Exl2 can automatically calculate the max batch size.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-07-30 18:42:25 -04:00

..

grammar.py

Model: Grammar: Fix lru_cache clear function

2024-07-08 15:10:15 -04:00

model.py

Model: Properly pass in max_batch_size from config

2024-07-30 18:42:25 -04:00

utils.py

Dependencies: Bump ExllamaV2

2024-07-11 14:22:50 -04:00