kingbri ab10b263fd Model: Add override base seq len ...

Some models (such as mistral and mixtral) set their base sequence
length to 32k due to assumptions of support for sliding window
attention.

Therefore, add this parameter to override the base sequence length
of a model which helps with auto-calculation of rope alpha.

If auto-calculation of rope alpha isn't being used, the max_seq_len
parameter works fine as is.

Signed-off-by: kingbri <bdashore3@proton.me>

2023-12-20 00:45:39 -05:00

..

chat_completion.py

Templating: Add generation prompt appending

2023-12-18 23:53:47 -05:00

common.py

OAI: Add logit bias support

2023-12-18 23:53:47 -05:00

completion.py

Adding token usage support

2023-11-27 20:05:05 -08:00

lora.py

OAI: Add ability to specify fastchat prompt template

2023-12-10 15:43:58 -05:00

model.py

Model: Add override base seq len

2023-12-20 00:45:39 -05:00

token.py

Api: Add ban_eos_token and add_bos_token support

2023-11-15 00:55:15 -05:00