kingbri 2f568ff573 Config: Expose auto GPU split reserve config ...

The GPU reserve is used as a VRAM buffer to prevent GPU overflow
when automatically deciding how to load a model on multiple GPUs.
Make this configurable.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-02-08 22:09:50 -05:00

..

model.py

Config: Expose auto GPU split reserve config

2024-02-08 22:09:50 -05:00

utils.py

Requirements: Update exllamav2, torch, and FA2

2024-02-02 23:53:42 -05:00