kingbri beb6d8faa5 Model: Adjust draft_gpu_split and add to config ...

The previous code overrode the existing gpu split and device idx
values. This now sets an independent draft_gpu_split value and
adjusts the gpu_devices check only if the draft_gpu_split array
is larger than the gpu_split array.

Draft gpu split is not Tensor Parallel, and defaults to gpu_split_auto
if a split is not provided.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>

2025-02-08 16:09:46 -05:00

..

grammar.py

Grammar: Cache the engine vocabulary

2024-12-05 21:36:37 -08:00

model.py

Model: Adjust draft_gpu_split and add to config

2025-02-08 16:09:46 -05:00

utils.py

fix issues with optional dependencies (#204)

2024-09-19 22:24:55 -04:00

version.py

Dependencies: Update ExllamaV2

2024-09-30 00:17:12 -04:00

vision.py

Dependencies: Fix OpenAPI generation

2024-11-22 17:59:20 -05:00