kingbri 27ebec3b35 Model: Add speculative decoding support via config ...

Speculative decoding makes use of draft models that ingest the prompt
before forwarding it to the main model.

Add options in the config to support this. API options will occur
in a different commit.

Signed-off-by: kingbri <bdashore3@proton.me>

2023-11-18 01:42:20 -05:00

15 KiB

Raw Blame History

View Raw