kingbri 26496c4db2 Dependencies: Require tokenizers ...

This is used for some models and isn't too big in size (compared to
other huggingface dependencies), so include it by default.

Signed-off-by: kingbri <bdashore3@proton.me>

2024-03-23 01:12:21 -04:00

9.4 KiB

Raw Blame History

View Raw