jaime-m-p be8232d40f Tokenizer WPM fixes (#7500) ...

* Update random test: add_bos_token.
* Update random test: add WPM models for testing.
* Build vocab.special_tokens_cache using vocab token types.
* Fix and improve WPM preprocessing.
  - Fix unicode edge case combinations.
  - Split by whitspace in the same pass.
* Discard all tokens when no matching found.

2024-05-28 21:46:34 +02:00

12 KiB

Raw Blame History

View Raw