ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-02-24 23:24:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
42adfb7c7c86b48b4008d94a8e9a975d8ccafe6e
ik_llama.cpp/models
History
Galunid 7b34c1046c Add more tokenizer tests (#3742) 
* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test
2023-10-24 09:17:17 +02:00
..
.editorconfig
gguf : new file format with flexible meta data (beta) (#2398)
2023-08-21 23:07:43 +03:00
ggml-vocab-aquila.gguf
Work on the BPE tokenizer (#3252)
2023-10-03 09:16:26 +02:00
ggml-vocab-baichuan.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00
ggml-vocab-falcon.gguf
Work on the BPE tokenizer (#3252)
2023-10-03 09:16:26 +02:00
ggml-vocab-gpt-neox.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00
ggml-vocab-llama.gguf
gguf : new file format with flexible meta data (beta) (#2398)
2023-08-21 23:07:43 +03:00
ggml-vocab-mpt.gguf
Add test for MPT tokenization (#3728)
2023-10-22 21:21:42 +02:00
ggml-vocab-refact.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00
ggml-vocab-starcoder.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00