dungquixote42 869b83bc49 Add Unicode allowlist (#1597) ...

* initial commit

* cleanup

* fix whitelist arg parsing and simplify keyword search state

* rename white* to allow*

* add vocab_pieces init function, rename update functions, delete accidentally added file

* delete temporary bias code

* auto-generate fill function with script data inside

* deduplicate allowlist unicode rule parsing

* minor cleanup

* delete unnecessary header

* refactor allowlist to support sequential rule sets via keywords

* add early exit for zero-rules case

* delete accidentally added file

2026-04-10 18:22:57 +02:00

2.6 KiB

Raw Blame History

View Raw