SillyTavern/SillyTavern-Tokenizers
1
0
Fork 0
mirror of https://github.com/SillyTavern/SillyTavern-Tokenizers.git synced 2026-10-15 18:12:02 +00:00
Code Issues Projects Packages Wiki Activity

[PR #2] [MERGED] Update tokenizers #2

New issue
Closed
opened 2026-10-15 18:07:51 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-15 18:07:51 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/SillyTavern/SillyTavern-Tokenizers/pull/2
Author: @equal-l2
Created: 7/31/2025
Status: ✅ Merged
Merged: 8/1/2025
Merged by: @Cohee1207

Base: main ← Head: update-minify

📝 Commits (3)

  • dc45817 Add JSON minifier script
  • dc18053 Add & minify Cohere tokenizers
  • bb559ca Minify nemo.json and qwen2.json

📊 Changes

6 files changed (+22 additions, -1221507 deletions)

View changed files

➕ command-a-vision.json (+1 -0)
📝 command-a.json (+1 -1)
📝 command-r.json (+1 -508770)
➕ minifier.py (+17 -0)
📝 nemo.json (+1 -409625)
📝 qwen2.json (+1 -303111)

📄 Description

  • Added a tokenizer: command-a-vision.json
  • Minified preexisting tokenizers: command-a.json, nemo.json, qwen2.json

(deepseek.json is already minified)

Note:

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/SillyTavern/SillyTavern-Tokenizers/pull/2 **Author:** [@equal-l2](https://github.com/equal-l2) **Created:** 7/31/2025 **Status:** ✅ Merged **Merged:** 8/1/2025 **Merged by:** [@Cohee1207](https://github.com/Cohee1207) **Base:** `main` ← **Head:** `update-minify` --- ### 📝 Commits (3) - [`dc45817`](https://github.com/SillyTavern/SillyTavern-Tokenizers/commit/dc458176d426801c4010981c573773d7854a54ed) Add JSON minifier script - [`dc18053`](https://github.com/SillyTavern/SillyTavern-Tokenizers/commit/dc180539116f64493ae04f8b23441c70e302775d) Add & minify Cohere tokenizers - [`bb559ca`](https://github.com/SillyTavern/SillyTavern-Tokenizers/commit/bb559cae4d55a508e83366e2334963e966c057fa) Minify nemo.json and qwen2.json ### 📊 Changes **6 files changed** (+22 additions, -1221507 deletions) <details> <summary>View changed files</summary> ➕ `command-a-vision.json` (+1 -0) 📝 `command-a.json` (+1 -1) 📝 `command-r.json` (+1 -508770) ➕ `minifier.py` (+17 -0) 📝 `nemo.json` (+1 -409625) 📝 `qwen2.json` (+1 -303111) </details> ### 📄 Description - Added a tokenizer: command-a-vision.json - Minified preexisting tokenizers: command-a.json, nemo.json, qwen2.json (deepseek.json is already minified) Note: - The minifying script is based on https://github.com/SillyTavern/SillyTavern-Tokenizers/pull/1#issuecomment-2722477112 - I don't add tokenizers for r7b and aya variants for now, as I don't use them personally. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-15 18:07:51 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
SillyTavern/SillyTavern-Tokenizers#2
No description provided.