[PR #1400] [MERGED] docs: name the transformers token splitter on the RAG page and tie the tokenizer model to it #1365

New issue

Closed

opened 2026-10-18 14:19:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 14:19:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/docs/pull/1400
Author: @silentoplayz
Created: 9/18/2026
Status: ✅ Merged
Merged: 9/18/2026
Merged by: @silentoplayz

Base: main ← Head: docs/rag-splitter-options

📝 Commits (1)

178c6a7 docs: name the transformers token splitter on the RAG page and tie the tokenizer model to it

📊 Changes

1 file changed (+1 additions, -1 deletions)

View changed files

📝 docs/features/chat-conversations/rag/index.md (+1 -1)

📄 Description

Summary

The RAG page's Text Splitter bullet lists character and token and says RAG_TOKENIZER_MODEL switches the token splitter to a HuggingFace tokenizer. In routers/retrieval.py the token splitter is Tiktoken only, and the tokenizer model is read by a third splitter, token_transformers, which the admin Documents settings also offer and which the slim image refuses. The bullet now names all three and attaches the tokenizer model to the right one.

The matching reference-page entries are corrected in #1395.

None.

Checklist

I have reviewed the relevant documentation and matched the existing style.
This PR meets Open WebUI's contribution standards: it is accurate, relevant to users, narrowly scoped, maintainable, and not promotional content, advertising, lead generation, SEO placement, or a request to list a product, service, provider, integration, gateway, tool, or company primarily for visibility.
I understand that PRs that do not meet these standards may be closed without review and will not be merged. Repeated, low-quality, off-topic, promotional, or intentionally misleading submissions may result in the contributor being blocked from future participation in Open WebUI repositories.

Notes for reviewers

Source on dev: the splitter branches in routers/retrieval.py (token builds a Tiktoken TokenTextSplitter, token_transformers uses get_transformers_tokenizer, which reads RAG_TOKENIZER_MODEL and raises 503 on slim), and the three <option> values in Documents.svelte.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/docs/pull/1400 **Author:** [@silentoplayz](https://github.com/silentoplayz) **Created:** 9/18/2026 **Status:** ✅ Merged **Merged:** 9/18/2026 **Merged by:** [@silentoplayz](https://github.com/silentoplayz) **Base:** `main` ← **Head:** `docs/rag-splitter-options` --- ### 📝 Commits (1) - [`178c6a7`](https://github.com/open-webui/docs/commit/178c6a7fe62ee79e92e46ea2dfe4b1a1ad2ae618) docs: name the transformers token splitter on the RAG page and tie the tokenizer model to it ### 📊 Changes **1 file changed** (+1 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `docs/features/chat-conversations/rag/index.md` (+1 -1) </details> ### 📄 Description ## Summary The RAG page's Text Splitter bullet lists `character` and `token` and says `RAG_TOKENIZER_MODEL` switches the `token` splitter to a HuggingFace tokenizer. In `routers/retrieval.py` the `token` splitter is Tiktoken only, and the tokenizer model is read by a third splitter, `token_transformers`, which the admin Documents settings also offer and which the slim image refuses. The bullet now names all three and attaches the tokenizer model to the right one. The matching reference-page entries are corrected in #1395. ## Related issue or discussion None. ## Checklist - [x] I have reviewed the relevant documentation and matched the existing style. - [x] This PR meets Open WebUI's contribution standards: it is accurate, relevant to users, narrowly scoped, maintainable, and not promotional content, advertising, lead generation, SEO placement, or a request to list a product, service, provider, integration, gateway, tool, or company primarily for visibility. - [x] I understand that PRs that do not meet these standards may be closed without review and will not be merged. Repeated, low-quality, off-topic, promotional, or intentionally misleading submissions may result in the contributor being blocked from future participation in Open WebUI repositories. ## Notes for reviewers Source on `dev`: the splitter branches in `routers/retrieval.py` (`token` builds a Tiktoken `TokenTextSplitter`, `token_transformers` uses `get_transformers_tokenizer`, which reads `RAG_TOKENIZER_MODEL` and raises 503 on slim), and the three `<option>` values in `Documents.svelte`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 14:19:10 +00:00