open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[PR #61] [CLOSED] feat(zotero): add Zotero connector #95

New issue

Closed

opened 2026-10-18 21:14:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:19 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/61
Author: @thiswillbeyourgithub
Created: 6/22/2026
Status: ❌ Closed

Base: main ← Head: add-zotero

📝 Commits (10+)

ea4a17e feat(zotero): add Zotero connector
03718a1 feat(cli): dispatch zotero: sources
9cf2dc4 build: add zotero optional dependency group
a330a84 docs: document Zotero connector
e709a84 docs: clarify ZOTERO_EXCLUDE paths are relative to synced root
ddec1d8 chore: gitignore perso/ scratch folder
5870306 fix(sync): create directories parent-first by sorting the mkdir list
a793298 fix(zotero): never let an un-fetchable attachment abort the whole sync
9d17fbd feat(sync): treat unavailable source files as warnings, not fatal errors
00b26bf feat(zotero): add ZOTERO_INCLUDE_NOTES / ZOTERO_INCLUDE_ANNOTATIONS

📊 Changes

12 files changed (+1124 additions, -13 deletions)

View changed files

📝 .gitignore (+3 -0)
📝 CHANGELOG.md (+7 -0)
📝 README.md (+45 -1)
📝 pyproject.toml (+3 -0)
📝 src/oikb/cli.py (+25 -0)
📝 src/oikb/connectors/__init__.py (+14 -0)
➕ src/oikb/connectors/zotero.py (+543 -0)
📝 src/oikb/daemon.py (+2 -0)
📝 src/oikb/sync.py (+39 -12)
➕ tests/test_sync_warnings.py (+111 -0)
➕ tests/test_zotero_notes_annotations.py (+199 -0)
➕ tests/test_zotero_unfiled.py (+133 -0)

📄 Description

Summary

Adds a Zotero connector to oikb. It syncs the text of the PDF attachments of items in a Zotero collection (and its subcollections) into an Open WebUI Knowledge Base, mapping the Zotero collection hierarchy onto KB directories. It is read-only with respect to Zotero: it never modifies, deletes, or adds anything to the library.

This is a Claude Code port of an earlier standalone tool I wrote, https://github.com/thiswillbeyourgithub/openwebui-knowledge-zotero-sync, reworked to fit oikb's connector interface.

Zotero connector

New zotero:<hierarchy> source scheme. %% separates collection names
(e.g. zotero:Research%%Machine Learning); a bare zotero: syncs every top-level collection.
Text extraction prefers Zotero's indexed fulltext API, falling back to PyMuPDF on the downloaded PDF.
Collection hierarchy maps to KB directories; an item that lives in several subcollections is handled.
Change detection via ZOTERO_CHECKSUM: version (cheap, hashes the Zotero item version, no download, default) or content (hashes the extracted text).
ZOTERO_EXCLUDE skips subcollections, relative to the synced root.
Auth/options via env: ZOTERO_LIBRARY_ID, ZOTERO_API_KEY, ZOTERO_LIBRARY_TYPE, ZOTERO_CHECKSUM, ZOTERO_EXCLUDE.
Optional dependency group: pip install oikb[zotero] (pyzotero + pymupdf).
README and docs/guide.md updated.

Sync change: unavailable source files are warnings, not fatal errors

A file the source advertises in its manifest but cannot provide content for (e.g. a Zotero attachment whose bytes aren't in storage: a web link, a linked file, or a WebDAV-only item) is a source-side data gap, not an oikb failure. Previously any such case landed in result.errors and exited the whole sync with code 1.

This PR adds a SourceFileUnavailable exception to the connector base. run_sync catches it specifically (no retry) and routes the file to a new result.warnings
list instead of result.errors, so a sync whose only problems are unavailable source files still succeeds (exit 0). Any other read_file() exception still fails the run exactly as before. The CLI and daemon surface warnings separately from errors; only result.errors affects the exit code / success status.

Tests

tests/test_sync_warnings.py covers all three paths: the Zotero missing-file mapping to SourceFileUnavailable, warning routing in run_sync, and that a generic read failure is still an error.

Notes

Opening as a draft for early feedback on the connector shape and on whether the warnings-vs-errors sync change is acceptable as written.
Implemented with Claude Code.
I initially wanted to add support for using only the yaml instead of env variables to simplify setup but it resuired some fundamental changes I felt could go beyond the scope of a single PR. Let me know if you'd like me to streamline this.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/61 **Author:** [@thiswillbeyourgithub](https://github.com/thiswillbeyourgithub) **Created:** 6/22/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `add-zotero` --- ### 📝 Commits (10+) - [`ea4a17e`](https://github.com/open-webui/oikb/commit/ea4a17ef32c919685c3c88a1e28259b187674bbc) feat(zotero): add Zotero connector - [`03718a1`](https://github.com/open-webui/oikb/commit/03718a18d54e0d2a0e2513fec1122a178bfa8b44) feat(cli): dispatch zotero: sources - [`9cf2dc4`](https://github.com/open-webui/oikb/commit/9cf2dc42069b7365884f82cf5cafa771123859f8) build: add zotero optional dependency group - [`a330a84`](https://github.com/open-webui/oikb/commit/a330a84eaf8514fd355de71b23d058cbec3c8e4c) docs: document Zotero connector - [`e709a84`](https://github.com/open-webui/oikb/commit/e709a84eab5ceb389b0379320709a831e68fc0fe) docs: clarify ZOTERO_EXCLUDE paths are relative to synced root - [`ddec1d8`](https://github.com/open-webui/oikb/commit/ddec1d8d472d4ebc40b6f3659e54be221630226f) chore: gitignore perso/ scratch folder - [`5870306`](https://github.com/open-webui/oikb/commit/58703069d119a11ca173939e6695498c5f0703c6) fix(sync): create directories parent-first by sorting the mkdir list - [`a793298`](https://github.com/open-webui/oikb/commit/a7932986281bb2931d02768b04b59239b602b30d) fix(zotero): never let an un-fetchable attachment abort the whole sync - [`9d17fbd`](https://github.com/open-webui/oikb/commit/9d17fbdc0fb7a1ffe85e3a16b1f4836ffb4cab9b) feat(sync): treat unavailable source files as warnings, not fatal errors - [`00b26bf`](https://github.com/open-webui/oikb/commit/00b26bf629f1bbd6e1f748ca76f447e70b08f0bc) feat(zotero): add ZOTERO_INCLUDE_NOTES / ZOTERO_INCLUDE_ANNOTATIONS ### 📊 Changes **12 files changed** (+1124 additions, -13 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+3 -0) 📝 `CHANGELOG.md` (+7 -0) 📝 `README.md` (+45 -1) 📝 `pyproject.toml` (+3 -0) 📝 `src/oikb/cli.py` (+25 -0) 📝 `src/oikb/connectors/__init__.py` (+14 -0) ➕ `src/oikb/connectors/zotero.py` (+543 -0) 📝 `src/oikb/daemon.py` (+2 -0) 📝 `src/oikb/sync.py` (+39 -12) ➕ `tests/test_sync_warnings.py` (+111 -0) ➕ `tests/test_zotero_notes_annotations.py` (+199 -0) ➕ `tests/test_zotero_unfiled.py` (+133 -0) </details> ### 📄 Description ## Summary Adds a **Zotero** connector to oikb. It syncs the text of the PDF attachments of items in a Zotero collection (and its subcollections) into an Open WebUI Knowledge Base, mapping the Zotero collection hierarchy onto KB directories. It is read-only with respect to Zotero: it never modifies, deletes, or adds anything to the library. This is a Claude Code port of an earlier standalone tool I wrote, https://github.com/thiswillbeyourgithub/openwebui-knowledge-zotero-sync, reworked to fit oikb's connector interface. ## Zotero connector - New `zotero:<hierarchy>` source scheme. `%%` separates collection names (e.g. `zotero:Research%%Machine Learning`); a bare `zotero:` syncs every top-level collection. - Text extraction prefers Zotero's indexed fulltext API, falling back to PyMuPDF on the downloaded PDF. - Collection hierarchy maps to KB directories; an item that lives in several subcollections is handled. - Change detection via `ZOTERO_CHECKSUM`: `version` (cheap, hashes the Zotero item version, no download, default) or `content` (hashes the extracted text). - `ZOTERO_EXCLUDE` skips subcollections, relative to the synced root. - Auth/options via env: `ZOTERO_LIBRARY_ID`, `ZOTERO_API_KEY`, `ZOTERO_LIBRARY_TYPE`, `ZOTERO_CHECKSUM`, `ZOTERO_EXCLUDE`. - Optional dependency group: `pip install oikb[zotero]` (pyzotero + pymupdf). - README and docs/guide.md updated. ## Sync change: unavailable source files are warnings, not fatal errors A file the source advertises in its manifest but cannot provide content for (e.g. a Zotero attachment whose bytes aren't in storage: a web link, a linked file, or a WebDAV-only item) is a source-side data gap, not an oikb failure. Previously any such case landed in `result.errors` and exited the whole sync with code 1. This PR adds a `SourceFileUnavailable` exception to the connector base. `run_sync` catches it specifically (no retry) and routes the file to a new `result.warnings` list instead of `result.errors`, so a sync whose only problems are unavailable source files still succeeds (exit 0). Any other `read_file()` exception still fails the run exactly as before. The CLI and daemon surface warnings separately from errors; only `result.errors` affects the exit code / success status. ## Tests `tests/test_sync_warnings.py` covers all three paths: the Zotero missing-file mapping to `SourceFileUnavailable`, warning routing in `run_sync`, and that a generic read failure is still an error. ## Notes - Opening as a draft for early feedback on the connector shape and on whether the warnings-vs-errors sync change is acceptable as written. - Implemented with Claude Code. - I initially wanted to add support for using only the yaml instead of env variables to simplify setup but it resuired some fundamental changes I felt could go beyond the scope of a single PR. Let me know if you'd like me to streamline this. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:14:19 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#95

No description provided.

Rows
Columns