[PR #81] [CLOSED] fix(history): give each thread its own SQLite connection (fixes #39) #103

New issue

Closed

opened 2026-10-18 21:14:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:22 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/81
Author: @calvinlow18
Created: 7/14/2026
Status: ❌ Closed

Base: main ← Head: fix/sqlite-thread-affinity

📝 Commits (1)

5dde025 fix(history): give each thread its own SQLite connection

📊 Changes

1 file changed (+27 additions, -9 deletions)

View changed files

📝 src/oikb/history.py (+27 -9)

📄 Description

Fixes #39.

The bug

SyncHistory caches a single sqlite3.Connection in self._conn, created lazily by whichever thread calls _get_conn() first. The daemon runs every sync through asyncio.to_thread(), so the next source lands on a different pool thread — and sqlite3 (with its default check_same_thread=True) refuses it:

SQLite objects created in a thread can only be used in that same thread.

The first source to finish wins the connection and succeeds; every other source raises. That matches #39's report exactly (4 of 5 sources failing), and it is connector-agnostic — #39 hit it with Google Drive, I hit it with outline: on 4 sources.

Why it's worse than it looks

SyncHistory.log() is called after run_sync() has already uploaded, and _run_entry_locked() catches the exception in its broad except Exception — which overwrites the just-set success state:

_scheduler_state[source] = { "status": status, "files_added": result.added, ... }  # success
...
await asyncio.to_thread(_history.log, ...)   # <-- raises here
except Exception as e:
    _scheduler_state[source] = {"status": "error", "error": str(e)}   # success is discarded

So the documents do reach the knowledge base, while /health, the dashboard and the history all report error for a sync that actually worked. Anything monitoring the daemon is permanently red, and the counts (files_added, etc.) are lost.

The fix

Hold the connection in a threading.local() so each thread gets its own. Also open with WAL and a busy timeout, so the now-genuinely-concurrent writers wait on the brief write lock instead of raising database is locked.

One file, no API change.

Reproduction

Against main, driving SyncHistory the way the daemon does:

h = SyncHistory(db_path=db)
async def log(i):
    await asyncio.to_thread(h.log, source=f"src{i}", kb_id=f"kb{i}",
                            status="success", started_at=0.0, files_added=1)
await asyncio.gather(*(log(i) for i in range(5)), return_exceptions=True)

Before: 4 of 5 raise ProgrammingError: SQLite objects created in a thread..., and the follow-up query() raises too.
After: sources logged OK: 5/5, rows in history: 5.

Also re-checked the single-threaded CLI path — log, query (plain / errors_only / kb_id), last_sync, clear, and a double close() — all unchanged.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/81 **Author:** [@calvinlow18](https://github.com/calvinlow18) **Created:** 7/14/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/sqlite-thread-affinity` --- ### 📝 Commits (1) - [`5dde025`](https://github.com/open-webui/oikb/commit/5dde025b83aca89d3e66db8fcfd14215642b006f) fix(history): give each thread its own SQLite connection ### 📊 Changes **1 file changed** (+27 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `src/oikb/history.py` (+27 -9) </details> ### 📄 Description Fixes #39. ## The bug `SyncHistory` caches a single `sqlite3.Connection` in `self._conn`, created lazily by whichever thread calls `_get_conn()` first. The daemon runs every sync through `asyncio.to_thread()`, so the next source lands on a *different* pool thread — and `sqlite3` (with its default `check_same_thread=True`) refuses it: ``` SQLite objects created in a thread can only be used in that same thread. ``` The first source to finish wins the connection and succeeds; **every other source raises.** That matches #39's report exactly (4 of 5 sources failing), and it is connector-agnostic — #39 hit it with Google Drive, I hit it with `outline:` on 4 sources. ### Why it's worse than it looks `SyncHistory.log()` is called *after* `run_sync()` has already uploaded, and `_run_entry_locked()` catches the exception in its broad `except Exception` — which overwrites the just-set success state: ```python _scheduler_state[source] = { "status": status, "files_added": result.added, ... } # success ... await asyncio.to_thread(_history.log, ...) # <-- raises here except Exception as e: _scheduler_state[source] = {"status": "error", "error": str(e)} # success is discarded ``` So the documents **do** reach the knowledge base, while `/health`, the dashboard and the history all report `error` for a sync that actually worked. Anything monitoring the daemon is permanently red, and the counts (`files_added`, etc.) are lost. ## The fix Hold the connection in a `threading.local()` so each thread gets its own. Also open with WAL and a busy timeout, so the now-genuinely-concurrent writers wait on the brief write lock instead of raising `database is locked`. One file, no API change. ## Reproduction Against `main`, driving `SyncHistory` the way the daemon does: ```python h = SyncHistory(db_path=db) async def log(i): await asyncio.to_thread(h.log, source=f"src{i}", kb_id=f"kb{i}", status="success", started_at=0.0, files_added=1) await asyncio.gather(*(log(i) for i in range(5)), return_exceptions=True) ``` **Before:** 4 of 5 raise `ProgrammingError: SQLite objects created in a thread...`, and the follow-up `query()` raises too. **After:** `sources logged OK: 5/5`, `rows in history: 5`. Also re-checked the single-threaded CLI path — `log`, `query` (plain / `errors_only` / `kb_id`), `last_sync`, `clear`, and a double `close()` — all unchanged. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:14:22 +00:00