[GH-ISSUE #23] False-positive "partial" status when OWUI processing is slower than client HTTP timeout #14

New issue

Open

opened 2026-10-18 21:13:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:30 +00:00

Owner

Originally created by @dvystrcil on GitHub (Jun 4, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/23

Summary

When syncing a GitHub source to a large/slow Open WebUI instance, the daemon reports status: "partial" with per-file timed out errors that are actually false positives — the file uploads completed server-side, but oikb's HTTP client gave up waiting before getting the response.

Repro

Deploy oikb daemon v0.3.5 against an Open WebUI instance whose RAG embedding pipeline takes ~10–30 seconds per file (typical for self-hosted embedders on modest hardware)
Configure a GitHub source with ~80 files
Run the initial sync

Observed

GET /health reports (paraphrased):

{
  "name": "wiki-readonly",
  "status": "partial",
  "files_added": 45,
  "errors": [
    "one-ring/entities/bree-land.md: timed out",
    "one-ring/entities/rivendell.md: timed out",
    ...41 entries...
  ]
}

But the Knowledge Base, queried directly via Open WebUI's GET /api/v1/knowledge/{kb_id}/files, shows all 86 files attached. The "timed out" files are in there.

The mismatch is between:

The daemon's files_added counter, incremented only when the HTTP POST returns within the client-side timeout
The actual state of the KB, which gets all the files because Open WebUI completes the embedding work asynchronously regardless of whether the client is still waiting

Why this matters

A partial status is supposed to signal "you need to investigate / retry." But here, no investigation is needed and a retry actively makes things worse — the second sync sees the same files as "needing upload" (because the daemon's bookkeeping doesn't know they landed), re-uploads them, and times out again. We watched a 23-minute initial sync followed by a retry that crashed on {"sync state":"error","added":0,"errs":0} because the recheck hit the same timeout wall.

Suggested fix (one of these)

After a timeout, verify against the KB before counting it as an error. A GET /api/v1/knowledge/{kb_id}/files query after the timeout would let oikb distinguish "client gave up" from "server didn't process."
Configurable HTTP timeout. Currently the timeout looks hardcoded somewhere. A defaults.http-timeout: 5m in .oikb.yaml (or environment variable) would let operators dial this up for slow OWUI installs.
Both. (1) is more correct; (2) is a useful escape hatch.

Environment

oikb v0.3.5 (image ghcr.io/open-webui/oikb:latest digest from 2026-06-04)
Open WebUI v0.9.6 self-hosted
Source: GitHub repo with 86 markdown files, mostly < 5 KB each
Sync configured: concurrency: 4, interval: 1h

Happy to provide more logs / .oikb.yaml details if useful.

Originally created by @dvystrcil on GitHub (Jun 4, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/23 ## Summary When syncing a GitHub source to a large/slow Open WebUI instance, the daemon reports `status: "partial"` with per-file `timed out` errors that are actually false positives — the file uploads completed server-side, but oikb's HTTP client gave up waiting before getting the response. ## Repro 1. Deploy `oikb daemon` v0.3.5 against an Open WebUI instance whose RAG embedding pipeline takes ~10–30 seconds per file (typical for self-hosted embedders on modest hardware) 2. Configure a GitHub source with ~80 files 3. Run the initial sync ## Observed `GET /health` reports (paraphrased): ```json { "name": "wiki-readonly", "status": "partial", "files_added": 45, "errors": [ "one-ring/entities/bree-land.md: timed out", "one-ring/entities/rivendell.md: timed out", ...41 entries... ] } ``` But the Knowledge Base, queried directly via Open WebUI's `GET /api/v1/knowledge/{kb_id}/files`, shows **all 86 files attached**. The "timed out" files are in there. The mismatch is between: - The daemon's `files_added` counter, incremented only when the HTTP POST returns within the client-side timeout - The actual state of the KB, which gets all the files because Open WebUI completes the embedding work asynchronously regardless of whether the client is still waiting ## Why this matters A `partial` status is supposed to signal "you need to investigate / retry." But here, no investigation is needed and a retry actively makes things worse — the second sync sees the same files as "needing upload" (because the daemon's bookkeeping doesn't know they landed), re-uploads them, and times out again. We watched a 23-minute initial sync followed by a retry that crashed on `{"sync state":"error","added":0,"errs":0}` because the recheck hit the same timeout wall. ## Suggested fix (one of these) 1. **After a timeout, verify against the KB before counting it as an error.** A `GET /api/v1/knowledge/{kb_id}/files` query after the timeout would let oikb distinguish "client gave up" from "server didn't process." 2. **Configurable HTTP timeout.** Currently the timeout looks hardcoded somewhere. A `defaults.http-timeout: 5m` in `.oikb.yaml` (or environment variable) would let operators dial this up for slow OWUI installs. 3. **Both.** (1) is more correct; (2) is a useful escape hatch. ## Environment - oikb v0.3.5 (image `ghcr.io/open-webui/oikb:latest` digest from 2026-06-04) - Open WebUI v0.9.6 self-hosted - Source: GitHub repo with 86 markdown files, mostly < 5 KB each - Sync configured: `concurrency: 4`, `interval: 1h` Happy to provide more logs / .oikb.yaml details if useful.

bsakaguchi referenced this issue

2026-10-18 21:14:16 +00:00

[PR #36] [MERGED] fix(connectors/gitlab): resolve subgroup path parsing dynamically #89