[GH-ISSUE #115] Files counted as added without verifying they linked, causing re-upload and re-embed every cycle

bsakaguchi commented

2026-10-18 21:14:03 +00:00

Owner

Originally created by @ArthurVardevanyan on GitHub (Sep 3, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/115

Issue

oikb uploads files to Open WebUI with fire-and-forget semantics — it POSTs to /files/ and treats the HTTP 200 response as success, even though the actual KB linkage (chunking + embedding) happens in the background and can fail silently.

Symptom

Files are reported as added every sync cycle, even though they never land in the KB. This causes:

The same files to be re-uploaded and re-embedded every hour
SyncResult reports N added when zero files were actually added
Orphaned file rows, orphaned vector collections, and wasted embedding cost

Root cause

OikbClient.upload_file does not pass process_in_background=false. Open WebUI returns 200 OK immediately, then the background task fails (e.g. 400: Duplicate content detected). Since oikb never verifies whether the file actually linked, it counts the upload as success and the next sync sees the file is still missing → adds it again.

Originally created by @ArthurVardevanyan on GitHub (Sep 3, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/115 ## Issue oikb uploads files to Open WebUI with fire-and-forget semantics — it POSTs to `/files/` and treats the HTTP 200 response as success, even though the actual KB linkage (chunking + embedding) happens in the background and can fail silently. ## Symptom Files are reported as `added` every sync cycle, even though they never land in the KB. This causes: - The same files to be re-uploaded and re-embedded every hour - `SyncResult` reports `N added` when zero files were actually added - Orphaned `file` rows, orphaned vector collections, and wasted embedding cost ## Root cause `OikbClient.upload_file` does not pass `process_in_background=false`. Open WebUI returns 200 OK immediately, then the background task fails (e.g. `400: Duplicate content detected`). Since oikb never verifies whether the file actually linked, it counts the upload as success and the next sync sees the file is still missing → adds it again.

bsakaguchi commented

2026-10-18 21:14:04 +00:00

Author

Owner

@tobru commented on GitHub (Sep 13, 2026):

Confirming this at scale on oikb 0.4.0 + Open WebUI 0.11.3, syncing ~1,380 Nextcloud notes hourly into one KB.

Over ~19 hours the Open WebUI file table grew from 1,125 to 16,874 rows, ~800 new rows per sync, ~80% of all rows in state failed. Every failed row carries the same error:

Duplicate content detected. Please provide unique content to proceed.

The loop is self-sustaining: the re-uploaded file is rejected because its content is already in the KB, so it never links, so the next sync's diff still doesn't see it, so it uploads again. oikb's own counters show the split, the most recent sync reported files_added=796, unmodified=567 against ~1,363 source notes, i.e. dedupe works for ~40% of files and fails permanently for the rest.

Secondary effect worth noting: the Open WebUI container eventually died with OSError: [Errno 24] Too many open files while serving a static asset, and has been restart-looping under the accumulated load.

Cleanup required deleting 15,486 orphaned file rows via the API. With sync stopped the instance is stable.

@tobru commented on GitHub (Sep 13, 2026): Confirming this at scale on oikb 0.4.0 + Open WebUI 0.11.3, syncing ~1,380 Nextcloud notes hourly into one KB. Over ~19 hours the Open WebUI file table grew from 1,125 to 16,874 rows, ~800 new rows per sync, ~80% of all rows in state failed. Every failed row carries the same error: `Duplicate content detected. Please provide unique content to proceed.` The loop is self-sustaining: the re-uploaded file is rejected because its content is already in the KB, so it never links, so the next sync's diff still doesn't see it, so it uploads again. oikb's own counters show the split, the most recent sync reported files_added=796, unmodified=567 against ~1,363 source notes, i.e. dedupe works for ~40% of files and fails permanently for the rest. Secondary effect worth noting: the Open WebUI container eventually died with `OSError: [Errno 24] Too many open files` while serving a static asset, and has been restart-looping under the accumulated load. Cleanup required deleting 15,486 orphaned file rows via the API. With sync stopped the instance is stable.

bsakaguchi commented

2026-10-18 21:14:04 +00:00

Author

Owner

@gbr-lat commented on GitHub (Sep 24, 2026):

Confirming on oikb 0.5.0 + Open WebUI 0.11.4 (Docker), with server-side evidence.

Repro: KB reset, file A synced alone and indexed; then A' added (different bytes,
same extracted text) and oikb sync -> "Sync complete: 1 added, 1 unchanged".
The server logs then show Duplicate content detected, and
Failed to link file <id> to knowledge <kb>: 400.

Server state afterwards (file table): the rejected upload has data.status = failed,
data.error = "Duplicate content detected...", hash = NULL, and is never linked
to the KB. --dry-run keeps listing it under + Added. Running oikb sync once
more created a new failed row for the same file, so one orphan accumulates per
sync (same mechanism as #106).

@gbr-lat commented on GitHub (Sep 24, 2026): Confirming on oikb 0.5.0 + Open WebUI 0.11.4 (Docker), with server-side evidence. Repro: KB reset, file A synced alone and indexed; then A' added (different bytes, same extracted text) and `oikb sync` -> "Sync complete: 1 added, 1 unchanged". The server logs then show `Duplicate content detected`, and `Failed to link file <id> to knowledge <kb>: 400`. Server state afterwards (file table): the rejected upload has `data.status = failed`, `data.error = "Duplicate content detected..."`, `hash = NULL`, and is never linked to the KB. `--dry-run` keeps listing it under `+ Added`. Running `oikb sync` once more created a new `failed` row for the same file, so one orphan accumulates per sync (same mechanism as #106).

bsakaguchi commented

2026-10-18 21:14:04 +00:00

Author

Owner

@julianlucks commented on GitHub (Oct 6, 2026):

This is the same mechanism as #106, and PR #110 addresses it (just brought up to date with v0.5.0).

Rather than switching to process_in_background=false (which makes every upload block on extraction/embedding — a problem with OCR-heavy PDFs and large batches), oikb records accepted uploads with their server file ID in its state DB. When the diff reports a tracked file as added again, it checks GET /files/{id}/process/status instead of re-uploading: failed → recorded as permanent failure and skipped until the source checksum changes; pending/processing → skipped for this run; completed but unlinked → linked via /knowledge/{id}/file/add. Permanently rejected files can be listed and reset via oikb failures.

Running in production since late August; the loop and the orphan growth stopped.

@julianlucks commented on GitHub (Oct 6, 2026): This is the same mechanism as #106, and PR #110 addresses it (just brought up to date with v0.5.0). Rather than switching to `process_in_background=false` (which makes every upload block on extraction/embedding — a problem with OCR-heavy PDFs and large batches), oikb records accepted uploads with their server file ID in its state DB. When the diff reports a tracked file as `added` again, it checks `GET /files/{id}/process/status` instead of re-uploading: `failed` → recorded as permanent failure and skipped until the source checksum changes; `pending`/`processing` → skipped for this run; `completed` but unlinked → linked via `/knowledge/{id}/file/add`. Permanently rejected files can be listed and reset via `oikb failures`. Running in production since late August; the loop and the orphan growth stopped.

bsakaguchi commented

2026-10-18 21:14:04 +00:00

Author

Owner

@Joni1717 commented on GitHub (Oct 8, 2026):

I'm running into the same problem. I synced ~3000 files with oikb into a Knowledge Base (Open WebUI on PostgreSQL, vector store on pgvector). A number of files end up fully processed and embedded (I can open them and see the content), but they are never linked to the knowledge base.

oikb shows no errors and the sync looks successful. But as others have noted, the unlinked files are reported as new on every run, oikb re-uploads them, and Open WebUI rejects them with "400: Duplicate content detected".

I'd really appreciate a fix, and the linked PR looks like it addresses exactly this. Happy to test it against my setup if that helps. Thanks!

@Joni1717 commented on GitHub (Oct 8, 2026): I'm running into the same problem. I synced ~3000 files with oikb into a Knowledge Base (Open WebUI on PostgreSQL, vector store on pgvector). A number of files end up fully processed and embedded (I can open them and see the content), but they are never linked to the knowledge base. oikb shows no errors and the sync looks successful. But as others have noted, the unlinked files are reported as new on every run, oikb re-uploads them, and Open WebUI rejects them with "400: Duplicate content detected". I'd really appreciate a fix, and the linked PR looks like it addresses exactly this. Happy to test it against my setup if that helps. Thanks!

bsakaguchi commented

2026-10-18 21:14:04 +00:00

Author

Owner

@kaiblan commented on GitHub (Oct 17, 2026):

Same here, adding a new sync directory is impossible right now.

@kaiblan commented on GitHub (Oct 17, 2026): Same here, adding a new sync directory is impossible right now.

bsakaguchi referenced this issue

2026-10-18 21:14:20 +00:00

[PR #69] [MERGED] Fix Jira pagination with nextPageToken #97