[PR #110] fix: stop re-upload loop for files rejected during server-side ingestion (#106) #119

New issue

Open

opened 2026-10-18 21:14:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/110
Author: @julianlucks
Created: 8/24/2026
Status: 🔄 Open

Base: main ← Head: fix/permanent-upload-failure-skip

📝 Commits (4)

b81b7b4 fix: skip files permanently rejected by the server until they change (#106)
13e0e7c fix: track async uploads, verify status instead of re-uploading (#106)
266af51 docs: changelog + guide entries for permanent-failure skip (#106)
3d03e5b Merge origin/main (v0.5.0) into fix/permanent-upload-failure-skip

📊 Changes

8 files changed (+403 additions, -4 deletions)

View changed files

📝 CHANGELOG.md (+10 -0)
📝 docs/guide.md (+18 -0)
📝 src/oikb/cli.py (+47 -0)
📝 src/oikb/client.py (+26 -0)
📝 src/oikb/daemon.py (+3 -0)
📝 src/oikb/history.py (+158 -1)
📝 src/oikb/kb_sync.py (+3 -0)
📝 src/oikb/sync.py (+138 -3)

📄 Description

Fixes #106.

When Open WebUI rejects a file during ingestion, it is never linked to the KB, so /sync/diff reports it as "added" on every subsequent sync and oikb re-uploads it on every cycle. Every upload creates a new file record and physical copy server-side, which accumulates unboundedly (orphaned records, disk growth).

There are two rejection channels, both handled:

Synchronous: the upload POST itself returns 4xx → recorded immediately in the state DB.
Asynchronous (dominant): uploads return HTTP 200 while ingestion runs in the background; processing errors are swallowed server-side (file status failed, or processed but never linked). Accepted uploads are now tracked (checksum + file id). When the diff reports a tracked file again, oikb checks GET /files/{id}/process/status instead of re-uploading:
- failed → recorded as permanent failure, skipped
- completed but unlinked → the existing file is re-linked in place (no second copy)
- pending/processing → skipped for that run, checked again next run
- gone (404) → untracked and re-uploaded normally

Notes:

Auth failures (401/403) are deliberately never recorded, so config problems keep surfacing as errors on every run instead of being silently skipped.
A changed source checksum is always retried (records are keyed with the checksum).
Successful uploads clear any previous failure record.
Tracked uploads that the diff no longer reports are untracked again, so the table stays small.
New oikb failures command (--kb-id, --clear, --json) to inspect/reset recorded rejections.
Sync results gained skipped_failed / skipped_pending counters (CLI summary + daemon dry-run response).

Test plan

Unit-level: stubbed connector + client covering both channels (sync 4xx, async failed, pending→linked, completed-unlinked→re-link, re-link rejected→permanent)
Verified against a live Open WebUI instance with a real file that always fails ingestion (empty-content rejection): one final upload, status-detected on the next sync, skipped from then on — no further file records created server-side

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/110 **Author:** [@julianlucks](https://github.com/julianlucks) **Created:** 8/24/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/permanent-upload-failure-skip` --- ### 📝 Commits (4) - [`b81b7b4`](https://github.com/open-webui/oikb/commit/b81b7b49ab476b3ef329e8a3a7bc092127fc860c) fix: skip files permanently rejected by the server until they change (#106) - [`13e0e7c`](https://github.com/open-webui/oikb/commit/13e0e7c87193a23b1ee7b85b9b29439000a7986a) fix: track async uploads, verify status instead of re-uploading (#106) - [`266af51`](https://github.com/open-webui/oikb/commit/266af514b747810374c85c82a25a09b361f14aa5) docs: changelog + guide entries for permanent-failure skip (#106) - [`3d03e5b`](https://github.com/open-webui/oikb/commit/3d03e5be8ca5c1ac804da51aacf82323c68ea104) Merge origin/main (v0.5.0) into fix/permanent-upload-failure-skip ### 📊 Changes **8 files changed** (+403 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+10 -0) 📝 `docs/guide.md` (+18 -0) 📝 `src/oikb/cli.py` (+47 -0) 📝 `src/oikb/client.py` (+26 -0) 📝 `src/oikb/daemon.py` (+3 -0) 📝 `src/oikb/history.py` (+158 -1) 📝 `src/oikb/kb_sync.py` (+3 -0) 📝 `src/oikb/sync.py` (+138 -3) </details> ### 📄 Description Fixes #106. When Open WebUI rejects a file during ingestion, it is never linked to the KB, so `/sync/diff` reports it as "added" on every subsequent sync and oikb re-uploads it on every cycle. Every upload creates a new file record and physical copy server-side, which accumulates unboundedly (orphaned records, disk growth). There are two rejection channels, both handled: - **Synchronous:** the upload POST itself returns 4xx → recorded immediately in the state DB. - **Asynchronous (dominant):** uploads return HTTP 200 while ingestion runs in the background; processing errors are swallowed server-side (file status `failed`, or processed but never linked). Accepted uploads are now tracked (checksum + file id). When the diff reports a tracked file again, oikb checks `GET /files/{id}/process/status` instead of re-uploading: - `failed` → recorded as permanent failure, skipped - `completed` but unlinked → the existing file is re-linked in place (no second copy) - `pending`/`processing` → skipped for that run, checked again next run - gone (404) → untracked and re-uploaded normally Notes: - Auth failures (401/403) are deliberately **never** recorded, so config problems keep surfacing as errors on every run instead of being silently skipped. - A changed source checksum is always retried (records are keyed with the checksum). - Successful uploads clear any previous failure record. - Tracked uploads that the diff no longer reports are untracked again, so the table stays small. - New `oikb failures` command (`--kb-id`, `--clear`, `--json`) to inspect/reset recorded rejections. - Sync results gained `skipped_failed` / `skipped_pending` counters (CLI summary + daemon dry-run response). #### Test plan - [x] Unit-level: stubbed connector + client covering both channels (sync 4xx, async `failed`, pending→linked, completed-unlinked→re-link, re-link rejected→permanent) - [x] Verified against a live Open WebUI instance with a real file that always fails ingestion (empty-content rejection): one final upload, status-detected on the next sync, skipped from then on — no further file records created server-side --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-18 21:14:30 +00:00

bsakaguchi referenced this issue

2026-10-18 21:14:31 +00:00

[PR #122] [MERGED] 0.5.0 #121