open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[GH-ISSUE #106] Bug : Failed Open WebUI ingestion causes repeated uploads and disk exhaustion during Google Drive sync #65

New issue

Open

opened 2026-10-18 21:14:01 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-18 21:14:01 +00:00

Owner

Originally created by @ShraddhanandKhot on GitHub (Aug 13, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/106

Bug Summary

When using oikb with the Google Drive connector to sync files into an Open WebUI Knowledge Base, a file that fails during Open WebUI ingestion can be uploaded repeatedly on every sync cycle.

The source file remains unchanged in Google Drive, but because the corresponding Open WebUI file is not successfully attached/processed in the Knowledge Base, the next sync can incorrectly treat the same source file as new and upload it again.

In my case, this resulted in 1,300+ copies of the same file, several GB of unnecessary storage usage, and eventually an Open WebUI backend crash loop caused by repeated processing/embedding attempts.

Environment

oikb: Google Drive connector
Open WebUI: 0.10.x
Sync interval: 1 minute
Deployment: Docker
Vector database: ChromaDB
Source: Google Drive
Files involved: PDF/DOCX files, including scanned/corrupt PDFs and documents with no extractable text

What happened

The problem occurs when a source file is downloaded from Google Drive and uploaded to Open WebUI, but Open WebUI fails while extracting/indexing/embedding the file.

For example:

Google Drive
    │
    │ file unchanged
    ▼
oikb detects file
    │
    ▼
Upload to Open WebUI
    │
    ▼
Open WebUI extraction / embedding
    │
    ├── extraction fails
    │   or
    ├── content is empty
    │   or
    └── embedding/indexing fails
    │
    ▼
File is not successfully linked/processed in the KB
    │
    ▼
Next oikb sync
    │
    ▼
Same source file is detected as "new"
    │
    ▼
Upload again
    │
    ▼
Repeat every sync cycle

With a one-minute sync interval, this can quickly create hundreds or thousands of duplicate Open WebUI file records.

Observed impact

One problematic file named:

AUDIT NOTE_Contingent Liabilities...pdf

was uploaded 1,314 times.

This caused:

thousands of duplicate file records
approximately 3.8 GB of files for the duplicated document
disk usage reaching 95–100%
No space left on device errors
repeated extraction/embedding attempts
ChromaDB corruption
Open WebUI backend restart/crash loops

In one incident, the Open WebUI backend was restarting approximately every 61 seconds, with more than 76 restarts observed.

The ChromaDB database eventually had to be rebuilt manually.

Expected behavior

If a file has already been uploaded to Open WebUI but its processing fails, oikb should not continuously re-upload the same unchanged source file on every sync cycle.

A failed ingestion should result in one of the following behaviors:

Mark the source file as failed and stop retrying until the source changes.
Record the failure and apply a retry/backoff policy.
Detect that the same source file/content already exists in Open WebUI and skip the upload.
Surface the failure clearly to the user/operator.
Optionally provide a mechanism to retry failed files manually.

Most importantly:

A failed ingestion should not result in an unlimited upload loop for an unchanged source file.

Additional issue: failed files accumulate

There is also a secondary cleanup problem.

Failed Open WebUI file records remain on disk even when they are no longer successfully linked to a Knowledge Base.

In my environment, there were approximately:

148 file records
40 files in failed status
approximately 79 files not linked to a Knowledge Base

These failed/orphaned files continue consuming storage.

A mechanism for cleaning up failed/orphaned files, or at least preventing them from accumulating indefinitely, would help prevent gradual disk exhaustion.

Why this is particularly dangerous

The issue is not limited to duplicate database rows.

Every repeated upload can trigger Open WebUI's extraction and embedding pipeline again.

For a problematic document, this means:

1 bad file
   ↓
1 failed extraction
   ↓
1 failed embedding attempt
   ↓
next sync
   ↓
same file uploaded again
   ↓
another extraction
   ↓
another embedding
   ↓
...
   ↓
hundreds/thousands of attempts

This can consume:

disk space
CPU
RAM
vector database storage
embedding resources

and potentially destabilize or crash the Open WebUI backend.

Reproduction

A simplified reproduction would be:

Configure oikb with a Google Drive source and a short sync interval.
Create a Knowledge Base in Open WebUI.
Place a problematic document in the Google Drive folder.
- Example: a corrupt PDF or scanned PDF with no extractable text.
Allow oikb to synchronize the file.
Allow Open WebUI's extraction/embedding process to fail.
Do not modify the source file.
Wait for the next oikb sync cycle.
Observe that the same source file is uploaded again.
Repeat for several sync cycles.
Observe multiple Open WebUI file records for the same source document.

Possible root cause

The failure appears to be caused by a mismatch between oikb's source-side diff state and Open WebUI's successful file/Knowledge Base state.

The source file remains unchanged, but the failed Open WebUI ingestion does not leave the system in a state where the next synchronization can reliably identify the source file as already handled.

This creates a loop:

Source unchanged
      +
Open WebUI ingestion failed
      +
File not recognized as successfully synchronized
      =
Upload again

There may also be an interaction with folder-less/root-level files and Open WebUI's file matching/diff behavior.

I would appreciate confirmation of the exact root cause from the maintainers.

Mitigations currently used

I implemented two temporary mitigations in my environment:

1. Open WebUI-side duplicate detection

I added a fallback that uses content hashing to recognize files that already exist even when the normal folder/filename matching does not identify them correctly.

2. oikb client-side duplicate guard

Before uploading, my oikb client checks the existing Open WebUI files and skips a file when the same filename + content hash already exists.

This currently prevents repeated uploads and the sync reports unchanged files instead of uploading them again.

However, these are workarounds rather than a complete solution because a new problematic file can still be uploaded and fail on its first ingestion attempt.

Suggested improvements

Potential improvements could include:

A. Persist failed ingestion state

Track the source file's content hash and ingestion status:

source hash
    ↓
Open WebUI ingestion
    ↓
FAILED
    ↓
remember failure

On the next sync:

same source hash?
    │
    ├── YES + previously failed
    │       ↓
    │   skip/retry according to policy
    │
    └── NO
            ↓
        upload/process

B. Retry with backoff

Instead of retrying every minute:

1 minute
1 minute
1 minute
1 minute
...

use something such as:

1st failure → retry
2nd failure → wait longer
3rd failure → longer backoff
4th failure → mark as failed

C. Avoid re-uploading known failed content

Use a stable source identifier/content hash so that an unchanged failed document is not uploaded repeatedly.

D. Cleanup failed/orphaned files

Provide a safe mechanism to identify and remove files that:

are permanently failed
are not linked to any Knowledge Base
have not changed since the last failed attempt

E. Alerting

Expose sync/ingestion failures through the existing notification mechanism so operators know that a document failed instead of discovering the problem only after the server becomes unavailable.

Additional context

The official oikb documentation describes the project as an incremental synchronization system that should upload only new and modified files.

The current documentation also notes that indexing happens server-side in Open WebUI after oikb uploads a file.

That separation is important here: oikb can successfully upload a file while Open WebUI subsequently fails during extraction or embedding. The synchronization layer therefore needs a reliable way to distinguish:

uploaded successfully

from:

successfully ingested into the Knowledge Base

Logs / evidence

I can provide:

oikb sync logs
Open WebUI backend logs
duplicate file counts
ChromaDB size before/after the incident
file metadata/status
reproduction steps
the duplicate-detection workaround currently used

if required.

Impact

This can cause severe operational problems for unattended scheduled synchronization:

Unchanged bad source file → repeated uploads → disk exhaustion → repeated embedding attempts → vector DB corruption/resource exhaustion → Open WebUI crash loop.

This makes the issue particularly important for production deployments using scheduled Google Drive synchronization.

Originally created by @ShraddhanandKhot on GitHub (Aug 13, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/106 ## Bug Summary When using **oikb with the Google Drive connector to sync files into an Open WebUI Knowledge Base**, a file that fails during Open WebUI ingestion can be uploaded repeatedly on every sync cycle. The source file remains unchanged in Google Drive, but because the corresponding Open WebUI file is not successfully attached/processed in the Knowledge Base, the next sync can incorrectly treat the same source file as new and upload it again. In my case, this resulted in **1,300+ copies of the same file**, several GB of unnecessary storage usage, and eventually an Open WebUI backend crash loop caused by repeated processing/embedding attempts. ## Environment * oikb: Google Drive connector * Open WebUI: 0.10.x * Sync interval: 1 minute * Deployment: Docker * Vector database: ChromaDB * Source: Google Drive * Files involved: PDF/DOCX files, including scanned/corrupt PDFs and documents with no extractable text ## What happened The problem occurs when a source file is downloaded from Google Drive and uploaded to Open WebUI, but Open WebUI fails while extracting/indexing/embedding the file. For example: ```text Google Drive │ │ file unchanged ▼ oikb detects file │ ▼ Upload to Open WebUI │ ▼ Open WebUI extraction / embedding │ ├── extraction fails │ or ├── content is empty │ or └── embedding/indexing fails │ ▼ File is not successfully linked/processed in the KB │ ▼ Next oikb sync │ ▼ Same source file is detected as "new" │ ▼ Upload again │ ▼ Repeat every sync cycle ``` With a one-minute sync interval, this can quickly create hundreds or thousands of duplicate Open WebUI file records. ## Observed impact One problematic file named: `AUDIT NOTE_Contingent Liabilities...pdf` was uploaded **1,314 times**. This caused: * thousands of duplicate file records * approximately 3.8 GB of files for the duplicated document * disk usage reaching 95–100% * `No space left on device` errors * repeated extraction/embedding attempts * ChromaDB corruption * Open WebUI backend restart/crash loops In one incident, the Open WebUI backend was restarting approximately every 61 seconds, with more than 76 restarts observed. The ChromaDB database eventually had to be rebuilt manually. ## Expected behavior If a file has already been uploaded to Open WebUI but its processing fails, oikb should **not continuously re-upload the same unchanged source file** on every sync cycle. A failed ingestion should result in one of the following behaviors: 1. Mark the source file as failed and stop retrying until the source changes. 2. Record the failure and apply a retry/backoff policy. 3. Detect that the same source file/content already exists in Open WebUI and skip the upload. 4. Surface the failure clearly to the user/operator. 5. Optionally provide a mechanism to retry failed files manually. Most importantly: > A failed ingestion should not result in an unlimited upload loop for an unchanged source file. ## Additional issue: failed files accumulate There is also a secondary cleanup problem. Failed Open WebUI file records remain on disk even when they are no longer successfully linked to a Knowledge Base. In my environment, there were approximately: * 148 file records * 40 files in `failed` status * approximately 79 files not linked to a Knowledge Base These failed/orphaned files continue consuming storage. A mechanism for cleaning up failed/orphaned files, or at least preventing them from accumulating indefinitely, would help prevent gradual disk exhaustion. ## Why this is particularly dangerous The issue is not limited to duplicate database rows. Every repeated upload can trigger Open WebUI's extraction and embedding pipeline again. For a problematic document, this means: ```text 1 bad file ↓ 1 failed extraction ↓ 1 failed embedding attempt ↓ next sync ↓ same file uploaded again ↓ another extraction ↓ another embedding ↓ ... ↓ hundreds/thousands of attempts ``` This can consume: * disk space * CPU * RAM * vector database storage * embedding resources and potentially destabilize or crash the Open WebUI backend. ## Reproduction A simplified reproduction would be: 1. Configure oikb with a Google Drive source and a short sync interval. 2. Create a Knowledge Base in Open WebUI. 3. Place a problematic document in the Google Drive folder. * Example: a corrupt PDF or scanned PDF with no extractable text. 4. Allow oikb to synchronize the file. 5. Allow Open WebUI's extraction/embedding process to fail. 6. Do not modify the source file. 7. Wait for the next oikb sync cycle. 8. Observe that the same source file is uploaded again. 9. Repeat for several sync cycles. 10. Observe multiple Open WebUI file records for the same source document. ## Possible root cause The failure appears to be caused by a mismatch between **oikb's source-side diff state** and **Open WebUI's successful file/Knowledge Base state**. The source file remains unchanged, but the failed Open WebUI ingestion does not leave the system in a state where the next synchronization can reliably identify the source file as already handled. This creates a loop: ```text Source unchanged + Open WebUI ingestion failed + File not recognized as successfully synchronized = Upload again ``` There may also be an interaction with folder-less/root-level files and Open WebUI's file matching/diff behavior. I would appreciate confirmation of the exact root cause from the maintainers. ## Mitigations currently used I implemented two temporary mitigations in my environment: ### 1. Open WebUI-side duplicate detection I added a fallback that uses content hashing to recognize files that already exist even when the normal folder/filename matching does not identify them correctly. ### 2. oikb client-side duplicate guard Before uploading, my oikb client checks the existing Open WebUI files and skips a file when the same filename + content hash already exists. This currently prevents repeated uploads and the sync reports unchanged files instead of uploading them again. However, these are workarounds rather than a complete solution because a **new problematic file can still be uploaded and fail on its first ingestion attempt**. ## Suggested improvements Potential improvements could include: ### A. Persist failed ingestion state Track the source file's content hash and ingestion status: ```text source hash ↓ Open WebUI ingestion ↓ FAILED ↓ remember failure ``` On the next sync: ```text same source hash? │ ├── YES + previously failed │ ↓ │ skip/retry according to policy │ └── NO ↓ upload/process ``` ### B. Retry with backoff Instead of retrying every minute: ```text 1 minute 1 minute 1 minute 1 minute ... ``` use something such as: ```text 1st failure → retry 2nd failure → wait longer 3rd failure → longer backoff 4th failure → mark as failed ``` ### C. Avoid re-uploading known failed content Use a stable source identifier/content hash so that an unchanged failed document is not uploaded repeatedly. ### D. Cleanup failed/orphaned files Provide a safe mechanism to identify and remove files that: * are permanently failed * are not linked to any Knowledge Base * have not changed since the last failed attempt ### E. Alerting Expose sync/ingestion failures through the existing notification mechanism so operators know that a document failed instead of discovering the problem only after the server becomes unavailable. ## Additional context The official oikb documentation describes the project as an incremental synchronization system that should upload only new and modified files. The current documentation also notes that indexing happens server-side in Open WebUI after oikb uploads a file. That separation is important here: oikb can successfully upload a file while Open WebUI subsequently fails during extraction or embedding. The synchronization layer therefore needs a reliable way to distinguish: ```text uploaded successfully ``` from: ```text successfully ingested into the Knowledge Base ``` ## Logs / evidence I can provide: * oikb sync logs * Open WebUI backend logs * duplicate file counts * ChromaDB size before/after the incident * file metadata/status * reproduction steps * the duplicate-detection workaround currently used if required. ## Impact This can cause severe operational problems for unattended scheduled synchronization: **Unchanged bad source file → repeated uploads → disk exhaustion → repeated embedding attempts → vector DB corruption/resource exhaustion → Open WebUI crash loop.** This makes the issue particularly important for production deployments using scheduled Google Drive synchronization.

bsakaguchi commented

2026-10-18 21:14:02 +00:00

Author

Owner

@julianlucks commented on GitHub (Aug 24, 2026):

Confirming this for the local directory connector as well — the loop is connector-agnostic. In our environment, files rejected during ingestion (duplicate content per the KB's content-hash dedup, or no extractable text) were re-uploaded on every sync cycle, accumulating orphaned file records and disk usage server-side.

One additional observation: the dominant channel here is asynchronous. Uploads return HTTP 200 while ingestion runs in the background, and processing errors are swallowed (_process_handler logs the error and sets the file status to failed without re-raising), so oikb never sees the failure synchronously. A client-side guard therefore can't rely on 4xx from POST /api/v1/files/ alone — checking GET /api/v1/files/{id}/process/status after the fact (or tracking uploads and verifying on the next cycle) is needed to detect these rejections.

I've opened #110 with a fix that covers both channels: permanent 4xx rejections are recorded and skipped until the source checksum changes, and accepted uploads are tracked so the next sync checks their server-side status instead of re-uploading. Happy to adjust based on your review.

@julianlucks commented on GitHub (Aug 24, 2026): Confirming this for the **local directory connector** as well — the loop is connector-agnostic. In our environment, files rejected during ingestion (duplicate content per the KB's content-hash dedup, or no extractable text) were re-uploaded on every sync cycle, accumulating orphaned file records and disk usage server-side. One additional observation: the dominant channel here is **asynchronous**. Uploads return HTTP 200 while ingestion runs in the background, and processing errors are swallowed (`_process_handler` logs the error and sets the file status to `failed` without re-raising), so oikb never sees the failure synchronously. A client-side guard therefore can't rely on 4xx from `POST /api/v1/files/` alone — checking `GET /api/v1/files/{id}/process/status` after the fact (or tracking uploads and verifying on the next cycle) is needed to detect these rejections. I've opened #110 with a fix that covers both channels: permanent 4xx rejections are recorded and skipped until the source checksum changes, and accepted uploads are tracked so the next sync checks their server-side status instead of re-uploading. Happy to adjust based on your review.

bsakaguchi commented

2026-10-18 21:14:02 +00:00

Author

Owner

@m63h17 commented on GitHub (Sep 21, 2026):

Experiencing the same issue. Is there anything one can do?

@m63h17 commented on GitHub (Sep 21, 2026): Experiencing the same issue. Is there anything one can do?

bsakaguchi commented

2026-10-18 21:14:02 +00:00

Author

Owner

@ShraddhanandKhot commented on GitHub (Sep 22, 2026):

Experiencing the same issue. Is there anything one can do?
The fix: We injected an AnyDoc pipeline hook into process_file() in retrieval.py that runs a filtration layer (detecting scanned/complex vs. simple files) + AnyDoc markdown converter (with OCR) BEFORE chunking/embedding - so every file, including previously unparseable ones, converts to clean Markdown successfully instead of failing and getting retried forever.

@ShraddhanandKhot commented on GitHub (Sep 22, 2026): > Experiencing the same issue. Is there anything one can do? The fix: We injected an AnyDoc pipeline hook into process_file() in retrieval.py that runs a filtration layer (detecting scanned/complex vs. simple files) + [AnyDoc](https://github.com/firecrawl/anydoc) markdown converter (with OCR) BEFORE chunking/embedding - so every file, including previously unparseable ones, converts to clean Markdown successfully instead of failing and getting retried forever.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#65

No description provided.

Rows
Columns