[GH-ISSUE #60] S3 source sync uploads files but never links them to the Knowledge Base (silent duplicate accumulation)

bsakaguchi commented

2026-10-18 21:13:45 +00:00

Owner

Originally created by @rocrdr on GitHub (Jun 20, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/60

When syncing an S3 source (self-hosted, S3-compatible endpoint with path-style addressing), oikb successfully uploads files via POST /api/v1/files/, but never performs the follow-up call to attach the file to the Knowledge Base (POST /api/v1/knowledge/{kb_id}/file/add). As a result, files appear in Open WebUI's global "Files" manager but never show up in the Knowledge Base itself, and are re-uploaded as duplicates on every subsequent sync.

Originally created by @rocrdr on GitHub (Jun 20, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/60 When syncing an S3 source (self-hosted, S3-compatible endpoint with path-style addressing), oikb successfully uploads files via POST /api/v1/files/, but never performs the follow-up call to attach the file to the Knowledge Base (POST /api/v1/knowledge/{kb_id}/file/add). As a result, files appear in Open WebUI's global "Files" manager but never show up in the Knowledge Base itself, and are re-uploaded as duplicates on every subsequent sync.

bsakaguchi commented

2026-10-18 21:13:47 +00:00

Author

Owner

@rocrdr commented on GitHub (Jun 20, 2026):

Config used (.oikb.yaml):

sources:
  - name: tresorit-source
    source: s3://pdfs
    kb-id: xxxxxxxxxxxxxxxxxxxxxxxxxx
    interval: 10m

Steps to reproduce:

Configure an S3 source pointing at a self-hosted S3-compatible bucket with one file in it.
Run either oikb daemon or oikb sync s3://pdfs --kb-id directly.
Log output shows:

POST .../api/v1/knowledge//sync/diff "HTTP/1.1 200 OK"
POST .../api/v1/files/ "HTTP/1.1 200 OK"
Synced s3://pdfs -> : 1 added (...ms)

Check GET /api/v1/knowledge/ — "files" is null / does not include the new file.
Run sync again — the same file is uploaded again as a new duplicate (file is never recognized as already present, since it was never linked).

Confirmed not an ID issue: The exact same kb-id works correctly when manually performing the two-step process via curl (POST /api/v1/files/ → grab id → POST /api/v1/knowledge//file/add with {"file_id": "..."}). This correctly links the file and is reflected in GET /api/v1/knowledge/.
Reproduces in both:

oikb daemon (scheduled mode)
oikb sync s3://... --kb-id ... (direct one-off CLI invocation)

So this isn't specific to daemon scheduling — it's in the core S3 source sync logic itself.
Expected: S3 source sync should call the file/add (or equivalent) endpoint after upload, same as it presumably does for local directory / GitHub sources, so files are correctly attached to the Knowledge Base and incremental diffing works as documented.
Environment: ghcr.io/open-webui/oikb:latest, oikb[s3] extra, Open WebUI 0.9.6+, custom S3-compatible endpoint (path-style, custom CA bundle).

@rocrdr commented on GitHub (Jun 20, 2026): Config used (.oikb.yaml): ``` sources: - name: tresorit-source source: s3://pdfs kb-id: xxxxxxxxxxxxxxxxxxxxxxxxxx interval: 10m ``` Steps to reproduce: Configure an S3 source pointing at a self-hosted S3-compatible bucket with one file in it. Run either oikb daemon or oikb sync s3://pdfs --kb-id <kb-id> directly. Log output shows: POST .../api/v1/knowledge/<kb-id>/sync/diff "HTTP/1.1 200 OK" POST .../api/v1/files/ "HTTP/1.1 200 OK" Synced s3://pdfs -> <kb-id>: 1 added (...ms) Check GET /api/v1/knowledge/<kb-id> — "files" is null / does not include the new file. Run sync again — the same file is uploaded again as a new duplicate (file is never recognized as already present, since it was never linked). Confirmed not an ID issue: The exact same kb-id works correctly when manually performing the two-step process via curl (POST /api/v1/files/ → grab id → POST /api/v1/knowledge/<kb-id>/file/add with {"file_id": "..."}). This correctly links the file and is reflected in GET /api/v1/knowledge/<kb-id>. Reproduces in both: oikb daemon (scheduled mode) oikb sync s3://... --kb-id ... (direct one-off CLI invocation) So this isn't specific to daemon scheduling — it's in the core S3 source sync logic itself. Expected: S3 source sync should call the file/add (or equivalent) endpoint after upload, same as it presumably does for local directory / GitHub sources, so files are correctly attached to the Knowledge Base and incremental diffing works as documented. Environment: ghcr.io/open-webui/oikb:latest, oikb[s3] extra, Open WebUI 0.9.6+, custom S3-compatible endpoint (path-style, custom CA bundle).

bsakaguchi commented

2026-10-18 21:13:47 +00:00

Author

Owner

@rocrdr commented on GitHub (Jun 23, 2026):

Possibly related to #10 (the "files" field on GET /knowledge/{kb_id} may not be the authoritative source) — but confirmed via the Open WebUI GUI itself that files uploaded through the S3 source never appear in the Knowledge Base, only in the standalone Files manager.

@rocrdr commented on GitHub (Jun 23, 2026): Possibly related to #10 (the "files" field on GET /knowledge/{kb_id} may not be the authoritative source) — but confirmed via the Open WebUI GUI itself that files uploaded through the S3 source never appear in the Knowledge Base, only in the standalone Files manager.

bsakaguchi commented

2026-10-18 21:13:47 +00:00

Author

Owner

@rocrdr commented on GitHub (Jul 14, 2026):

Root cause: upload_file() in client.py only calls POST /files/ and sets knowledge_id in the file's own metadata — it never calls the actual attach endpoint POST /knowledge/{kb_id}/file/add. That's why files never register in the KB and get re-uploaded as duplicates on every sync. This affects all connectors, not just S3.

Suggested fix in client.py:

resp = self._http.post(
    "/files/",
    files={"file": (filename, file_content)},
    data={"metadata": json.dumps(metadata)},
)
resp.raise_for_status()
uploaded = resp.json()

file_id = uploaded.get("id")
if file_id:
    self._attach_file(kb_id, file_id)

return uploaded

def _attach_file(self, kb_id: str, file_id: str) -> None:
    """POST /knowledge/{id}/file/add — link an uploaded file into the KB."""
    resp = self._http.post(
        f"/knowledge/{kb_id}/file/add",
        json={"file_id": file_id},
    )
    resp.raise_for_status()

This also looks like the root cause of #10 ("List KB is always empty and status error"): ⁠list_kb_files()⁠ and ⁠get_kb()⁠ both read the ⁠files⁠ field from ⁠GET /knowledge/{kb_id}⁠, which server-side only contains files that went through ⁠file/add⁠. Since that call is never made, ⁠oikb ls⁠ and ⁠oikb status⁠ report an empty KB even though files were uploaded. Fixing the attach call above should resolve both issues.
Happy to open a PR if that's useful, just flagging the fix here for now.

@rocrdr commented on GitHub (Jul 14, 2026): Root cause: `upload_file()` in `client.py` only calls `POST /files/` and sets `knowledge_id` in the file's own metadata — it never calls the actual attach endpoint `POST /knowledge/{kb_id}/file/add`. That's why files never register in the KB and get re-uploaded as duplicates on every sync. This affects all connectors, not just S3. Suggested fix in `client.py`: ```python resp = self._http.post( "/files/", files={"file": (filename, file_content)}, data={"metadata": json.dumps(metadata)}, ) resp.raise_for_status() uploaded = resp.json() file_id = uploaded.get("id") if file_id: self._attach_file(kb_id, file_id) return uploaded def _attach_file(self, kb_id: str, file_id: str) -> None: """POST /knowledge/{id}/file/add — link an uploaded file into the KB.""" resp = self._http.post( f"/knowledge/{kb_id}/file/add", json={"file_id": file_id}, ) resp.raise_for_status() ``` This also looks like the root cause of #10 ("List KB is always empty and status error"): ⁠list_kb_files()⁠ and ⁠get_kb()⁠ both read the ⁠files⁠ field from ⁠GET /knowledge/{kb_id}⁠, which server-side only contains files that went through ⁠file/add⁠. Since that call is never made, ⁠oikb ls⁠ and ⁠oikb status⁠ report an empty KB even though files were uploaded. Fixing the attach call above should resolve both issues. Happy to open a PR if that's useful, just flagging the fix here for now.

bsakaguchi commented

2026-10-18 21:13:48 +00:00

Author

Owner

@weberdigital commented on GitHub (Jul 29, 2026):

Same issue for syncing local files via oikb daemon to open webui. Request gets status 200 OK on every sync, but the file never appears in the knowledge base. When uploading manually via open webui gui, i get an error because the file is not ok. I need this error info while oikb sync, so i can handle automatically. Can you confirm that this is the same problem on your side?

@weberdigital commented on GitHub (Jul 29, 2026): Same issue for syncing local files via oikb daemon to open webui. Request gets status 200 OK on every sync, but the file never appears in the knowledge base. When uploading manually via open webui gui, i get an error because the file is not ok. I need this error info while oikb sync, so i can handle automatically. Can you confirm that this is the same problem on your side?

bsakaguchi commented

2026-10-18 21:13:48 +00:00

Author

Owner

@rocrdr commented on GitHub (Jul 29, 2026):

Can confirm the same root cause on my side (S3 source) — sync uploads via POST /api/v1/files/ but never calls the KB link endpoint. Difference: manual upload via the GUI works fine for me, no validation error there.

@rocrdr commented on GitHub (Jul 29, 2026): Can confirm the same root cause on my side (S3 source) — sync uploads via POST /api/v1/files/ but never calls the KB link endpoint. Difference: manual upload via the GUI works fine for me, no validation error there.

bsakaguchi commented

2026-10-18 21:13:48 +00:00

Author

Owner

@brownhouse commented on GitHub (Sep 10, 2026):

Confirming this also affects local-directory sync, not just S3, and not
specific to daemon mode (weberdigital's Jul 29 comment already covers
daemon; mine is a plain manual oikb sync).

Config:

sources:
  - name: local-docs
    source: E:\Obsidian\jasonlbrown
    kb-id: 96c5c1bc-5797-4fd1-ad17-4082e540256f
    filter:
      exclude: ["**/*.mp3"]

oikb 0.4.0, run manually via oikb sync, no daemon.

Symptoms match the root cause in the Jul 14 comment exactly:

Each sync run reports files as "added" and HTTP 200 the whole way.
The Knowledge Base itself showed only 14 files in the Open WebUI UI.
Open WebUI's global Files manager (Settings > Data Controls > Manage
Files) showed 109 files, matching the cumulative total oikb had
reported as "added" across several runs.
Because the files were never linked, oikb's own diffing never sees
them as "already synced," so most of the same files get re-uploaded
as new duplicates on every subsequent run instead of being reported
unchanged.

Working around it for now by calling POST /knowledge/{kb_id}/file/add
myself for every unlinked file after each sync, and cleaning up the
orphaned duplicates that piled up in the global Files manager before
this was diagnosed. Would rather not keep doing that by hand long term,
happy to test a fix if one lands.

@brownhouse commented on GitHub (Sep 10, 2026): Confirming this also affects local-directory sync, not just S3, and not specific to daemon mode (weberdigital's Jul 29 comment already covers daemon; mine is a plain manual `oikb sync`). Config: ```yaml sources: - name: local-docs source: E:\Obsidian\jasonlbrown kb-id: 96c5c1bc-5797-4fd1-ad17-4082e540256f filter: exclude: ["**/*.mp3"] ``` oikb 0.4.0, run manually via `oikb sync`, no daemon. Symptoms match the root cause in the Jul 14 comment exactly: - Each sync run reports files as "added" and HTTP 200 the whole way. - The Knowledge Base itself showed only 14 files in the Open WebUI UI. - Open WebUI's global Files manager (Settings > Data Controls > Manage Files) showed 109 files, matching the cumulative total oikb had reported as "added" across several runs. - Because the files were never linked, oikb's own diffing never sees them as "already synced," so most of the same files get re-uploaded as new duplicates on every subsequent run instead of being reported unchanged. Working around it for now by calling `POST /knowledge/{kb_id}/file/add` myself for every unlinked file after each sync, and cleaning up the orphaned duplicates that piled up in the global Files manager before this was diagnosed. Would rather not keep doing that by hand long term, happy to test a fix if one lands.

Rows
Columns

[GH-ISSUE #60] S3 source sync uploads files but never links them to the Knowledge Base (silent duplicate accumulation) #40