open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[GH-ISSUE #70] Bug in GDrive Connector: Subfolders not synced #48

New issue

Closed

opened 2026-10-18 21:13:50 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-18 21:13:50 +00:00

Owner

Originally created by @MoeLa on GitHub (Jun 30, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/70

Description

The Google Drive connector only syncs files from the root folder. All subfolders and their contents are ignored. Additionally, if a previous sync had already ingested files from subfolders (e.g. via manual upload), a subsequent connector sync deletes those entries from the Knowledge Base, because the manifest no longer contains them.

Root Cause

application/vnd.google-apps.folder is listed in _SKIP_MIMES, which causes the early continue in _walk_folder to fire before the recursive branch is ever reached:

_SKIP_MIMES = frozenset({
    "application/vnd.google-apps.folder",  # <-- skips folders before recursion
    "application/vnd.google-apps.form",
    ...
})

def _walk_folder(self, ...):
    for item in resp.get("files", []):
        mime = item["mimeType"]

        if mime in _SKIP_MIMES:  # folder hits this first
            continue

        if mime == "application/vnd.google-apps.folder":  # dead code
            self._walk_folder(...)  # never reached
            continue

The recursive subfolder handling is effectively dead code.

Fix

Remove application/vnd.google-apps.folder from _SKIP_MIMES so the dedicated recursion branch is reached:

_SKIP_MIMES = frozenset({
    # "application/vnd.google-apps.folder",  # handled separately below
    "application/vnd.google-apps.form",
    "application/vnd.google-apps.map",
    "application/vnd.google-apps.site",
    "application/vnd.google-apps.shortcut",
})

Steps to Reproduce

Create a Google Drive folder with at least one subfolder containing files
Configure the GDrive connector pointing to the root folder
Run oikb sync
Observe that only files in the root are synced; subfolder contents are absent from the manifest

Impact

Subfolder contents are never synced
Previously ingested subfolder files are deleted from the Knowledge Base on the next sync run, because the manifest shrinks

Originally created by @MoeLa on GitHub (Jun 30, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/70 ### Description The Google Drive connector only syncs files from the root folder. All subfolders and their contents are ignored. Additionally, if a previous sync had already ingested files from subfolders (e.g. via manual upload), a subsequent connector sync **deletes** those entries from the Knowledge Base, because the manifest no longer contains them. ### Root Cause `application/vnd.google-apps.folder` is listed in `_SKIP_MIMES`, which causes the early `continue` in `_walk_folder` to fire before the recursive branch is ever reached: ```python _SKIP_MIMES = frozenset({ "application/vnd.google-apps.folder", # <-- skips folders before recursion "application/vnd.google-apps.form", ... }) def _walk_folder(self, ...): for item in resp.get("files", []): mime = item["mimeType"] if mime in _SKIP_MIMES: # folder hits this first continue if mime == "application/vnd.google-apps.folder": # dead code self._walk_folder(...) # never reached continue ``` The recursive subfolder handling is effectively dead code. ### Fix Remove `application/vnd.google-apps.folder` from `_SKIP_MIMES` so the dedicated recursion branch is reached: ```python _SKIP_MIMES = frozenset({ # "application/vnd.google-apps.folder", # handled separately below "application/vnd.google-apps.form", "application/vnd.google-apps.map", "application/vnd.google-apps.site", "application/vnd.google-apps.shortcut", }) ``` ### Steps to Reproduce 1. Create a Google Drive folder with at least one subfolder containing files 2. Configure the GDrive connector pointing to the root folder 3. Run `oikb sync` 4. Observe that only files in the root are synced; subfolder contents are absent from the manifest ### Impact - Subfolder contents are never synced - Previously ingested subfolder files are **deleted** from the Knowledge Base on the next sync run, because the manifest shrinks

bsakaguchi closed this issue

2026-10-18 21:13:50 +00:00

bsakaguchi commented

2026-10-18 21:13:52 +00:00

Author

Owner

@taricjain-pbl commented on GitHub (Jul 9, 2026):

Facing this issue as well. Can't wait to add this to our setup 🚀 I acknowledge Tim and everyone else have a lot on their plates, but would love to see this merged into main

@taricjain-pbl commented on GitHub (Jul 9, 2026): Facing this issue as well. Can't wait to add this to our setup 🚀 I acknowledge Tim and everyone else have a lot on their plates, but would love to see this merged into main

bsakaguchi commented

2026-10-18 21:13:52 +00:00

Author

Owner

@ShraddhanandKhot commented on GitHub (Aug 5, 2026):

I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks!

@ShraddhanandKhot commented on GitHub (Aug 5, 2026): I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks!

bsakaguchi commented

2026-10-18 21:13:53 +00:00

Author

Owner

@taricjain-pbl commented on GitHub (Aug 5, 2026):

I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks!

The PR got merged but I'm still seeing the same error on my end, not sure why.

@taricjain-pbl commented on GitHub (Aug 5, 2026): > I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks! > The PR got merged but I'm still seeing the same error on my end, not sure why.

bsakaguchi commented

2026-10-18 21:13:53 +00:00

Author

Owner

@ShraddhanandKhot commented on GitHub (Aug 5, 2026):

I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks!

The PR got merged but I'm still seeing the same error on my end, not sure why.

I haven't tried the latest version yet. I made the same change locally in my installation, and it fixed the issue for me.
I'll check the latest version when I get a chance.

@ShraddhanandKhot commented on GitHub (Aug 5, 2026): > > I ran into the same issue and ended up finding the same fix on my own before I came across this issue. Glad to see it's already been reported. Looking forward to seeing the fix merged into the main branch. Thanks! > > The PR got merged but I'm still seeing the same error on my end, not sure why. I haven't tried the latest version yet. I made the same change locally in my installation, and it fixed the issue for me. I'll check the latest version when I get a chance.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#48

No description provided.

Rows
Columns