open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[GH-ISSUE #96] SharePoint connector ignores `@odata.nextLink` — folders with more than one page of children are silently truncated #61

New issue

Open

opened 2026-10-18 21:14:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:00 +00:00

Owner

Originally created by @wichmann-git on GitHub (Aug 4, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/96

Summary

SharePointConnector._walk_folder issues a single GET .../children request per folder and reads only the value array. Microsoft Graph paginates this endpoint: when a folder has more children than fit in one page, the response carries an @odata.nextLink pointing at the next page. That field is never read, so everything beyond the first page is invisible to the connector.

There is no error. raise_for_status() passes, the sync reports success, and the missing files are indistinguishable — from oikb's point of view — from files that do not exist in SharePoint.

Impact

Worse than "some files are not uploaded": because the manifest drives the diff, files that were synced before the folder crossed the page boundary are now considered deleted at the source and get removed from the Knowledge Base on the next run. A sync that reports N unchanged, M deleted looks healthy while silently shrinking the KB.

Two aggravating factors:

/children returns folders and files in one list, so the page budget is shared. A folder with 150 files and 60 subfolders is already past 200 entries without ever holding 200 files.
A truncated subfolder entry costs the entire subtree. _walk_folder only recurses into folders it actually saw in value, so one dropped folder entry silently removes every file beneath it, at any depth.

Affected code

src/oikb/connectors/sharepoint.py — identical on main and in the v0.4.0 release image:

def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None:
    url = f"/drives/{self._drive_id}/root/children" if folder_path == "/" else f"/drives/{self._drive_id}/root:/{folder_path}:/children"
    resp = self._http.get(url)
    resp.raise_for_status()

    for item in resp.json().get("value", []):
        if "folder" in item:
            ...
            self._walk_folder(child_path, sub, entries)
        elif "file" in item:
            ...

Reached from build_manifest(), so every SharePoint sync is affected.

Reproduction

The default page size for driveItem children is 200, so this only bites on large folders. $top reproduces it at any size — the connector's behaviour is identical, only the threshold moves:

# Same request shape the connector makes, against a folder with 187 children.
base = f"https://graph.microsoft.com/v1.0/drives/{drive_id}/root:/{folder}:/children"

for url in (base, base + "?$top=50"):
    page = requests.get(url, headers=auth).json()
    print(len(page.get("value", [])), "@odata.nextLink" in page)

# 187 False   <- fits in one page, connector is correct
#  50 True    <- connector returns 50 of 187 and reports success

Following @odata.nextLink on the same folder yields all 187 entries across 4 pages.

Observed against a real tenant: a document library with 536 files in three monthly folders (154 / 181 / 187 entries). Nothing is lost today; the largest folder is 13 entries from the threshold and grows monthly.

Suggested fix

Loop over pages instead of reading one response:

def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None:
    url = (
        f"/drives/{self._drive_id}/root/children"
        if folder_path == "/"
        else f"/drives/{self._drive_id}/root:/{folder_path}:/children"
    )

    while url:
        resp = self._http.get(url)
        resp.raise_for_status()
        data = resp.json()

        for item in data.get("value", []):
            if "folder" in item:
                sub = f"{prefix}/{item['name']}" if prefix else item["name"]
                child_path = f"{folder_path}/{item['name']}" if folder_path != "/" else item["name"]
                self._walk_folder(child_path, sub, entries)
            elif "file" in item:
                etag = (item.get("eTag") or item.get("cTag", "")).strip('"')
                entries.append(ManifestEntry(
                    filename=item["name"],
                    path=prefix,
                    checksum=etag[:16] if etag else "",
                    size=item.get("size", 0),
                ))

        url = data.get("@odata.nextLink")

One implementation note: @odata.nextLink is an absolute URL. httpx uses absolute URLs as given even when the client has a base_url, so it can be passed straight back into self._http.get().

Prior art in this repo: the Teams connector talks to Graph and already follows @odata.nextLink, and #69 fixed the same class of bug for Jira. SharePoint looks like an oversight rather than a design decision.

Environment

oikb v0.4.0 (ghcr.io/open-webui/oikb:0.4.0), daemon mode
Auth: client secret (SHAREPOINT_CLIENT_ID / SHAREPOINT_TENANT_ID / SHAREPOINT_CLIENT_SECRET), Graph application permission Sites.Read.All
Source: sharepoint:<site-id>/Documents
Verified unfixed on main at the time of writing

Only sharepoint.py was examined — other connectors may share the pattern.

Originally created by @wichmann-git on GitHub (Aug 4, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/96 ## Summary `SharePointConnector._walk_folder` issues a single `GET .../children` request per folder and reads only the `value` array. Microsoft Graph paginates this endpoint: when a folder has more children than fit in one page, the response carries an `@odata.nextLink` pointing at the next page. That field is never read, so everything beyond the first page is invisible to the connector. There is no error. `raise_for_status()` passes, the sync reports success, and the missing files are indistinguishable — from oikb's point of view — from files that do not exist in SharePoint. ## Impact Worse than "some files are not uploaded": because the manifest drives the diff, files that were synced before the folder crossed the page boundary are now considered *deleted at the source* and get removed from the Knowledge Base on the next run. A sync that reports `N unchanged, M deleted` looks healthy while silently shrinking the KB. Two aggravating factors: 1. **`/children` returns folders and files in one list**, so the page budget is shared. A folder with 150 files and 60 subfolders is already past 200 entries without ever holding 200 files. 2. **A truncated *subfolder* entry costs the entire subtree.** `_walk_folder` only recurses into folders it actually saw in `value`, so one dropped folder entry silently removes every file beneath it, at any depth. ## Affected code `src/oikb/connectors/sharepoint.py` — identical on `main` and in the `v0.4.0` release image: ```python def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None: url = f"/drives/{self._drive_id}/root/children" if folder_path == "/" else f"/drives/{self._drive_id}/root:/{folder_path}:/children" resp = self._http.get(url) resp.raise_for_status() for item in resp.json().get("value", []): if "folder" in item: ... self._walk_folder(child_path, sub, entries) elif "file" in item: ... ``` Reached from `build_manifest()`, so every SharePoint sync is affected. ## Reproduction The default page size for `driveItem` children is 200, so this only bites on large folders. `$top` reproduces it at any size — the connector's behaviour is identical, only the threshold moves: ```python # Same request shape the connector makes, against a folder with 187 children. base = f"https://graph.microsoft.com/v1.0/drives/{drive_id}/root:/{folder}:/children" for url in (base, base + "?$top=50"): page = requests.get(url, headers=auth).json() print(len(page.get("value", [])), "@odata.nextLink" in page) # 187 False <- fits in one page, connector is correct # 50 True <- connector returns 50 of 187 and reports success ``` Following `@odata.nextLink` on the same folder yields all 187 entries across 4 pages. Observed against a real tenant: a document library with 536 files in three monthly folders (154 / 181 / 187 entries). Nothing is lost today; the largest folder is 13 entries from the threshold and grows monthly. ## Suggested fix Loop over pages instead of reading one response: ```python def _walk_folder(self, folder_path: str, prefix: str, entries: list[ManifestEntry]) -> None: url = ( f"/drives/{self._drive_id}/root/children" if folder_path == "/" else f"/drives/{self._drive_id}/root:/{folder_path}:/children" ) while url: resp = self._http.get(url) resp.raise_for_status() data = resp.json() for item in data.get("value", []): if "folder" in item: sub = f"{prefix}/{item['name']}" if prefix else item["name"] child_path = f"{folder_path}/{item['name']}" if folder_path != "/" else item["name"] self._walk_folder(child_path, sub, entries) elif "file" in item: etag = (item.get("eTag") or item.get("cTag", "")).strip('"') entries.append(ManifestEntry( filename=item["name"], path=prefix, checksum=etag[:16] if etag else "", size=item.get("size", 0), )) url = data.get("@odata.nextLink") ``` One implementation note: `@odata.nextLink` is an absolute URL. `httpx` uses absolute URLs as given even when the client has a `base_url`, so it can be passed straight back into `self._http.get()`. Prior art in this repo: the Teams connector talks to Graph and already follows `@odata.nextLink`, and #69 fixed the same class of bug for Jira. SharePoint looks like an oversight rather than a design decision. ## Environment - oikb `v0.4.0` (`ghcr.io/open-webui/oikb:0.4.0`), daemon mode - Auth: client secret (`SHAREPOINT_CLIENT_ID` / `SHAREPOINT_TENANT_ID` / `SHAREPOINT_CLIENT_SECRET`), Graph application permission `Sites.Read.All` - Source: `sharepoint:<site-id>/Documents` - Verified unfixed on `main` at the time of writing Only `sharepoint.py` was examined — other connectors may share the pattern.

bsakaguchi referenced this issue

2026-10-18 21:14:19 +00:00

[PR #61] [CLOSED] feat(zotero): add Zotero connector #95

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#61

No description provided.

Rows
Columns

[GH-ISSUE #96] SharePoint connector ignores @odata.nextLink — folders with more than one page of children are silently truncated #61