open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[PR #139] fix(sharepoint): follow @odata.nextLink when listing folders #130

New issue

Open

opened 2026-10-18 21:14:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/139
Author: @roydonsequeira
Created: 10/5/2026
Status: 🔄 Open

Base: main ← Head: fix/sharepoint-children-paging

📝 Commits (2)

add2172 fix(sharepoint): follow @odata.nextLink when listing folders
3e577bf Merge remote-tracking branch 'origin/main' into fix/sharepoint-children-paging

📊 Changes

2 files changed (+70 additions, -5 deletions)

View changed files

📝 src/oikb/connectors/sharepoint.py (+10 -4)
📝 tests/test_sharepoint.py (+60 -1)

📄 Description

Fixes #96.

_walk_folder makes one GET .../children request per folder and reads only its value:

resp = self._http.get(url)
resp.raise_for_status()

for item in resp.json().get("value", []):

Graph returns 200 children per page by default and links the rest through @odata.nextLink (list children, paging). Nothing reads that link, so:

a folder with more than 200 entries is cut short, and the sync still reports success;
the manifest drives /sync/diff, so files synced before the folder outgrew one page come back as deleted, and sync_cleanup removes them from the Knowledge Base;
folders and files share the page, so a subfolder that lands on a later page loses its whole subtree.

_walk_folder now collects every page of a folder, following @odata.nextLink until Graph stops sending one, then walks the items exactly as before. The link is an absolute URL, and httpx sends absolute URLs as given even with base_url set. So it goes back unchanged, as the paging docs ask, whichever cloud it points at. An empty page that still carries a link is followed too; the docs say a page may hold zero results.

Verification

I don't have a SharePoint tenant, so this is verified against Graph-shaped responses. #96 has a live reproduction against a real library: $top=50 on a 187-entry folder returns 50 entries and a nextLink.

The new test in tests/test_sharepoint.py builds the connector against mocked Graph routes:

a root listing split over three pages, the middle one empty;
a subfolder that only appears on the last page and is itself split over two pages.

On main the manifest has 2 of the 5 files (Reports/q1.xlsx, a.pdf). With this change it has all 5, and the root is requested three times, using each nextLink exactly as given. The test also fails if the loop stops at an empty page or follows only one link. Full suite: 52 passed on main (672094b), 53 on this branch (Python 3.11).

#137 landed first and also added tests/test_sharepoint.py, so this branch now merges main (3e577bf). Its tests are unchanged, except that their token, site and drive routes moved into _mock_site, which the paging test reuses.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/139 **Author:** [@roydonsequeira](https://github.com/roydonsequeira) **Created:** 10/5/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/sharepoint-children-paging` --- ### 📝 Commits (2) - [`add2172`](https://github.com/open-webui/oikb/commit/add2172bab04fb3bdfaae3285398c43a375690c4) fix(sharepoint): follow @odata.nextLink when listing folders - [`3e577bf`](https://github.com/open-webui/oikb/commit/3e577bf0a96b1fd4693fe50841ca0447eabb39f9) Merge remote-tracking branch 'origin/main' into fix/sharepoint-children-paging ### 📊 Changes **2 files changed** (+70 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `src/oikb/connectors/sharepoint.py` (+10 -4) 📝 `tests/test_sharepoint.py` (+60 -1) </details> ### 📄 Description Fixes #96. `_walk_folder` makes one `GET .../children` request per folder and reads only its `value`: ```python resp = self._http.get(url) resp.raise_for_status() for item in resp.json().get("value", []): ``` Graph returns 200 children per page by default and links the rest through `@odata.nextLink` ([list children](https://learn.microsoft.com/en-us/graph/api/driveitem-list-children), [paging](https://learn.microsoft.com/en-us/graph/paging)). Nothing reads that link, so: - a folder with more than 200 entries is cut short, and the sync still reports success; - the manifest drives `/sync/diff`, so files synced before the folder outgrew one page come back as deleted, and `sync_cleanup` removes them from the Knowledge Base; - folders and files share the page, so a subfolder that lands on a later page loses its whole subtree. `_walk_folder` now collects every page of a folder, following `@odata.nextLink` until Graph stops sending one, then walks the items exactly as before. The link is an absolute URL, and httpx sends absolute URLs as given even with `base_url` set. So it goes back unchanged, as the paging docs ask, whichever cloud it points at. An empty page that still carries a link is followed too; the docs say a page may hold zero results. ### Verification I don't have a SharePoint tenant, so this is verified against Graph-shaped responses. #96 has a live reproduction against a real library: `$top=50` on a 187-entry folder returns 50 entries and a nextLink. The new test in `tests/test_sharepoint.py` builds the connector against mocked Graph routes: - a root listing split over three pages, the middle one empty; - a subfolder that only appears on the last page and is itself split over two pages. On main the manifest has 2 of the 5 files (`Reports/q1.xlsx`, `a.pdf`). With this change it has all 5, and the root is requested three times, using each nextLink exactly as given. The test also fails if the loop stops at an empty page or follows only one link. Full suite: 52 passed on main (672094b), 53 on this branch (Python 3.11). #137 landed first and also added `tests/test_sharepoint.py`, so this branch now merges main (3e577bf). Its tests are unchanged, except that their token, site and drive routes moved into `_mock_site`, which the paging test reuses. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-18 21:14:35 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#130

No description provided.

Rows
Columns