open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[PR #138] fix: let **/ match zero directories in filters and .oikbignore #129

New issue

Open

opened 2026-10-18 21:14:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/138
Author: @roydonsequeira
Created: 10/5/2026
Status: 🔄 Open

Base: main ← Head: fix/glob-double-star-zero-dirs

📝 Commits (1)

f013516 fix: let **/ match zero directories in filters and .oikbignore

📊 Changes

3 files changed (+91 additions, -5 deletions)

View changed files

📝 src/oikb/ignore.py (+27 -2)
📝 src/oikb/sync.py (+3 -3)
➕ tests/test_ignore.py (+61 -0)

📄 Description

Fixes #33.

build_manifest_filter and .oikbignore both match with fnmatch. Its * crosses /, but the / after ** is still a literal, so **/ needs at least one directory before it can match:

>>> fnmatch.fnmatch("Note.md", "**/*.md")
False
>>> fnmatch.fnmatch("docs/index.md", "docs/**/*.md")
False

So the patterns in the README and guide don't do what they read as:

include: ["**/*.md"] (the config in #33) syncs every Markdown file except the ones at the top of the source.
include: ["docs/**/*.md"], the README example, skips docs/index.md and everything else directly in docs/.
exclude: ["**/*.tmp"], and .oikbignore lines like **/*.tmp or **/archive/, never apply at the root.

The filter runs on every connector's manifest, so this isn't specific to local directories.

glob_match in oikb/ignore.py tries each pattern with every **/ that starts the pattern or follows a / both kept and dropped, which is how gitignore reads it. The unchanged pattern is tried first, so everything that matched before still matches, including * crossing / (the Confluence tree filters such as FAQ* rely on that). ** glued to other characters, as in a**/b, stays a plain *.

Sources that use these patterns will see a one-off diff on their next sync: the top-level files they were missing get added, and top-level files their exclude or .oikbignore names get removed.

Verification

FilesystemConnector on the vault layout from #33, on main and on this branch:

include: ["**/*.md"]: main drops Documentation-X.md, Note-Y.md and Reference-Z.md; this branch keeps all seven notes.
include: ["docs/**/*.md"]: main keeps only docs/guide/setup.md; this branch also keeps docs/index.md.
exclude: ["**/*.tmp"] and .oikbignore with **/*.tmp / **/archive/: main still syncs scratch.tmp and archive/; this branch leaves them out.

tests/test_ignore.py covers the filter and should_ignore on that layout, plus cases that must not match (mydocs/index.md for docs/**/*.md, notes.mdx for **/*.md, ab for a**/b) and a pattern with two **/ where only one spans a directory. The two behaviour tests fail on main. Full suite: 39 passed on main, 48 on this branch (Python 3.11).

On a 50,000-entry manifest with three **/ patterns the filter takes about 0.4 s, up from 0.17 s.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/138 **Author:** [@roydonsequeira](https://github.com/roydonsequeira) **Created:** 10/5/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/glob-double-star-zero-dirs` --- ### 📝 Commits (1) - [`f013516`](https://github.com/open-webui/oikb/commit/f013516feb6e38f655e4e8c4997abd87b43683d5) fix: let **/ match zero directories in filters and .oikbignore ### 📊 Changes **3 files changed** (+91 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `src/oikb/ignore.py` (+27 -2) 📝 `src/oikb/sync.py` (+3 -3) ➕ `tests/test_ignore.py` (+61 -0) </details> ### 📄 Description Fixes #33. `build_manifest_filter` and `.oikbignore` both match with `fnmatch`. Its `*` crosses `/`, but the `/` after `**` is still a literal, so `**/` needs at least one directory before it can match: ```python >>> fnmatch.fnmatch("Note.md", "**/*.md") False >>> fnmatch.fnmatch("docs/index.md", "docs/**/*.md") False ``` So the patterns in the README and guide don't do what they read as: - `include: ["**/*.md"]` (the config in #33) syncs every Markdown file except the ones at the top of the source. - `include: ["docs/**/*.md"]`, the README example, skips `docs/index.md` and everything else directly in `docs/`. - `exclude: ["**/*.tmp"]`, and `.oikbignore` lines like `**/*.tmp` or `**/archive/`, never apply at the root. The filter runs on every connector's manifest, so this isn't specific to local directories. `glob_match` in `oikb/ignore.py` tries each pattern with every `**/` that starts the pattern or follows a `/` both kept and dropped, which is how gitignore reads it. The unchanged pattern is tried first, so everything that matched before still matches, including `*` crossing `/` (the Confluence tree filters such as `FAQ*` rely on that). `**` glued to other characters, as in `a**/b`, stays a plain `*`. Sources that use these patterns will see a one-off diff on their next sync: the top-level files they were missing get added, and top-level files their `exclude` or `.oikbignore` names get removed. ### Verification `FilesystemConnector` on the vault layout from #33, on main and on this branch: - `include: ["**/*.md"]`: main drops `Documentation-X.md`, `Note-Y.md` and `Reference-Z.md`; this branch keeps all seven notes. - `include: ["docs/**/*.md"]`: main keeps only `docs/guide/setup.md`; this branch also keeps `docs/index.md`. - `exclude: ["**/*.tmp"]` and `.oikbignore` with `**/*.tmp` / `**/archive/`: main still syncs `scratch.tmp` and `archive/`; this branch leaves them out. `tests/test_ignore.py` covers the filter and `should_ignore` on that layout, plus cases that must not match (`mydocs/index.md` for `docs/**/*.md`, `notes.mdx` for `**/*.md`, `ab` for `a**/b`) and a pattern with two `**/` where only one spans a directory. The two behaviour tests fail on main. Full suite: 39 passed on main, 48 on this branch (Python 3.11). On a 50,000-entry manifest with three `**/` patterns the filter takes about 0.4 s, up from 0.17 s. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-18 21:14:35 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#129

No description provided.

Rows
Columns