[GH-ISSUE #127] Webhook triggers only one knowledge base when one repository is split across several #73

New issue

Open

opened 2026-10-18 21:14:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:06 +00:00

Owner

Originally created by @etho201 on GitHub (Sep 20, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/127

Summary

When a single repository is split across multiple knowledge bases — the pattern
documented under Splitting Sources Across KBs in docs/guide.md — a push
webhook syncs only one of them. The others wait for their scheduled interval.

Expected

A push triggers a sync for every affected source, or at minimum every entry
carrying webhook: true.

Actual

Exactly one entry syncs. Which one depends on its position in sources:.

Reproduction

.oikb.yaml:

defaults:
  interval: 1h

sources:
  - name: shared
    source: github:ORG/REPO
    path: _shared
    target-path: shared
    kb-id: <kb-1>
    webhook: true
    github_secret: ${GITHUB_WEBHOOK_SECRET}

  - name: client-a
    source: github:ORG/REPO
    path: clients/a
    target-path: a
    kb-id: <kb-2>
    webhook: true

  # ... four more entries, same source, different kb-id

Daemon startup:

6 source(s) configured
Scheduling github:ORG/REPO every 3600s     (repeated 6×)

Push to the repository. One sync runs against <kb-1>; the other five knowledge
bases stay stale until their interval elapses.

Adding webhook: true to every entry does not change this.

Cause

src/oikb/webhooks.py returns on the first match:

def _find_entry_by_source(prefix: str, match: str) -> dict | None:
    for entry in self._entries:
        source = entry.get("source", "")
        if source.startswith(prefix) and match in source and entry.get("webhook"):
            return entry                      # <-- stops here

@router.post("/github")
async def github_webhook(request: Request):
    ...
    repo = data.get("repository", {}).get("full_name", "")
    entry = _find_entry_by_source("github:", repo)
    if entry and _run_entry:
        asyncio.create_task(_run_entry(entry))   # one entry only

_run_entry_locked then narrows further to a single knowledge base:

entries = [e for e in _entries if e.get("kb-id") == entry["kb-id"]] or [entry]

All entries here share the same source string, so match in source is true for
every one — but the loop returns the first and never reaches the rest.

Impact

The documented Splitting Sources Across KBs workflow has no working
push-trigger path. Affected users must either poll on a short interval or run an
external fan-out calling POST /sync/<name> once per entry.

Suggested fixes

Any of these would resolve it:

Fan out to all matching entries. Collect every entry with webhook: true
matching the repo and schedule _run_entry for each distinct kb-id.
_run_entry already takes a per-KB lock, so overlapping schedules are safe:

def _find_entries_by_source(prefix, match):
    return [e for e in _entries
            if e.get("source", "").startswith(prefix)
            and match in e.get("source", "")
            and e.get("webhook")]

A top-level webhooks: section listing which entries a delivery should
trigger, so the target set is explicit rather than implied by ordering.
Accept a target in the URL — /webhooks/github/<entry-name> — allowing one
webhook per entry. GitHub permits multiple webhooks per repository.

Documentation

The Webhooks section and the Splitting Sources Across KBs section of
docs/guide.md never reference each other. Regardless of which fix is chosen, a
sentence noting that one delivery syncs one knowledge base would save others this
debugging.

Environment

oikb 0.5.0, ghcr.io/open-webui/oikb:latest, daemon mode
Six sources, one GitHub repository, six knowledge bases
Each entry scoped with path:, with target-path: for citation prefixes

Originally created by @etho201 on GitHub (Sep 20, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/127 ### Summary When a single repository is split across multiple knowledge bases — the pattern documented under *Splitting Sources Across KBs* in `docs/guide.md` — a push webhook syncs only **one** of them. The others wait for their scheduled interval. ### Expected A push triggers a sync for every affected source, or at minimum every entry carrying `webhook: true`. ### Actual Exactly one entry syncs. Which one depends on its position in `sources:`. ### Reproduction `.oikb.yaml`: ```yaml defaults: interval: 1h sources: - name: shared source: github:ORG/REPO path: _shared target-path: shared kb-id: <kb-1> webhook: true github_secret: ${GITHUB_WEBHOOK_SECRET} - name: client-a source: github:ORG/REPO path: clients/a target-path: a kb-id: <kb-2> webhook: true # ... four more entries, same source, different kb-id ``` Daemon startup: ``` 6 source(s) configured Scheduling github:ORG/REPO every 3600s (repeated 6×) ``` Push to the repository. One sync runs against `<kb-1>`; the other five knowledge bases stay stale until their interval elapses. Adding `webhook: true` to *every* entry does not change this. ### Cause `src/oikb/webhooks.py` returns on the first match: ```python def _find_entry_by_source(prefix: str, match: str) -> dict | None: for entry in self._entries: source = entry.get("source", "") if source.startswith(prefix) and match in source and entry.get("webhook"): return entry # <-- stops here @router.post("/github") async def github_webhook(request: Request): ... repo = data.get("repository", {}).get("full_name", "") entry = _find_entry_by_source("github:", repo) if entry and _run_entry: asyncio.create_task(_run_entry(entry)) # one entry only ``` `_run_entry_locked` then narrows further to a single knowledge base: ```python entries = [e for e in _entries if e.get("kb-id") == entry["kb-id"]] or [entry] ``` All entries here share the same `source` string, so `match in source` is true for every one — but the loop returns the first and never reaches the rest. ### Impact The documented *Splitting Sources Across KBs* workflow has no working push-trigger path. Affected users must either poll on a short interval or run an external fan-out calling `POST /sync/<name>` once per entry. ### Suggested fixes Any of these would resolve it: 1. **Fan out to all matching entries.** Collect every entry with `webhook: true` matching the repo and schedule `_run_entry` for each distinct `kb-id`. `_run_entry` already takes a per-KB lock, so overlapping schedules are safe: ```python def _find_entries_by_source(prefix, match): return [e for e in _entries if e.get("source", "").startswith(prefix) and match in e.get("source", "") and e.get("webhook")] ``` 2. **A top-level `webhooks:` section** listing which entries a delivery should trigger, so the target set is explicit rather than implied by ordering. 3. **Accept a target in the URL** — `/webhooks/github/<entry-name>` — allowing one webhook per entry. GitHub permits multiple webhooks per repository. ### Documentation The Webhooks section and the *Splitting Sources Across KBs* section of `docs/guide.md` never reference each other. Regardless of which fix is chosen, a sentence noting that one delivery syncs one knowledge base would save others this debugging. ### Environment - oikb 0.5.0, `ghcr.io/open-webui/oikb:latest`, daemon mode - Six sources, one GitHub repository, six knowledge bases - Each entry scoped with `path:`, with `target-path:` for citation prefixes

bsakaguchi referenced this issue

2026-10-18 21:14:20 +00:00

[PR #73] [CLOSED] added working_dir to docker compose #99