open-webui/oikb

Fork 0

mirror of https://github.com/open-webui/oikb.git synced 2026-11-05 01:27:25 +00:00

[PR #104] [MERGED] Confluence: keep the text held in macro bodies, skip pages that have none #113

New issue

Closed

opened 2026-10-18 21:14:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/oikb/pull/104
Author: @dakotamccarty
Created: 8/12/2026
Status: ✅ Merged
Merged: 9/16/2026
Merged by: @tjbck

Base: main ← Head: confluence-macro-text

📝 Commits (1)

f492a75 Confluence: keep the text held in macro bodies, skip pages that have none

📊 Changes

1 file changed (+53 additions, -6 deletions)

View changed files

📝 src/oikb/connectors/confluence.py (+53 -6)

📄 Description

Two bugs in the Confluence connector, found while syncing a 321-page space.

1. Macro bodies are deleted

_storage_to_text flattens a page with re.sub(r"<[^>]+>", " ", storage_html). That regex treats <![CDATA[...]]> as one tag and removes it whole, so everything Confluence keeps in a macro body — code blocks, info and note panels, expand sections — never reaches the Knowledge Base.

The page still syncs. It is just quietly missing the part that was usually worth reading. In the space I checked, five pages were affected, the worst losing 1.3KB of a documented HTML snippet.

It also fails in the other direction: when the CDATA content itself contains >, the tag match ends early and the rest leaks into the output as text, along with a stray ]]>.

Macro bodies are now parked before the markup passes and restored after the whitespace pass, so a code block keeps its own line breaks, indentation and angle brackets instead of having half of itself stripped as tags. Block ends become newlines too — output used to collapse onto a single line, which reads as one thought and chunks as one.

>>> _storage_to_text('<p>Paste this:</p>' + code_block('SELECT 1;'))
'Paste this:\nSELECT 1;'      # was 'Paste this:'

Link labels are dropped along with their links: <ac:plain-text-link-body> holds the title of another page in the space, and that page syncs as its own file, so keeping labels turns every section index into a document that is nothing but titles already in the KB.

2. Pages with no text fail with a 400 on every run

That leaves pages which genuinely have no text — a blank page, or one holding only a children-display macro. They extract to zero bytes, and Open WebUI extracts text as part of POST /files/ and answers a bare 400 for a file it can get nothing out of:

Homepage.txt: Client error '400 Bad Request' for url 'https://…/api/v1/files/'
Pricing.txt: Client error '400 Bad Request' for url 'https://…/api/v1/files/'
… 18 more

Twenty of the 321 pages were section indexes and archive stubs like this. Nothing about that resolves itself, so the space could never sync clean. read_file now raises SourceFileUnavailable for them, which run_sync already reports as a warning rather than an error.

Validation

Both extractors run over all 321 pages of a real space:

empty before: 20   empty after: 20   (same set — the index stubs)
pages that gained text: 5
  +58 words — an HTML email signature in a code block
   +6 words — an info panel
   +2 words — a note panel
   +2 words — a note panel
   +1 words — an expand section

No page lost anything but the stray ]]> fragments the old regex leaked. This has been running in production against that space since today.

One thing I left alone

A Confluence checksum is built from the page version rather than the page text, so a space already synced picks up the better extraction only as pages are edited. Salting the manifest with an extractor version fixes that immediately — it is what I did downstream — but it re-uploads every page of every Confluence space on the first sync after upgrading, which felt like a maintainer's call rather than something to slip into this patch. Happy to add it if you want it.

There is no test suite in the repo, so I have not added one; glad to if you would like tests for this.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/oikb/pull/104 **Author:** [@dakotamccarty](https://github.com/dakotamccarty) **Created:** 8/12/2026 **Status:** ✅ Merged **Merged:** 9/16/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `main` ← **Head:** `confluence-macro-text` --- ### 📝 Commits (1) - [`f492a75`](https://github.com/open-webui/oikb/commit/f492a75807424972bc655cf4340b75ff499800fa) Confluence: keep the text held in macro bodies, skip pages that have none ### 📊 Changes **1 file changed** (+53 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `src/oikb/connectors/confluence.py` (+53 -6) </details> ### 📄 Description Two bugs in the Confluence connector, found while syncing a 321-page space. ## 1. Macro bodies are deleted `_storage_to_text` flattens a page with `re.sub(r"<[^>]+>", " ", storage_html)`. That regex treats `<![CDATA[...]]>` as one tag and removes it whole, so everything Confluence keeps in a macro body — code blocks, info and note panels, expand sections — never reaches the Knowledge Base. The page still syncs. It is just quietly missing the part that was usually worth reading. In the space I checked, five pages were affected, the worst losing 1.3KB of a documented HTML snippet. It also fails in the other direction: when the CDATA content itself contains `>`, the tag match ends early and the rest leaks into the output as text, along with a stray `]]>`. Macro bodies are now parked before the markup passes and restored after the whitespace pass, so a code block keeps its own line breaks, indentation and angle brackets instead of having half of itself stripped as tags. Block ends become newlines too — output used to collapse onto a single line, which reads as one thought and chunks as one. ```python >>> _storage_to_text('<p>Paste this:</p>' + code_block('SELECT 1;')) 'Paste this:\nSELECT 1;' # was 'Paste this:' ``` Link labels are dropped along with their links: `<ac:plain-text-link-body>` holds the title of another page in the space, and that page syncs as its own file, so keeping labels turns every section index into a document that is nothing but titles already in the KB. ## 2. Pages with no text fail with a 400 on every run That leaves pages which genuinely have no text — a blank page, or one holding only a children-display macro. They extract to zero bytes, and Open WebUI extracts text as part of `POST /files/` and answers a bare 400 for a file it can get nothing out of: ``` Homepage.txt: Client error '400 Bad Request' for url 'https://…/api/v1/files/' Pricing.txt: Client error '400 Bad Request' for url 'https://…/api/v1/files/' … 18 more ``` Twenty of the 321 pages were section indexes and archive stubs like this. Nothing about that resolves itself, so the space could never sync clean. `read_file` now raises `SourceFileUnavailable` for them, which `run_sync` already reports as a warning rather than an error. ## Validation Both extractors run over all 321 pages of a real space: ``` empty before: 20 empty after: 20 (same set — the index stubs) pages that gained text: 5 +58 words — an HTML email signature in a code block +6 words — an info panel +2 words — a note panel +2 words — a note panel +1 words — an expand section ``` No page lost anything but the stray `]]>` fragments the old regex leaked. This has been running in production against that space since today. ## One thing I left alone A Confluence checksum is built from the page version rather than the page text, so a space already synced picks up the better extraction only as pages are edited. Salting the manifest with an extractor version fixes that immediately — it is what I did downstream — but it re-uploads every page of every Confluence space on the first sync after upgrading, which felt like a maintainer's call rather than something to slip into this patch. Happy to add it if you want it. There is no test suite in the repo, so I have not added one; glad to if you would like tests for this. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:14:28 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#113

No description provided.

Rows
Columns