[GH-ISSUE #132] feat: allow webscraper to remove certain HTML content #78

New issue

Open

opened 2026-10-18 21:14:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:14:08 +00:00

Owner

Originally created by @guenhter on GitHub (Sep 28, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/132

We have a wiki which should be scraped, but the content of each page is more then 90% boilerplate useless stuff which comes from the wiki like sidebar (which is huge as the wiki is huge), footer, ... All those elements are worthless to rag as they are redundant on every site, but they ruin rag as it not longer fetches the relevant parts any more.
What we have in a custom scraper is the concept of "transformers". When a page gets downloaded and before it gets inserted into the knowledge base, the page runs through transformers. On of the transformers is the HtmlContentRemover which kicks out elements based on css selectors:

transformer:
  type: HtmlContentRemover
  params:
    selectors:
      - "#menu-top"
      - ".menu-side"
      - "footer"
      - "#bottom"

This drastically improved our data quality.

Of course, this HtmlContentRemover must be configured individually as other wikis/pages have other selectors.

It would be very helpful if such a transformer architecture also would be possible for oikb and a sort of HtmlContentRemover can be configured to remove certain parts...

Originally created by @guenhter on GitHub (Sep 28, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/132 We have a wiki which should be scraped, but the content of each page is more then 90% boilerplate useless stuff which comes from the wiki like sidebar (which is huge as the wiki is huge), footer, ... All those elements are worthless to rag as they are redundant on every site, but they ruin rag as it not longer fetches the relevant parts any more. What we have in a custom scraper is the concept of "transformers". When a page gets downloaded and before it gets inserted into the knowledge base, the page runs through transformers. On of the transformers is the `HtmlContentRemover` which kicks out elements based on css selectors: ``` transformer: type: HtmlContentRemover params: selectors: - "#menu-top" - ".menu-side" - "footer" - "#bottom" ``` This drastically improved our data quality. Of course, this `HtmlContentRemover` must be configured individually as other wikis/pages have other selectors. It would be very helpful if such a transformer architecture also would be possible for oikb and a sort of `HtmlContentRemover` can be configured to remove certain parts...

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/oikb#78

No description provided.

Rows
Columns