[GH-ISSUE #28] feat: custom web RAG with Firecrawl #23

New issue

Closed

opened 2026-10-18 21:01:38 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-18 21:01:38 +00:00

Owner

Originally created by @justinh-rahb on GitHub (May 31, 2024).
Original GitHub issue: https://github.com/open-webui/pipelines/issues/28

Firecrawl is highly suitable for custom web Retrieval-Augmented Generation (RAG) pipelines due to its advanced features and flexibility. Here are the key highlights:

Smart LLM Scraping: Converts websites into clean, LLM-ready markdown, ensuring data is optimized for language models which improves the accuracy and relevance of RAG outputs.
Comprehensive Capabilities: Efficiently handles JavaScript-heavy sites and complex site navigations for thorough data retrieval, critical for feeding accurate information into RAG systems.
Self-Hostable and Scalable: Offers both a hosted version and a self-hostable option, giving you control over deployment and scalability, which is ideal for handling large-scale data needs in RAG pipelines.
Efficient and Cost-Effective: Noted for significant savings in computational costs and time, enhancing operational efficiency in large-scale deployments.

This combination of smart scraping, comprehensive data handling, scalability, and cost-efficiency makes Firecrawl an excellent choice for enhancing custom RAG pipelines
Firecrawl.dev | Introduction | Scrape Docs

Originally created by @justinh-rahb on GitHub (May 31, 2024). Original GitHub issue: https://github.com/open-webui/pipelines/issues/28 Firecrawl is highly suitable for custom web Retrieval-Augmented Generation (RAG) pipelines due to its advanced features and flexibility. Here are the key highlights: 1. **Smart LLM Scraping**: Converts websites into clean, LLM-ready markdown, ensuring data is optimized for language models which improves the accuracy and relevance of RAG outputs. 2. **Comprehensive Capabilities**: Efficiently handles JavaScript-heavy sites and complex site navigations for thorough data retrieval, critical for feeding accurate information into RAG systems. 3. **Self-Hostable and Scalable**: Offers both a hosted version and a self-hostable option, giving you control over deployment and scalability, which is ideal for handling large-scale data needs in RAG pipelines. 4. **Efficient and Cost-Effective**: Noted for significant savings in computational costs and time, enhancing operational efficiency in large-scale deployments. This combination of smart scraping, comprehensive data handling, scalability, and cost-efficiency makes Firecrawl an excellent choice for enhancing custom RAG pipelines [Firecrawl.dev](https://www.firecrawl.dev/) | [Introduction](https://docs.firecrawl.dev/introduction) | [Scrape Docs](https://docs.firecrawl.dev/features/scrape)

bsakaguchi closed this issue

2026-10-18 21:01:38 +00:00

bsakaguchi commented

2026-10-18 21:01:40 +00:00

Author

Owner

@tjbck commented on GitHub (May 31, 2024):

I guess the issue here is knowing when to search the web, one solution here is asking the users to prefix their prompt with a designated command (e.g. !search what's open webui?)

@tjbck commented on GitHub (May 31, 2024): I guess the issue here is knowing when to search the web, one solution here is asking the users to prefix their prompt with a designated command (e.g. `!search what's open webui?`)

bsakaguchi commented

2026-10-18 21:01:40 +00:00

Author

Owner

@bannert1337 commented on GitHub (Jun 10, 2024):

Why not as a tool the LLM can call whenever it wants/needs it.

@bannert1337 commented on GitHub (Jun 10, 2024): Why not as a tool the LLM can call whenever it wants/needs it.

bsakaguchi commented

2026-10-18 21:01:40 +00:00

Author

Owner

@atgehrhardt commented on GitHub (Jun 18, 2024):

One issue I can think of with it being a tool is that it's going to be really hit or miss. This would be so useful, that while a tool implementation would be cool, smaller models would really struggle, so I think more people would get benefit out of being able to invoke it explicitly.

It can always be implemented both ways, but while the tool would be cooler, it also would be less applicable across the board.

@atgehrhardt commented on GitHub (Jun 18, 2024): One issue I can think of with it being a tool is that it's going to be really hit or miss. This would be so useful, that while a tool implementation would be cool, smaller models would really struggle, so I think more people would get benefit out of being able to invoke it explicitly. It can always be implemented both ways, but while the tool would be cooler, it also would be less applicable across the board.