[GH-ISSUE #45] Rate limit exceeded for Github #28

New issue

Closed

opened 2026-10-18 21:13:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:36 +00:00

Owner

Originally created by @zyberwoof on GitHub (Jun 8, 2026).
Original GitHub issue: https://github.com/open-webui/oikb/issues/45

EDIT: Based on Options 1 and 2 above being handled, I believe that this is no longer an issue. Option 3 would be a "nice-to-have" feature, but not a necessity.

Problem

GitHub rate limits unauthenticated users to 60 API requests per hour. This can be seen with 403 and 429 errors in log files.

Example

Synchronize Open WebUI's docs from GitHub to create a RAG based on their own documentation. When trying to do this, I quickly hit the rate-limit for Github. It appears that GitHub limits unauthenticated users to 60 REST API request per hour.

This can be reproduced by adding the source source: github:open-webui/docs. I also used an include filter of ["docs/**/*.mdx"].

Solutions

Below are a handful of solution paths I've come up with. Any one of these should alleviate the original problem for GitHub. But all 3 could be implemented and provide value.

Option 1 - Modify the current method

~~Update the GitHub connector to be more efficient. There may be a smarter or more efficient method of retrieving files than what is used now. I suspect it is doing an API call for each file.~~

EDIT: It appears that the current method only generates a large number of API calls when the number of new or modified files is large. For most repositories, this may be the case. This means that after the initial sync, updates should be quick and have minimal API calls. And when a rate-limit is exceeded, subsequent schedules syncs would slowly chip away at remaining files.

If this is the case, I believe that the connector in its current state is handling things efficiently.

Option 2 - Authenticate the user

~~Provide a way for the GitHub connector to allow users to pull content based on their own credentials. It appears that authenticated users are bumped up to a rate limit of 5000 requests per hour.~~

EDIT: This is available today. Setting the environment variable GITHUB_TOKEN allows for authorized API requests.

Login with your GitHub account in a browser.
Navigate to page to Github token settings
Choose to generate a classic token.
Give the token the public_repo permission.
Use this token as oikb's GITHUB_TOKEN environment variable.

Option 3 - Add a global rate limit feature

In addition to interval and concurrency, add the ability for users to set default and source-specific rate limits. This would not only help mitigate this issue for GitHub, but it would also provide users the ability to mitigate the issue for other services.

Originally created by @zyberwoof on GitHub (Jun 8, 2026). Original GitHub issue: https://github.com/open-webui/oikb/issues/45 **EDIT: Based on Options 1 and 2 above being handled, I believe that this is no longer an issue. Option 3 would be a "nice-to-have" feature, but not a necessity.** # Problem GitHub rate limits unauthenticated users to 60 API requests per hour. This can be seen with 403 and 429 errors in log files. # Example Synchronize Open WebUI's docs from GitHub to create a RAG based on their own documentation. When trying to do this, I quickly hit the rate-limit for Github. It appears that GitHub limits [unauthenticated users](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2026-03-10#primary-rate-limit-for-unauthenticated-users) to 60 REST API request per hour. This can be reproduced by adding the source `source: github:open-webui/docs`. I also used an include filter of `["docs/**/*.mdx"]`. # Solutions Below are a handful of solution paths I've come up with. Any one of these should alleviate the original problem for GitHub. But all 3 could be implemented and provide value. ## Option 1 - Modify the current method ~~Update the GitHub connector to be more efficient. There may be a smarter or more efficient method of retrieving files than what is used now. I suspect it is doing an API call for each file.~~ EDIT: It appears that the current method only generates a large number of API calls when the number of new or modified files is large. For most repositories, this may be the case. This means that after the initial sync, updates should be quick and have minimal API calls. And when a rate-limit is exceeded, subsequent schedules syncs would slowly chip away at remaining files. If this is the case, I believe that the connector in its current state is handling things efficiently. ## Option 2 - Authenticate the user ~~Provide a way for the GitHub connector to allow users to pull content based on their own credentials. It appears that authenticated users are bumped up to a rate limit of 5000 requests per hour.~~ EDIT: This is available today. Setting the environment variable GITHUB_TOKEN allows for authorized API requests. 1. Login with your GitHub account in a browser. 2. Navigate to page to [Github token settings](https://github.com/settings/tokens) 3. Choose to generate a classic token. 4. Give the token the `public_repo` permission. 5. Use this token as oikb's GITHUB_TOKEN environment variable. ## Option 3 - Add a global rate limit feature In addition to interval and concurrency, add the ability for users to set default and source-specific rate limits. This would not only help mitigate this issue for GitHub, but it would also provide users the ability to mitigate the issue for other services.

bsakaguchi closed this issue

2026-10-18 21:13:37 +00:00

bsakaguchi referenced this issue

2026-10-18 21:14:15 +00:00

[PR #28] Add supporting confluence api v1 and authorization by bearer token #86