[PR #3] [MERGED] Updated cleanup_pg.py #3

New issue

Closed

opened 2026-10-18 21:09:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:09:22 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/scripts/pull/3
Author: @Ithanil
Created: 2/5/2026
Status: ✅ Merged
Merged: 2/23/2026
Merged by: @tjbck

Base: main ← Head: update_cleanup

📝 Commits (1)

d1fe255 updated cleanup_pg.py with \u0020 sanitization and check for files used in memory&folders

📊 Changes

1 file changed (+92 additions, -12 deletions)

View changed files

📝 scripts/cleanup_pg/cleanup_pg.py (+92 -12)

📄 Description

This Pull Request improves the cleanup_pg.py script by introducing automated chat content sanitization and expanding the scope of the database cleanup to ensure no orphaned files or collections remain after deletion.

Key Changes:

Chat Sanitization: Added a new --sanitize flag and sanitize_chats function to replace null characters (\u0000) with spaces (\u0020) in the chat JSONB column to prevent potential database processing errors.
Expanded File Reference Detection: Updated find_referenced_files to search across more tables and paths, including:
- chat_file and knowledge_file relational tables.
- File references nested within the folder table's metadata.
- Additional JSON paths in the chat table ($.files.id).
Improved Chat Deletion: The cleanup_chats function now performs a cascading-style cleanup by removing entries in the chat_file table associated with deleted chats.
Enhanced Collection Tracking: Refined find_referenced_collections to better identify collections used in user memory and new file attachment structures.
General Improvements: Added a Python shebang for direct execution and updated logging and dry-run logic for better transparency during the cleanup process.

Notes:

Missing: Channels are still ignored, so they are neither cleaned up nor are there any checks for files/collection referenced in channels.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/scripts/pull/3 **Author:** [@Ithanil](https://github.com/Ithanil) **Created:** 2/5/2026 **Status:** ✅ Merged **Merged:** 2/23/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `main` ← **Head:** `update_cleanup` --- ### 📝 Commits (1) - [`d1fe255`](https://github.com/open-webui/scripts/commit/d1fe255c5b7ed39a4eaff8a3065fa62c605d2564) updated cleanup_pg.py with \u0020 sanitization and check for files used in memory&folders ### 📊 Changes **1 file changed** (+92 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `scripts/cleanup_pg/cleanup_pg.py` (+92 -12) </details> ### 📄 Description This Pull Request improves the `cleanup_pg.py` script by introducing automated chat content sanitization and expanding the scope of the database cleanup to ensure no orphaned files or collections remain after deletion. ### Key Changes: * **Chat Sanitization:** Added a new `--sanitize` flag and `sanitize_chats` function to replace null characters (`\u0000`) with spaces (`\u0020`) in the chat JSONB column to prevent potential database processing errors. * **Expanded File Reference Detection:** Updated `find_referenced_files` to search across more tables and paths, including: * `chat_file` and `knowledge_file` relational tables. * File references nested within the `folder` table's metadata. * Additional JSON paths in the `chat` table (`$.files.id`). * **Improved Chat Deletion:** The `cleanup_chats` function now performs a cascading-style cleanup by removing entries in the `chat_file` table associated with deleted chats. * **Enhanced Collection Tracking:** Refined `find_referenced_collections` to better identify collections used in user memory and new file attachment structures. * **General Improvements:** Added a Python shebang for direct execution and updated logging and dry-run logic for better transparency during the cleanup process. ### Notes: * **Missing**: **Channels** are still ignored, so they are neither cleaned up nor are there any checks for files/collection referenced in channels. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:09:22 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/scripts#3

No description provided.

Rows
Columns