[PR #114] Feature: Use document extractors for grep and centralize file text extraction #125

New issue

Open

opened 2026-10-18 21:11:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:11:01 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-terminal/pull/114
Author: @pfurovYnP
Created: 4/21/2026
Status: 🔄 Open

Base: main ← Head: main

📝 Commits (5)

c6f860b Refactor shared document extraction logic for read and grep
bcb7593 Merge pull request #1 from pfurovYnP/codex/update-grep-to-use-text-representations
c1b3ffa update doc for grep
b0812cf Avoid executor starvation for PTY reads
f5e5bc8 Merge pull request #3 from pfurovYnP/codex/investigate-asynchronous-operations-for-terminal

📊 Changes

3 files changed (+177 additions, -53 deletions)

View changed files

📝 open_terminal/main.py (+91 -48)
📝 open_terminal/utils/runner.py (+48 -5)
➕ tests/test_runner.py (+38 -0)

📄 Description

(Codex)

#113

Motivation

Ensure the same document text extraction logic used by read_file is also applied when searching files with grep_search so searchable content from PDFs/Office files is included.
Reduce duplication by centralizing the extraction logic into helper functions.
Improve error handling for binary/unsupported files in the grep flow to avoid reading raw bytes as text.

Description

Introduced _extract_text_with_supported_document_extractors(file_path, mime) to encapsulate lookup and invocation of EXTRACTORS from open_terminal.utils.documents and return extracted text or None.
Added _read_file_as_text_representation_for_grep(file_path) which mirrors read_file behavior: attempt UTF-8 read first, fall back to the document extractors, and raise UnicodeDecodeError for unsupported binary files.
Refactored read_file to call the new extractor helper and simplified the previous inline extractor loop.
Updated grep_search to use _read_file_as_text_representation_for_grep so document-extracted text is searched line-by-line, preserving existing match and truncation behaviour.

Testing

Ran the automated test suite with pytest, and all tests completed successfully.
Exercised read_file and grep_search behavior manually against text files and PDF/Office files to verify extracted text is returned and searchable as expected.
Verified that binary files with allowed MIME prefixes are still returned as raw and unsupported binaries are rejected during search.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-terminal/pull/114 **Author:** [@pfurovYnP](https://github.com/pfurovYnP) **Created:** 4/21/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (5) - [`c6f860b`](https://github.com/open-webui/open-terminal/commit/c6f860b4ad7931478eacaac4d9ebff23d5893504) Refactor shared document extraction logic for read and grep - [`bcb7593`](https://github.com/open-webui/open-terminal/commit/bcb7593f2340e1523a23c04597eb1e8559e49e22) Merge pull request #1 from pfurovYnP/codex/update-grep-to-use-text-representations - [`c1b3ffa`](https://github.com/open-webui/open-terminal/commit/c1b3ffa5903a6acf9b7ee43bd5da50e1dadb0a95) update doc for grep - [`b0812cf`](https://github.com/open-webui/open-terminal/commit/b0812cff34c5feb07c5726d53087dbfac5e65a74) Avoid executor starvation for PTY reads - [`f5e5bc8`](https://github.com/open-webui/open-terminal/commit/f5e5bc8fa0b1dbf7f7ae5c138d1119690c9f0e7a) Merge pull request #3 from pfurovYnP/codex/investigate-asynchronous-operations-for-terminal ### 📊 Changes **3 files changed** (+177 additions, -53 deletions) <details> <summary>View changed files</summary> 📝 `open_terminal/main.py` (+91 -48) 📝 `open_terminal/utils/runner.py` (+48 -5) ➕ `tests/test_runner.py` (+38 -0) </details> ### 📄 Description (Codex) #113 ### Motivation - Ensure the same document text extraction logic used by `read_file` is also applied when searching files with `grep_search` so searchable content from PDFs/Office files is included. - Reduce duplication by centralizing the extraction logic into helper functions. - Improve error handling for binary/unsupported files in the grep flow to avoid reading raw bytes as text. ### Description - Introduced `_extract_text_with_supported_document_extractors(file_path, mime)` to encapsulate lookup and invocation of `EXTRACTORS` from `open_terminal.utils.documents` and return extracted text or `None`. - Added `_read_file_as_text_representation_for_grep(file_path)` which mirrors `read_file` behavior: attempt UTF-8 read first, fall back to the document extractors, and raise `UnicodeDecodeError` for unsupported binary files. - Refactored `read_file` to call the new extractor helper and simplified the previous inline extractor loop. - Updated `grep_search` to use `_read_file_as_text_representation_for_grep` so document-extracted text is searched line-by-line, preserving existing match and truncation behaviour. ### Testing - Ran the automated test suite with `pytest`, and all tests completed successfully. - Exercised `read_file` and `grep_search` behavior manually against text files and PDF/Office files to verify extracted text is returned and searchable as expected. - Verified that binary files with allowed MIME prefixes are still returned as raw and unsupported binaries are rejected during search. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-18 21:11:01 +00:00

No labels

bug

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/open-terminal#125

No description provided.

Rows
Columns