open-webui/open-terminal

Fork 0

mirror of https://github.com/open-webui/open-terminal.git synced 2026-11-05 01:27:28 +00:00

[PR #100] Feature: Virtual Desktop & noVNC Integration for GUI-based "Computer Use" capabilities. #124

New issue

Open

opened 2026-10-18 21:11:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:11:01 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-terminal/pull/100
Author: @volkermauel
Created: 3/31/2026
Status: 🔄 Open

Base: main ← Head: desktop

📝 Commits (4)

5f12f9d feat: add virtual desktop & noVNC integration for GUI-based computer use (#44)
eee0abd feat: add human-like randomized delay between keystrokes in desktop type endpoint
b8d74ca fixed chrome not launching
d2bf9b8 feat: add window management, cursor overlay, grounding support for computer use desktop

📊 Changes

6 files changed (+1759 additions, -458 deletions)

View changed files

📝 Dockerfile (+20 -1)
📝 entrypoint.sh (+26 -0)
📝 open_terminal/env.py (+33 -0)
📝 open_terminal/main.py (+779 -457)
➕ open_terminal/utils/desktop.py (+701 -0)
➕ test_desktop.sh (+200 -0)

📄 Description

Summary

Implements #44 — Virtual Desktop & noVNC Integration for GUI-based "Computer Use" capabilities.

Adds an optional virtual desktop environment (Xvfb + x11vnc + noVNC + openbox) that enables programmatic GUI interaction via screenshot, mouse, and keyboard APIs. Feature-gated behind OPEN_TERMINAL_ENABLE_DESKTOP (default false) so existing deployments are unaffected.

Changes

open_terminal/utils/desktop.py (new) — DesktopManager class managing Xvfb, x11vnc, noVNC, and openbox lifecycle, with screenshot (scrot), mouse (xdotool), and keyboard input methods
open_terminal/env.py — Added 5 config vars: ENABLE_DESKTOP, DESKTOP_DISPLAY, DESKTOP_SCREEN_SIZE, DESKTOP_VNC_PORT, DESKTOP_NOVNC_PORT
open_terminal/main.py — 10 new API endpoints, desktop info in system prompt and /api/config
Dockerfile — Added xvfb, x11vnc, novnc, openbox, xdotool, scrot, chromium, fonts; exposed port 6080
entrypoint.sh — DISPLAY setup and stale X11 lock file cleanup when desktop is enabled
test_desktop.sh (new) — 27 integration tests covering full desktop lifecycle

API Endpoints

Method	Path	Description
GET	`/desktop`	Desktop status (running, screen size, ports)
POST	`/desktop/start`	Start virtual desktop
POST	`/desktop/stop`	Stop virtual desktop
POST	`/desktop/screenshot`	Capture PNG screenshot (base64 JSON or raw binary)
POST	`/desktop/click`	Mouse click at (x, y)
POST	`/desktop/mouse_move`	Move cursor to (x, y)
POST	`/desktop/drag`	Mouse drag operation
POST	`/desktop/type`	Type text (with human-like randomized delay by default)
POST	`/desktop/key`	Press key / key combo (e.g. `ctrl+c`)
POST	`/desktop/scroll`	Scroll at position

Usage

docker run -p 8000:8000 -p 6080:6080 \
  -e OPEN_TERMINAL_API_KEY=your-key \
  -e OPEN_TERMINAL_ENABLE_DESKTOP=true \
  open-terminal

API: http://localhost:8000
noVNC web client: http://localhost:6080/vnc.html

Test Results

All 27 integration tests pass against a locally built container.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-terminal/pull/100 **Author:** [@volkermauel](https://github.com/volkermauel) **Created:** 3/31/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `desktop` --- ### 📝 Commits (4) - [`5f12f9d`](https://github.com/open-webui/open-terminal/commit/5f12f9df68cc610588ee292767565f8b11d7f942) feat: add virtual desktop & noVNC integration for GUI-based computer use (#44) - [`eee0abd`](https://github.com/open-webui/open-terminal/commit/eee0abdb1f9dda052cdb6069b91599f6ac56db77) feat: add human-like randomized delay between keystrokes in desktop type endpoint - [`b8d74ca`](https://github.com/open-webui/open-terminal/commit/b8d74ca169d10819a535399d2416d974e5d9e051) fixed chrome not launching - [`d2bf9b8`](https://github.com/open-webui/open-terminal/commit/d2bf9b8680c922aaf81718fb7a516b74829f7517) feat: add window management, cursor overlay, grounding support for computer use desktop ### 📊 Changes **6 files changed** (+1759 additions, -458 deletions) <details> <summary>View changed files</summary> 📝 `Dockerfile` (+20 -1) 📝 `entrypoint.sh` (+26 -0) 📝 `open_terminal/env.py` (+33 -0) 📝 `open_terminal/main.py` (+779 -457) ➕ `open_terminal/utils/desktop.py` (+701 -0) ➕ `test_desktop.sh` (+200 -0) </details> ### 📄 Description ## Summary Implements #44 — Virtual Desktop & noVNC Integration for GUI-based "Computer Use" capabilities. Adds an optional virtual desktop environment (Xvfb + x11vnc + noVNC + openbox) that enables programmatic GUI interaction via screenshot, mouse, and keyboard APIs. Feature-gated behind `OPEN_TERMINAL_ENABLE_DESKTOP` (default `false`) so existing deployments are unaffected. ## Changes - **`open_terminal/utils/desktop.py`** (new) — `DesktopManager` class managing Xvfb, x11vnc, noVNC, and openbox lifecycle, with screenshot (scrot), mouse (xdotool), and keyboard input methods - **`open_terminal/env.py`** — Added 5 config vars: `ENABLE_DESKTOP`, `DESKTOP_DISPLAY`, `DESKTOP_SCREEN_SIZE`, `DESKTOP_VNC_PORT`, `DESKTOP_NOVNC_PORT` - **`open_terminal/main.py`** — 10 new API endpoints, desktop info in system prompt and `/api/config` - **`Dockerfile`** — Added xvfb, x11vnc, novnc, openbox, xdotool, scrot, chromium, fonts; exposed port 6080 - **`entrypoint.sh`** — DISPLAY setup and stale X11 lock file cleanup when desktop is enabled - **`test_desktop.sh`** (new) — 27 integration tests covering full desktop lifecycle ## API Endpoints | Method | Path | Description | |--------|------|-------------| | GET | `/desktop` | Desktop status (running, screen size, ports) | | POST | `/desktop/start` | Start virtual desktop | | POST | `/desktop/stop` | Stop virtual desktop | | POST | `/desktop/screenshot` | Capture PNG screenshot (base64 JSON or raw binary) | | POST | `/desktop/click` | Mouse click at (x, y) | | POST | `/desktop/mouse_move` | Move cursor to (x, y) | | POST | `/desktop/drag` | Mouse drag operation | | POST | `/desktop/type` | Type text (with human-like randomized delay by default) | | POST | `/desktop/key` | Press key / key combo (e.g. `ctrl+c`) | | POST | `/desktop/scroll` | Scroll at position | ## Usage ```bash docker run -p 8000:8000 -p 6080:6080 \ -e OPEN_TERMINAL_API_KEY=your-key \ -e OPEN_TERMINAL_ENABLE_DESKTOP=true \ open-terminal ``` - API: `http://localhost:8000` - noVNC web client: `http://localhost:6080/vnc.html` ## Test Results All 27 integration tests pass against a locally built container. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-18 21:11:01 +00:00

No labels

bug

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/open-terminal#124

No description provided.

Rows
Columns