Linux Desktop GUI Automation

Automate any X11 Linux desktop: capture screens, find and click elements, type, use hotkeys, manage windows.

Preferred screenshot interpretation path: capture with capture.sh and interpret the image directly in your OpenClaw chat (existing image-capable model connection).

Prerequisites

• X11 session running (XFCE, GNOME on X11, KDE on X11, i3, openbox, etc.)
• DISPLAY environment variable set (usually :0)
• Run bash install.sh once to install dependencies
• No extra key needed for screenshot interpretation when using OpenClaw's image-capable chat path

Quick Reference

Typical Automation Workflow

For most GUI automation tasks, follow this pattern:

1. Capture a screenshot with capture.sh — note the file path printed
2. Look at the screenshot yourself to understand what's on screen
3. Find the target element by examining the screenshot and estimating its pixel coordinates
4. Act using the coordinates: click.sh --x X --y Y
5. Verify by capturing another screenshot and checking the result

Example: Click the Save button in a dialog

# Step 1: Capture the screen
SCREENSHOT=$(bash capture.sh | tail -1)

# Step 2: Look at the screenshot (read the image file with your vision)
# Examine the image and identify the Save button's position

# Step 3: Click at the coordinates you identified
bash click.sh --x 450 --y 320

Example: Type into a specific application

# Focus the terminal window and type a command
bash type.sh --window "Terminal" "ls -la"
bash hotkey.sh "Return"

Example: Window management

# Maximize Firefox, then focus a terminal
bash window.sh --action maximize --window "Firefox"
bash window.sh --action focus --window "Terminal"

JSON Output

All tools support a --json flag for machine-readable output:

{"success": true, "output": "...", "error": null}

On failure:

{"success": false, "output": null, "error": "Error description"}

Environment Setup

If DISPLAY is not set (e.g., running over SSH), set it before calling any tool:

export DISPLAY=:0

For headless servers with a virtual display:

Xvfb :0 -screen 0 1920x1080x24 &
export DISPLAY=:0

Hotkey Reference

Key names follow X11 conventions:

Combine with +: ctrl+c, ctrl+shift+t, alt+F4, super+d

Limitations

• X11 only — does not work on Wayland sessions
• Cannot interact with Wayland-native apps in a Wayland session
• Some apps with custom rendering (games, Electron apps with security flags) may resist automation
• Screenshot quality depends on compositor; disable compositing if captures look wrong