Agent Desktop
A desktop GUI automation CLI tool for AI agents to invoke.
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: A desktop GUI automation CLI tool for AI agents to invoke. Full Skill content: https://321skill.com/skills/agent-desktop-x-9/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the issue where AI agents cannot directly observe and interact with desktop applications. In practical development, developers often need AI to automatically perform desktop interaction tasks such as clicking buttons, filling forms, navigating menus, and reading UI states. However, traditional solutions either rely on screenshots + coordinates (fragile and prone to failure) or require complex scripting. agent-desktop retrieves structured UI information via the native OS accessibility tree and outputs JSON with ref identifiers, enabling AI to precisely locate and manipulate interface elements.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and grant it Accessibility and Screen Recording permissions. You can then use natural language instructions to have your AI agent invoke 58 commands like snapshot, click, type, and select to observe and operate desktop applications. We recommend adopting a "progressive skeleton traversal" strategy: first obtain an application skeleton overview (--skeleton), then drill down into target areas (--root), and finally execute actions and verify state changes. This can reduce token consumption by 78-96%.
It is well-suited for teams or individuals who need to extend AI agent capabilities to desktop GUIs. This is especially useful for developers already using AI coding assistants like Claude, Cursor, or Trae, who want these agents to automate cross-application operations (e.g., copying data from a browser to Excel, auto-filling OA forms, managing multi-window tasks). Test engineers can use it to write automation test scripts, and operations staff can use it to monitor desktop application states.
It is recommended to use this tool uniformly on macOS 12+ and to pre-enable "Accessibility" and "Screen Recording" permissions for the terminal in System Preferences. Note that agent-desktop itself is not an AI agent; it is merely a tool—the observe-act loop is controlled by the AI agent that invokes it. Currently, only the macOS Phase 1 adapter is supported; Windows and Linux versions are in the planning stages.
Key Features
Unlike screenshot+coordinate-based desktop automation tools like PyAutoGUI or SikuliX, agent-desktop retrieves structured UI elements (with ref identifiers) via the native OS accessibility tree, outputting JSON instead of pixel coordinates. This allows AI agents to stably locate elements, unaffected by window position or resolution changes.
Limitations
Currently only supports macOS 12+ and requires the terminal to have Accessibility and Screen Recording permissions; Windows and Linux adapters are still in the planning stages.
FAQ
Which operating systems are supported?
Currently, only macOS 12+ (Phase 1 adapter) is supported. Windows and Linux versions are under development.
What system permissions are required?
You need to enable "Accessibility" permission (for accessibility tree access) and "Screen Recording" permission (for screenshots) for the terminal in System Preferences.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-9/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-9/raw/index.md