Agent Desktop
A desktop GUI automation CLI that enables AI agents to control native applications.
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: A desktop GUI automation CLI that enables AI agents to control native applications. Full Skill content: https://321skill.com/skills/agent-desktop-x-5/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the challenge of AI agents being unable to directly interact with desktop GUI applications. In practical development, AI agents need to click buttons, fill out forms, navigate menus, read UI states, take screenshots, etc. Traditional methods rely on image recognition or coordinate positioning, which are unstable and susceptible to environmental changes. Agent Desktop retrieves structured UI information from the native OS accessibility tree, allowing AI agents to control desktop applications with human-like precision.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and grant it Accessibility and Screen Recording permissions. You can then instruct your AI agent using natural language commands, such as snapshot --skeleton to get the app skeleton, click @e12 to click an element, or type @e5 to input text. It employs a progressive skeleton traversal strategy, first obtaining a UI overview before drilling down into target areas, significantly reducing token consumption.
It is ideal for teams or individuals needing to integrate AI agents into desktop workflows. Developers already using AI coding assistants like Claude or Cursor can seamlessly add desktop automation capabilities, enabling end-to-end automation from code to GUI operations. Test engineers can use it to write GUI test scripts, while operations engineers can automate desktop maintenance tasks.
It is recommended for use on macOS 12+ systems, ensuring the terminal has been granted Accessibility and Screen Recording permissions. Note that it currently only supports the macOS Phase 1 adapter; Windows and Linux versions are in the planning stages. For simple applications (e.g., Calculator, TextEdit), a full snapshot can be used directly. For complex applications, the skeleton traversal strategy is recommended to save tokens.
Key Features
Unlike tools such as PyAutoGUI or SikuliX, which rely on image recognition or coordinate positioning, Agent Desktop retrieves structured UI elements (with ref identifiers) from the native OS accessibility tree, outputting JSON instead of pixel coordinates. This makes it immune to changes in screen resolution, themes, or element position offsets, resulting in higher stability.
Limitations
Currently only supports macOS 12+ and requires Accessibility and Screen Recording permissions. Windows and Linux versions are in the planning stages. Relies on the OS's native accessibility APIs, so some non-standard controls may not be recognized.
FAQ
How do I install Agent Desktop?
Install globally via npm: `npm install -g agent-desktop`, or via bun: `bun install -g --trust agent-desktop`. A Node.js environment is required.
What system permissions are required?
Requires macOS 12+. In System Preferences, grant your terminal Accessibility permissions (for UI control) and Screen Recording permissions (for screenshots).
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-5/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-5/raw/index.md