Agent Desktop
A desktop automation CLI that controls GUI via the accessibility tree.
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: A desktop automation CLI that controls GUI via the accessibility tree. Full Skill content: https://321skill.com/skills/agent-desktop-x-3/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the pain point where AI agents cannot directly interact with desktop GUIs. In practical development, developers often need AI to automate desktop interaction tasks such as clicking buttons, filling forms, navigating menus, and reading UI states. However, traditional coordinate-based methods are unstable and susceptible to resolution changes. agent-desktop leverages the native OS accessibility tree to output structured JSON, enabling AI agents to reliably observe and operate desktop applications.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and ensure macOS has granted Accessibility and Screen Recording permissions. You can then invoke 58 commands like snapshot, click, type, and select using natural language instructions. A progressive skeleton traversal strategy is recommended: first use snapshot --skeleton to get an overview of the application skeleton, then drill down into specific areas using the --root parameter, perform the action, and finally re-drill to verify state changes. This process can reduce token consumption by 78-96%.
It is ideal for agent developers, test engineers, and DevOps engineers who need desktop GUI automation—particularly teams or individuals who have already built AI agent workflows and need their agents to directly operate local desktop applications (e.g., filling out OA systems, automating desktop software testing, batch processing file dialogs). Currently, it supports the macOS Phase 1 adapter; Windows and Linux versions are in the planning stages.
We recommend using this tool consistently for scenarios requiring stable and reliable desktop automation. Note that it primarily relies on macOS's native accessibility APIs, so you must ensure your terminal app is authorized in System Preferences → Privacy & Security → Accessibility. Screenshot functionality also requires Screen Recording permission. agent-desktop itself is not an AI agent but a tool for AI agents to call; the observe-act loop is controlled by the caller.
Key Features
Unlike automation tools such as PyAutoGUI or SikuliX that rely on coordinates or image recognition, agent-desktop outputs structured JSON (with ref identifiers) based on the native OS accessibility tree. It is unaffected by screen resolution, window position, or visual changes, offering higher stability. Designed specifically for AI agents, it supports a progressive skeleton traversal strategy that significantly reduces token consumption.
Limitations
Currently only supports macOS 12+ and requires Accessibility and Screen Recording permissions; Windows and Linux adaptations are not yet complete. Requires global npm installation and a Node.js environment.
FAQ
Does it support Windows or Linux?
Currently, only the macOS Phase 1 adapter is supported. Windows and Linux versions are planned; the core contract is defined but not yet implemented.
What system permissions are required?
Requires macOS Accessibility permissions (to control other apps) and Screen Recording permissions (for screenshot functionality). You must authorize your terminal app in System Preferences.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-3/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-3/raw/index.md