Agent Desktop
AI Agent Desktop Automation CLI Tool
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: AI Agent Desktop Automation CLI Tool Full Skill content: https://321skill.com/skills/agent-desktop-x/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This skill addresses the problem of AI agents being unable to directly observe and operate desktop applications. In practical development, developers often need AI agents to automate desktop GUI interaction tasks such as clicking buttons, filling out forms, navigating menus, and reading UI states. However, traditional methods either rely on screenshot OCR (low accuracy, high token consumption) or require writing complex scripts. agent-desktop directly retrieves structured UI information from the native OS accessibility tree, enabling AI agents to 'see' and 'operate' the desktop like a human.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and grant it Accessibility and Screen Recording permissions. You can then use natural language instructions to have the AI agent perform desktop operations, such as 'Open Calculator and calculate 123+456'. The AI agent will first call agent-desktop snapshot --skeleton to obtain the desktop UI skeleton, identify the target area, then use snapshot --root @e3 to drill down into that area, and finally execute operations like click and type. The entire process employs a progressive skeleton traversal strategy, which can reduce token consumption by 78-96% compared to full screenshots.
It is well-suited for teams or individuals who need AI agents to automate desktop applications. This is especially true for developers already using AI coding assistants like Claude or Cursor and looking to extend their capabilities to desktop GUI operations. Test engineers can use it to automate desktop software testing, operations engineers can use it to manage local applications, and agent developers can use it as the 'hands' and 'eyes' of an AI Agent for end-to-end desktop task automation.
It is recommended to use this tool uniformly when AI agents need to operate desktop applications. Note that it currently only supports macOS 12+ (Phase 1 adapter), with Windows and Linux support planned. Before first use, you must enable Accessibility permissions for the terminal (System Settings → Privacy & Security → Accessibility), and the screenshot feature also requires Screen Recording permissions. Additionally, agent-desktop itself is not an AI agent; it is merely a tool, and the AI agent calling it must maintain the observation-action loop.
Key Features
Unlike GUI automation tools like PyAutoGUI, which rely on coordinates and image recognition, agent-desktop retrieves structured UI elements (including ref identifiers) from the native OS accessibility tree, outputting JSON instead of screenshots. AI agents can directly parse the element hierarchy and attributes, eliminating the need for OCR or coordinate guessing. Furthermore, the progressive skeleton traversal strategy significantly reduces token consumption.
Limitations
Currently only supports macOS 12+, requires the terminal to have Accessibility and Screen Recording permissions. Windows and Linux adapters are not yet released.
FAQ
After installation, it fails to run, prompting insufficient permissions. What should I do?
You need to check your terminal application in System Settings → Privacy & Security → Accessibility. The screenshot feature also requires authorization in Screen Recording. After granting permissions, reopen the terminal.
Which operating systems are supported?
Currently only macOS 12+ is supported (Phase 1 adapter). Windows and Linux support is planned, with the core interface already designed uniformly.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x/raw/index.md