Agent Desktop
A desktop automation CLI tool based on the system accessibility tree
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: A desktop automation CLI tool based on the system accessibility tree Full Skill content: https://321skill.com/skills/agent-desktop-x-4/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the issue of AI agents being unable to directly control desktop applications. In practical development, AI coding assistants or agents need to perform actions like clicking buttons, filling out forms, reading UI states, and capturing screens. However, traditional solutions either rely on coordinate-based positioning (fragile and prone to failure) or require code injection (high privileges, significant risks). Agent Desktop retrieves application interface structures via the native OS accessibility tree, enabling AI to 'see' and operate the desktop like a human.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and grant it Accessibility and Screen Recording permissions. You can then instruct the AI using natural language commands to invoke it. For example, with the instruction 'Open the calculator and calculate 123×456,' the AI will first execute a snapshot to obtain the interface skeleton, locate and click the buttons, and finally return the calculation result. The core workflow follows a progressive traversal pattern of 'skeleton scan → targeted drilling → execution → state verification,' which can reduce Token consumption by 78–96%.
It is well-suited for teams or individuals requiring desktop GUI automation, especially agent developers, test engineers, and DevOps engineers. For instance, testers can use it to automate regression testing for desktop applications, DevOps personnel can write scripts to batch-manage windows and notifications, and AI developers can integrate it into their agents to give AI 'hands-on' capabilities. Currently, it supports macOS 12+. Windows and Linux versions are under development.
We recommend using this tool consistently for scenarios requiring stable, reusable desktop interactions. Note that it primarily relies on the accessibility tree. If the target application does not implement accessibility interfaces (e.g., some games or custom controls), it may not fully retrieve UI elements. Additionally, on first use, you must enable Accessibility permissions for the terminal in System Preferences, or commands will fail.
Key Features
Unlike PyAutoGUI or Selenium, Agent Desktop does not rely on screen coordinates or browser drivers. Instead, it retrieves structured JSON (containing ref identifiers) via the native OS accessibility tree, allowing AI agents to precisely locate elements and perform operations, ensuring stable functionality even when window positions change.
Limitations
Currently only supports macOS 12+ and requires the terminal to have Accessibility and Screen Recording permissions. Windows and Linux adaptations are still in the planning stages.
FAQ
What dependencies does Agent Desktop require?
Simply install the `agent-desktop` CLI globally (`npm install -g agent-desktop`), ensure macOS version ≥ 12, and enable Accessibility and Screen Recording permissions for the terminal in System Preferences.
How is it different from AppleScript or UI Automation?
Agent Desktop outputs structured JSON, designed specifically for AI agents. It supports progressive skeleton traversal (reducing Token consumption) and provides 58 commands covering observation, interaction, keyboard/mouse, window management, clipboard, and more, making it easier for AI to invoke compared to AppleScript.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-4/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-4/raw/index.md