Agent Desktop
AI Agent Desktop Automation CLI Tool
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: AI Agent Desktop Automation CLI Tool Full Skill content: https://321skill.com/skills/agent-desktop-x-7/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the issue of AI agents being unable to directly control desktop applications. In practical development, AI agents need to perform actions like observing, clicking buttons, filling out forms, switching windows, and taking screenshots, but lack a unified interface to interact with desktop GUIs. agent-desktop leverages the native OS accessibility tree, enabling AI agents to operate desktop applications just like a human would.
Usage is straightforward: simply install the agent-desktop CLI in your terminal and grant it Accessibility and Screen Recording permissions. Afterwards, you can use natural language instructions to have your AI agent call agent-desktop to perform tasks, such as "Open the calculator and calculate 7+3." It employs a progressive skeleton traversal strategy, first obtaining an overview of the UI skeleton before drilling down into target areas, significantly reducing Token consumption.
It is well-suited for teams or individuals developing desktop automation agents. This is especially true for developers already using AI coding assistants like Claude or Cursor, who wish to have AI directly control local desktop applications (e.g., IDEs, browsers, system settings). Test engineers can also use it to automate GUI testing.
It is recommended for use on macOS 12+ systems, ensuring the terminal has been granted Accessibility and Screen Recording permissions. Note that it is primarily designed for scenarios where AI agents make the calls, not as an automation scripting tool for direct human use. Support for Windows and Linux is currently planned.
Key Features
Unlike traditional GUI automation tools such as PyAutoGUI or Selenium, agent-desktop retrieves UI element structures via the native OS accessibility tree, outputting structured JSON (including ref identifiers). This allows AI agents to precisely identify and manipulate elements without relying on image recognition or fixed coordinates. Furthermore, its support for progressive skeleton traversal reduces Token consumption by 78-96%.
Limitations
Currently only supports macOS 12+. Requires Accessibility and Screen Recording permissions. Adaptation for Windows and Linux is planned. Dependent on a Node.js environment.
FAQ
Which operating systems are supported?
Currently only macOS 12+ is supported. Support for Windows and Linux is in the planning stages.
What permissions are required?
The terminal must be granted Accessibility permissions. For screenshot functionality, Screen Recording permission is also required.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-7/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-7/raw/index.md