Agent Desktop
AI Agent Desktop Automation CLI Tool
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: Agent Desktop. It is used for: AI Agent Desktop Automation CLI Tool Full Skill content: https://321skill.com/skills/agent-desktop-x-8/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the challenge of AI agents being unable to directly control desktop applications. In practical development, AI agents need to perform actions like observing, clicking buttons, filling forms, switching windows, and taking screenshots. However, traditional solutions rely on coordinates or image recognition, which are unstable and susceptible to environmental changes. agent-desktop solves this by accessing the native operating system's Accessibility Tree to obtain UI structure, outputting structured JSON. This enables AI agents to precisely locate and manipulate interface elements.
Usage is straightforward: simply install agent-desktop in your terminal and grant it Accessibility and Screen Recording permissions. You can then use natural language instructions for your AI agent to invoke agent-desktop commands, such as "click the login button," "fill the username input field," or "screenshot the current window." The core process employs a progressive skeleton traversal strategy: first, obtain a UI skeleton overview; then, drill down into the target area; finally, execute the action and verify state changes. This approach significantly reduces token consumption.
It is ideal for AI agent developers, test engineers, and operations personnel who require desktop GUI automation. It's particularly suited for teams already using AI coding assistants like Claude or Cursor and looking to extend their capabilities to desktop application interaction. For scenarios involving batch processing of desktop tasks, automating desktop software testing, or building agent workflows, agent-desktop provides stable, programmable underlying capabilities.
It is recommended for use on macOS 12+ systems. Ensure you have enabled Accessibility and Screen Recording permissions for your terminal in System Preferences beforehand. Note that it currently only supports the macOS Phase 1 adapter; Windows and Linux support is planned. For simple applications (e.g., Calculator, TextEdit), you can use a full snapshot directly. For complex applications, using progressive skeleton traversal is recommended to save tokens.
Key Features
Unlike solutions like pyautogui that rely on coordinates or image recognition, agent-desktop retrieves UI element structure via the operating system's native Accessibility Tree, outputting JSON with ref identifiers. This makes it unaffected by changes in screen resolution, themes, or window position, resulting in more stable and reliable element targeting.
Limitations
Currently only supports macOS 12+. Requires the terminal to have Accessibility and Screen Recording permissions. Adapters for Windows and Linux are still in the planning stages.
FAQ
How do I install agent-desktop?
Run `npm install -g agent-desktop` or `bun install -g --trust agent-desktop` for a global installation.
What system permissions are required?
Requires macOS 12+. You must enable Accessibility permissions (for UI interaction) and Screen Recording permissions (for screenshots) for your terminal in System Preferences.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-desktop-x-8/raw/index.md to read the original Skill definition (Markdown format) for Agent Desktop, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-desktop-x-8/raw/index.md