Agent Desktop

This Skill addresses the challenge of AI agents being unable to directly control native desktop applications. In practical development, AI coding assistants or agents often need to click buttons, fill forms, switch windows, or read UI states, but traditional solutions rely on image recognition or browser automation, which are ineffective for native desktop apps. agent-desktop leverages the operating system's native accessibility tree, enabling AI to observe and operate any desktop software just like a human.

Usage is straightforward: simply install the CLI tool in your terminal and grant it accessibility permissions. You can then use natural language instructions to have your AI agent invoke agent-desktop to perform actions, such as "Open Calculator, click number 7, click plus, click 3, click equals, screenshot the result." It employs a progressive skeleton traversal strategy, first obtaining a UI skeleton overview and then expanding details as needed, significantly reducing token consumption. All interactions return structured JSON containing ref identifiers, making it easy for AI agents to maintain state.

It is ideal for teams or individuals looking to extend AI capabilities to desktop applications. This is especially useful for developers already using AI coding agents like Claude or Cursor, who wish to automate desktop software testing, batch-process office tasks, or build agent workflows. Test engineers can use it to write automated test cases for desktop GUIs; operations engineers can use it to monitor and operate management tools; and agent developers can integrate it as part of their toolchain.

It is recommended to use this tool uniformly in a macOS 12+ environment. Note that it currently only supports the macOS Phase 1 adapter; Windows and Linux versions are in the planning stages. Before use, you must grant the terminal app "Accessibility" and "Screen Recording" permissions in System Preferences. Due to its reliance on the native accessibility tree, some non-standard UI frameworks (e.g., games, custom controls) may not be fully recognized.