@agent-sh/computer-use-linux
Linux Desktop Control MCP Server
Install & Use
Copy this prompt and send it to your AI assistant (Claude / Cursor / TRAE / Codex / WorkBuddy etc.) to auto-install:
Help me install this AI Skill: @agent-sh/computer-use-linux. It is used for: Linux Desktop Control MCP Server Full Skill content: https://321skill.com/skills/agent-sh-computer-use-linux/raw/index.md Read that page and install it.
The prompt includes a link to the full Skill content. You can also view the full content.
This Skill addresses the issue of AI agents being unable to directly interact with local GUI applications on Linux desktop environments. In practical development, developers often need AI to automate desktop actions like clicking, typing, and taking screenshots. While macOS has native AXUIElement, Linux faces fragmentation in its Wayland and X11 ecosystems, causing traditional tools (like xdotool) to fail under Wayland. This tool leverages the AT-SPI accessibility tree, Portal remote desktop interfaces, and various window manager backends, enabling AI to control the Linux desktop just like a human.
Usage is straightforward: simply configure this server in any MCP host (e.g., Claude Desktop, Codex Desktop) or run computer-use-linux directly from the command line. You can then execute complex desktop operations using natural language instructions, such as "Open a terminal and enter a command," "Take a screenshot of the current window and analyze it," or "Click the Wi-Fi toggle in GNOME Settings." It also includes a built-in doctor command that performs a one-click system readiness check, returning a structured JSON report that clearly indicates which features are available and which require additional configuration.
It is ideal for teams or individuals seeking to equip their AI assistants with desktop control capabilities, especially developers already using MCP hosts like Claude Desktop or Hermes Agent but struggling with limited Linux desktop support. Test engineers can use it to automate GUI test cases; operations engineers can monitor remote desktop status; and agent developers can integrate it into multi-agent workflows to achieve cross-application automation.
It is recommended for use on mainstream desktop environments like GNOME, KDE, Hyprland, and i3 for optimal compatibility. Note that it primarily relies on AT-SPI for semantic recognition. For interfaces without accessibility tree support, such as pure Canvas or games, it falls back to pixel-coordinate operations. Under Wayland, some features (like window management) require installing corresponding tools (e.g., ydotool or GNOME Shell extensions). It is advised to run computer-use-linux doctor before first use to ensure all dependencies are ready.
Key Features
Unlike macOS-only computer-use tools, it natively supports both Wayland and X11 on Linux. It locates elements using AT-SPI semantic selectors (role/name/text) instead of relying solely on xdotool's pixel coordinates, significantly improving operational accuracy and portability in complex desktop environments.
Limitations
Relies on the AT-SPI accessibility service; for purely rendered interfaces without accessibility tree support (e.g., games, Canvas), it can only fall back to pixel-coordinate operations. Under Wayland, some window management features require installing GNOME Shell extensions or ydotool, and backend support may vary across different desktop environments.
FAQ
Which Linux desktop environments are supported?
Supports mainstream desktop environments like GNOME, KDE/KWin, Hyprland, i3, and COSMIC, with priority given to Wayland and best-effort compatibility for X11.
Does it require root privileges?
Root privileges are not required, but you need to install ydotool and start the ydotoild daemon, or obtain input permissions via the Portal RemoteDesktop interface.
Installation guide for AI assistants
If your AI coding assistant (Claude Code, Cursor, TRAE etc.) can see this page, send it this message to auto-install:
Visit https://321skill.com/skills/agent-sh-computer-use-linux/raw/index.md to read the original Skill definition (Markdown format) for @agent-sh/computer-use-linux, and install it according to the instructions.
Raw Markdown URL for AI: /skills/agent-sh-computer-use-linux/raw/index.md