agent-desktop

agent-desktop

Enables AI agents to read and interact with desktop applications on macOS, Windows, and Linux using accessibility trees instead of screenshots. It provides a CLI and library for navigation, input, and state management.

Made by vercel-labs on GitHub. These notes are Minda's, written from its README; the project's own files and license are in its repository.

About

What it is: Agent-desktop allows AI agents to operate desktop applications on macOS, Windows, and Linux. It reads the accessibility tree of an application and provides a reference for every element the agent can act upon.

Who it suits: This kit suits developers who want to build AI agents that interact with desktop software without using visual screenshots or pixel matching.

Where to use it:

• A local terminal or command line
• An agent framework supporting MCP
• A custom application using the C library

How to set it up:

1. Run the npm install command to add the tool to your system.
2. Check your platform permissions to ensure the agent can access UI elements.

npm install -g agent-desktop
agent-desktop status
agent-desktop permissions --guide

How to use it: You interact with the tool by running commands that trigger snapshots or specific actions like clicking or typing based on reference IDs found in those snapshots.

agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8

For newcomers and experts: If you are new to this, think of it as giving your AI a map of buttons and fields rather than a picture of the screen. You can start by running a simple snapshot of an open application to see how the system identifies the buttons and menus you see on your desktop.

Experts understand this tool interfaces directly with native accessibility APIs. It avoids the latency and inaccuracy of vision-based models by working with the object model of the UI.

Good to know:

• Accesses desktop UI elements via accessibility APIs
• Requires appropriate OS permissions
• Supports headless operation
• Licensed under Apache-2.0

Platform Requirements Platform | Version | API macOS | macOS 13 or later | AX API Windows | Windows 10 1809 or later | UI Automation Linux | Ubuntu 24.04 or later | AT-SPI

On GitHub, 4 November 2026

• Stars: 12,903 (3,791 that day)
• Forks: 888
• Watching: 30
• Open issues and pull requests: 148
• Language: Rust
• License: Apache-2.0
• Topics: accessibility, accessibility-api, ai-agents, automation, cli, computer-use, computer-use-agent, computer-use-agents, desktop-automation, macos, mcp, rust
• Trending: #2 repository of the day on GitHub trending (4 November 2026)

Links

• Code: https://github.com/vercel-labs/agent-desktop
• Website: https://agent-browser.dev/desktop
• Releases: https://github.com/vercel-labs/agent-desktop/releases
• Issues: https://github.com/vercel-labs/agent-desktop/issues
• Star history: https://www.star-history.com/#vercel-labs/agent-desktop&Date

Intro videos

• vercel-labs/agent-desktop - Gource visualisation: https://www.youtube.com/watch?v=AkRkdce2Hig
• Vercel agent-desktop: How AI Reads Desktop Apps: https://www.youtube.com/watch?v=boEer-ZIFp0
• agent-desktop: Computer Use for Any AI Coding Agent (Vercel Labs, Open Source): https://www.youtube.com/watch?v=ocd8XZudT3M

Get the code

git clone https://github.com/vercel-labs/agent-desktop.git

Then follow the setup in its README. Nothing in this kit runs by itself.