Vercel Labs takes agent-desktop to version 1.0: agents operate native apps through the accessibility tree

Medium Oct. 30, 2026 · GitHub (Vercel Labs)

Published in edition no. 2 of Noticiário do Joãozinho · Saturday, October 31, 2026, evening edition

Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese

The open source tool (Apache-2.0), written in Rust, reached version 1.0 on 10/28 and 1.0.2 on 10/30. It reads the accessibility tree of macOS, Windows and Linux and returns a JSON with a reference for each element the agent can act on; the agent acts on that reference and takes a new snapshot of the screen to check, without comparing pixels. The decision loop stays with the agent that calls it.

Why it matters Computer use by agents still fails a lot with screenshots. Reading the actual structure of the interface tends to be more reliable and cheaper, and it works for any agent, including those running locally.

Original source: GitHub (Vercel Labs) — https://github.com/vercel-labs/agent-desktop

Full edition PDF (in Portuguese) Read the full edition