Computer use in 2026: three different bets on the screen

The pitch: skip the API, use the screen

Most agent-to-tool integration runs through a protocol: MCP calls a function, A2A hands off a task, an API returns structured data. Computer use skips all of that. The agent gets a screenshot, a mouse, and a keyboard, and it operates software the way a person would — because for most of the software in the world, no API was ever built. There's no schema to call against a legacy desktop app, an internal tool nobody documented, or a website that actively resists automation.

That's the appeal, and it's also why computer use is the hardest agent surface to supervise: there's no request payload to inspect before it fires, just pixels and a click.

Three companies have shipped computer-use systems, and by mid-2026 they've made three different architectural bets about where the screen should live.

Anthropic: a portable tool, still labeled beta

Anthropic shipped the first version of this idea on October 22, 2024, giving Claude 3.5 Sonnet the ability to view a screenshot and issue mouse/keyboard actions inside a sandboxed virtual display. On the OSWorld benchmark — a suite of open-ended desktop tasks — that first version scored 14.9%, well ahead of the 7.7% next-best model at the time but nowhere near the roughly 70-75% human baseline Anthropic cited for comparison (Developing a computer use model).

Almost two years later, computer use is still explicitly a beta feature gated behind an API header, but the tool itself has iterated through several versions. The current one, computer_20251124, requires the computer-use-2025-11-24 beta header and works with Claude Opus 5, Sonnet 5, Opus 4.8/4.7/4.6, and Sonnet 4.6. Its one new capability is zoom: the model can crop into a screen region at full resolution when it needs to read small text — a sidebar filename, a status-bar label — that doesn't survive the screenshot's default downscaling (Computer use tool docs).

The docs are candid about the risk surface. Anthropic recommends running the tool in a dedicated VM with minimal privileges, blocking internet access to an allowlist, and asking for human confirmation before anything with real-world consequences — a payment, a terms-of-service acceptance. It also runs classifiers over screenshots to catch prompt injection embedded in on-screen content, and when one fires, it steers Claude to stop and ask before continuing. None of that is optional guidance tucked in a footnote; it's the top section of the reference page, ahead of the quick-start code.

OpenAI: computer use as a desktop feature, not a standalone product

OpenAI's path went through a dead end first. Operator, the standalone browser-automation agent launched in January 2025, lasted seven months before being folded into ChatGPT agent and shut down as a separate product in mid-2025 (tracker: Operator).

The current approach lives inside Codex instead. On April 16, 2026, OpenAI shipped "background computer use" as part of what it called "Codex for (almost) everything": Codex gets its own cursor and can click and type inside Figma, Xcode, Slack, browsers, and other Mac apps, running as a background session that doesn't take over the mouse you're actively using (Codex for (almost) everything). The feature launched macOS-only; reporting from PC-focused outlets put a Windows 11 version at the end of May 2026, though on Windows the equivalent capability runs foreground-only rather than as a background session — a real capability gap between the two platforms, not just a rollout lag (PCWorld coverage).

The framing is telling: this isn't pitched as a general-purpose computer-use API for developers to build on, the way Anthropic's is. It's a feature inside a specific product, scoped to specific workflows Codex already handles.

Google: betting the browser is the right unit, not the desktop

Google took a narrower bet than either. Gemini 2.5 Computer Use, released to public preview on October 7, 2025 via the Gemini API on Google AI Studio and Vertex AI, is built on Gemini 2.5 Pro and optimized specifically for web and mobile UI control rather than a generic desktop — clicking, typing, scrolling, filling forms, handling dropdowns (Introducing the Gemini 2.5 Computer Use model). It grew out of Project Mariner, Google's earlier browser-agent research project, and now powers pieces of Search's AI Mode and Google's own Firebase Testing Agent internally.

Scoping to the browser is a real trade-off, not just caution. A browser gives an agent a DOM it can sometimes read directly instead of only a bitmap, and it boxes in the blast radius — no accidental file-system writes, no arbitrary desktop app access. It also means Gemini's computer use can't touch the long tail of non-browser software that's the whole reason Anthropic and OpenAI built desktop-level tools in the first place.

Three bets, one table

None of the three call this finished. Anthropic's is still behind a beta header nearly two years in. OpenAI's background mode is macOS-first with a documented capability gap on Windows. Google's is a preview model, not GA. That's consistent with what the OSWorld numbers imply — real progress off a genuinely bad 2024 baseline, still short of a system you'd let run unsupervised on a machine with your files and accounts on it.

What this means for interface design

Computer use doesn't fit the approval-gate patterns built for structured tool calls, because there's no structured call to gate — a click is a click, not a named function with typed arguments a human can scan before it fires. The interface answer so far has converged on the same handful of moves regardless of vendor: run it in an isolated environment, narrow what it can reach before it starts rather than trying to catch every bad action after the fact, and force a stop for anything expensive or hard to undo. Anthropic bakes that into the API surface itself; OpenAI and Google bake it into product scoping (foreground-only, browser-only). Different levers, same instinct — the sandbox is doing the supervision work that a typed schema would do anywhere else.

More on where this sits relative to the rest of the agent-to-software stack is on the tracker.