Computer-Use Agents

Computer-use agents are artificial intelligence systems that operate a computer the way a person does — perceiving the screen, moving a cursor, clicking, typing and navigating applications — rather than connecting to software through purpose-built APIs. The approach allows agents to work with legacy systems, desktop applications and websites that have no automation interface, and it became an active commercial field after Anthropic released a computer-use capability for Claude in October 2024 and OpenAI launched its Operator agent in early 2025.^[1][2]

History

Anthropic introduced computer use in public beta on 22 October 2024, alongside an upgraded Claude 3.5 Sonnet — the first frontier model with the capability. In that early release Claude scored 14.9 per cent on OSWorld, a benchmark of real desktop tasks, against 7.8 per cent for the next-best system, while the company described the capability as experimental and error-prone. OpenAI followed on 23 January 2025 with Operator, a research-preview agent powered by a Computer-Using Agent model that browsed the web in a cloud sandbox for Pro subscribers; the standalone product was folded into ChatGPT Agent in July 2025. Google's Project Mariner and later Gemini-in-Chrome integrations pursued the browser-agent path, while open-source projects such as Browser Use (MIT-licensed, model-agnostic) and ByteDance's UI-TARS gave developers self-hosted alternatives. By 2026, Anthropic's newest models had lifted OSWorld-Verified performance from the original 14.9 per cent into the low 80s, and the company packaged the capability for desktop productivity workflows; Browser Use reported 89.1 per cent on the WebVoyager benchmark for browser tasks.^{[3][4][5][6][7]}

Key Concepts and Technology

A computer-use agent runs a perception–planning–action loop: it captures screenshots of the current screen, reasons about the task, and emits low-level actions such as moving the mouse, clicking coordinates, typing text or scrolling, repeating until the task completes. Some implementations also read the browser's document structure or accessibility tree, which can be more robust than pure vision for finding elements, and most run inside sandboxed virtual machines because the agent has full control of the desktop it operates. Contrasts are drawn with traditional robotic process automation, which executes brittle scripts tied to specific interface layouts; a computer-use agent can adapt when an interface changes, at the cost of slower execution and higher token spend, since every step involves image analysis. The security profile is the field's central concern: a computer-use agent is exposed to prompt injection — malicious instructions hidden in web pages or documents it reads — and vendors recommend human approval for consequential actions, restricted credentials, and confined environments. Benchmarks such as OSWorld, WebVoyager and WebArena measure task success, but practitioners note that production reliability trails benchmark scores because real websites deploy CAPTCHAs, bot defences and session limits that test environments do not.^[3][5][7][8]

Applications and Impact

Early adopters use computer-use agents for software testing and quality assurance — driving applications end-to-end the way a user would — for back-office workflows that span legacy desktop systems, for web research and form-filling, and for data entry across systems with no APIs. Customer-service and operations teams explore them as a flexible layer above existing tools, and developers use them to automate repetitive browser tasks in procurement, finance and compliance. Analysts position the technology as an evolution of RPA rather than a replacement, with a trade-off between adaptability and cost, and adoption is concentrated where the value of automating an unintegrated workflow exceeds the expense of screenshot-based reasoning. Regulatory attention is growing in parallel, with guidance converging on disclosure, human oversight and audit trails for agents taking real-world actions.^[5][7][8]

>See Also

• AI Agents
• Agentic AI
• Autonomous Agents
• AI Browser
• Prompt Injection