How AI Agents Are Learning to Use Computers Like Humans Do

A new generation of AI can see a screen, move a cursor, and click through unfamiliar software the way a person would and it's still failing at roughly half the tasks an untrained human finishes without thinking.

Knowlegic Editorial TeamAugust 30, 20266 min read7 views

Share

How AI Agents Are Learning to Use Computers Like Humans Do

A new hire's first week usually looks the same regardless of the job: squinting at an unfamiliar dashboard, hunting for the right button, double-checking before hitting submit on a form that has no undo. Nobody hands them the software's source code. They just watch the screen and figure it out, the way every person who has ever used a computer already has.

For the first time, AI systems are starting the same way. Instead of connecting to an app's backend through a tidy, pre-built API, they take a screenshot, look at the buttons and menus a human would see, and decide where the cursor should go next, one click, one keystroke at a time. Anthropic calls this "computer use." OpenAI called its version Operator. Google called its Project Mariner, before quietly shutting it down.

That shift, from AI that only talks to AI that acts inside the same messy software humans already use, is the most consequential change happening in AI agents right now and it is still closer to a hesitant new hire than a capable assistant, stumbling over exactly the kind of screen that a person would breeze past.

Teaching a Model to See a Screen Instead of Read an API

Most software that talks to AI does it through an API - a clean, structured back door built specifically for machines to exchange data. It works well, but it only works for the software that bothered to build that door. The vast majority of programs people use every day, from a hospital's decades-old records system to a small business's custom spreadsheet macro, never built one.

Computer use skips the back door entirely. In October 2024, Anthropic introduced the capability as part of Claude 3.5 Sonnet: give the model a screenshot, and it identifies what's on screen, decides on an action, move the cursor here, click, type this and takes it, then looks at the new screenshot to see what happened next (Anthropic). It's the same think-act-observe loop underlying how AI agents work in general, just aimed at a mouse and keyboard instead of a search tool or a code editor.

The appeal is obvious: an agent that can see a screen can, in principle, use anything a human can use, without a developer ever building a special integration for it. The catch is that human interfaces were never designed to be legible to a machine. A button might be a picture with no label. A dropdown might only appear after a hover.

People navigate that ambiguity by pattern-matching against a lifetime of using software. An AI has to learn the same intuition from scratch, one pixel-level screenshot at a time.

Did You Know?

Claude's computer-use system operates behind a permission-first design: it has to ask before touching a new application for the first time, and a person can interrupt it mid-task at any moment a safeguard Anthropic built in specifically because an agent that can click anything on a screen can also, in theory, click somewhere it shouldn't.

Three Companies, Three Different Bets on the Same Idea

Anthropic wasn't alone for long. In January 2025, OpenAI launched Operator, a browser-based agent built on a model it called CUA - Computer-Using Agent - trained specifically to interact with the buttons, menus, and text fields of a graphical interface rather than reading raw code (OpenAI). Operator ran in its own sandboxed browser on OpenAI's servers, letting it book tickets or fill out an online order without ever touching the user's actual machine. By July 2025, OpenAI folded Operator into ChatGPT itself as a general-purpose "agent mode," rather than keeping it as a separate product (OpenAI).

Google took a related but distinct path with Project Mariner, a Chrome extension unveiled in December 2024 that watched browser screenshots and issued its own navigation commands to fill out forms and click through multi-step web tasks. It ran for roughly a year and a half before Google shut it down in May 2026, folding its strongest capabilities into Gemini's own agent tools and Chrome's built-in automation features rather than keeping it as a standalone project (Digital Trends). The underlying idea didn't disappear - it just stopped needing its own separate name once agent features became a standard part of the product.

Where Claude Code operates mainly inside a developer's terminal and codebase, computer-use agents are aimed at the opposite end of the spectrum: the ordinary, human-facing screen that was never built with automation in mind. Both are agents in the sense of taking repeated actions toward a goal rather than just answering a single question, but they're solving different problems - one for code, one for everything else.

The Test That Keeps Everyone Honest

Vendor demos make computer use look effortless, which is exactly why an independent benchmark matters. OSWorld, introduced by a team of researchers in 2024, poses 369 real tasks installing software, reformatting a spreadsheet, configuring a setting buried three menus deep on actual Windows, macOS, and Ubuntu desktops, then checks whether the task was genuinely completed, not just attempted.

The gap it exposed was stark. When the benchmark launched, the best available agent completed only about one in eight of those tasks. The researchers who built the test also had ordinary humans try the same tasks, without special training, and found they completed roughly seven in ten of them on their first attempt (arXiv, OSWorld) — worth reading as the benchmark's own internal measurement rather than an independently replicated figure, since it comes from a single study.

OpenAI's own reported score for its first Operator model, using a similar OSWorld-style evaluation, came in at roughly 38% real progress from where the field started, but still well short of that human baseline.

That persistent gap is the honest headline of computer-use AI in 2026: genuinely useful, meaningfully improved from where it started less than two years ago, and still not something you'd trust unsupervised with a task that actually matters if it goes wrong.

The reason computer-use agents struggle most isn't reasoning it's often something as mundane as clicking precisely. Models can usually describe the correct next step in plain language; the harder problem is translating "click the small icon in the top-right corner" into the exact pixel coordinates on a screen that might be a different size or resolution than anything the model has seen before.

Knowlegic Perspective

The most interesting part of this shift isn't that AI can now click buttons, it's what that requirement reveals about the software we've all been using for decades. Every confusing menu, every unlabeled icon, every workflow that only makes sense because a human already knows the trick, is now a measurable obstacle course for a machine trying to learn it from scratch.

Computer-use agents are, in effect, the first systematic audit of how genuinely usable our interfaces actually are. It's a reasonable bet that the software of the next decade gets redesigned partly with an AI operator in mind, not just a human one the same way websites eventually got redesigned once search engines became a second audience nobody could ignore.

They're improving fast, but they are still closer to a hesitant new hire than a capable assistant, and the gap between what they can do and what an ordinary human does without thinking is the most honest measure of how far this technology still has to go.