Skip to content
CLAUDEXIA
enru
AUTOMATION

Claude Computer Use in 2026: Browser Automation That Actually Works

Claude's computer use lets the model see screens and click — automating real browsers and desktops. Architecture, sandbox setup, costs, and reliability tips.

For most of the API era, getting an LLM to "use" software meant wiring up tools — a search function here, a database query there, maybe a Playwright script if you were brave. Computer use flips that model. You hand Claude a screenshot of a real screen and a tiny set of primitive actions (click, type, scroll, key), and the model drives the UI the same way a human would. In 2026, with Sonnet 4.5+ and Opus 4.7, this finally works well enough to put into narrow production workflows. This post walks through what it is, when to reach for it, how to deploy it safely, and what it actually costs.

What computer use actually is

The mechanic is deceptively simple. On every turn:

  1. Your harness takes a screenshot of a virtual display.
  2. You send the screenshot to Claude as an image, along with the user's goal and the running action history.
  3. Claude responds with a tool call: computer with an action like { action: "left_click", coordinate: [834, 412] }, key, type, screenshot, scroll, etc.
  4. Your harness executes the action against the OS, takes a fresh screenshot, and loops.

The model is not running on your machine. It is doing visual reasoning on pixels and emitting low-level intents. That is what makes it work on legacy software with no API, on websites that fight scrapers, and on internal tools that nobody is going to wrap in REST any time soon.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: process.env.CLAUDEXIA_API_KEY!,
  baseURL: "https://api.claudexia.tech/v1",
});

const response = await client.beta.messages.create({
  model: "claude-sonnet-4.5",
  max_tokens: 1024,
  tools: [
    {
      type: "computer_20250124",
      name: "computer",
      display_width_px: 1280,
      display_height_px: 800,
      display_number: 1,
    },
  ],
  betas: ["computer-use-2025-01-24"],
  messages: [
    { role: "user", content: "Open the orders dashboard and export today's CSV." },
  ],
});

When to reach for it (and when not to)

Computer use is a power tool with a sharp edge. The cases where it earns its keep:

  • Legacy software with no API. Internal ERPs, mainframe terminals, vendor portals from 2009 — anything where building a real integration is more expensive than the work itself.
  • Sites behind login flows that block scrapers. Where a Playwright selector breaks every week, a model that reads the rendered page is dramatically more robust.
  • End-to-end testing for visual regressions. Not unit-level, but the "does the checkout flow still work after we redesigned the cart" tier.
  • RPA replacement. Most UiPath/Automation Anywhere bots are brittle click-by-coordinate scripts. Computer use replaces them with a system that adapts when the button moves.

The cases where you should walk away:

  • Anything with a real API. Computer use is 100–1000× more expensive than a direct call.
  • High-volume, latency-sensitive paths. Each step is a vision-tokenised round trip.
  • Tasks where the model touching the wrong button is catastrophic (production money movement, infrastructure changes, deletes). Use APIs with explicit allow-lists instead.

Sandbox setup: Docker + xvfb + Playwright

Never run computer use against your real desktop. The pattern that works in production is a disposable Docker container with a virtual framebuffer and a browser the model can drive:

FROM mcr.microsoft.com/playwright:v1.50.0-jammy

RUN apt-get update && apt-get install -y \
    xvfb x11vnc fluxbox imagemagick && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY package.json ./
RUN npm install

COPY . .

ENV DISPLAY=:99
CMD ["bash", "-c", "Xvfb :99 -screen 0 1280x800x24 & fluxbox & node harness.js"]

Inside the container, your harness owns the loop: it boots Chromium with Playwright, takes screenshots via the Page or via import from ImageMagick, sends them to Claude, parses the tool call, and dispatches the action through Playwright's page.mouse, page.keyboard, and navigation primitives. Keep the container ephemeral — one task, one container, hard timeout, then destroy. That is your blast radius.

The loop pattern

The minimum viable harness is small:

async function runTask(goal: string, maxSteps = 30) {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: goal },
  ];

  for (let step = 0; step < maxSteps; step++) {
    const screenshot = await takeScreenshot();
    messages.push({
      role: "user",
      content: [
        { type: "image", source: { type: "base64", media_type: "image/png", data: screenshot } },
        { type: "text", text: `Step ${step}. What's next?` },
      ],
    });

    const res = await client.beta.messages.create({
      model: "claude-sonnet-4.5",
      max_tokens: 1024,
      tools: computerTools,
      betas: ["computer-use-2025-01-24"],
      messages,
    });

    if (res.stop_reason === "end_turn") return "done";
    const toolUse = res.content.find((c) => c.type === "tool_use");
    if (!toolUse) return "stuck";
    await executeAction(toolUse.input);
    messages.push({ role: "assistant", content: res.content });
  }
  throw new Error("max_steps exceeded");
}

A few non-obvious details matter here. Always carry the assistant's prior turns forward — Claude needs the action history to avoid repeating itself. Always cap max_steps; a runaway loop is the single most expensive failure mode. And always log every screenshot and action to disk: when something goes wrong, the trace is the only debugger you have.

Reliability tips that actually move the needle

Months of running these loops in production tend to converge on the same practices:

  1. Decompose ruthlessly. "Process this 50-row spreadsheet" fails. "Open row 1, copy the email field, paste into the form, submit, return to spreadsheet" succeeds, run 50 times.
  2. State the goal explicitly each turn. Re-stating "your goal is to export the CSV" in the user message every step keeps the model anchored when the page changes.
  3. Screenshot every step, not just when the model asks. Models sometimes act on stale memory. A fresh image per turn cuts that class of error.
  4. Hard cap steps and wall-clock time. A 30-step / 5-minute ceiling per task catches infinite loops cheaply.
  5. Validate outcomes with a second model call. After the loop ends, take one final screenshot and ask Sonnet "did the task succeed? answer yes or no with one sentence." This is your cheap CI signal.

Cost reality

This is where intuition usually misleads people. Computer use is dominated by vision tokens, not output tokens. A 1280×800 PNG runs roughly 1500–2000 input tokens depending on detail level. A 30-step task sending one screenshot per step is therefore around 45 000–60 000 input tokens before the model has said anything. Output is small by comparison — a few tool calls per step.

Empirically, on Sonnet 4.5 a well-behaved task lands at about $0.05 to $0.20 end-to-end. Long, exploratory tasks on Opus 4.7 can hit $1+ if you are not careful. Two levers help: aggressive prompt caching of the system prompt and tool schemas (see our pricing notes for the math), and downsampling screenshots to the smallest resolution the task tolerates. 1024×640 is often enough.

Failure modes and how to handle them

The most common failure is hallucinated coordinates: Claude clicks at (420, 380) when the button is at (430, 395). The robust fix is a grid-overlay retry. On a failed action, render a faint coordinate grid onto the screenshot and re-ask. The model uses the grid as a visual anchor and the second attempt almost always lands.

Other recurring issues:

  • Stale screenshots after navigation. Always wait for networkidle or a short setTimeout before the next capture.
  • Modal dialogs the model ignores. Add a "if you see a popup, dismiss it before continuing" line to the system prompt.
  • Captchas. Stop. You should not be automating these. Bail the task out cleanly and surface to a human queue.

Versus OpenAI Operator

OpenAI's Operator and Anthropic's computer use have converged on broadly the same shape: a screenshot-in, action-out loop with a sandboxed browser. Two practical differences in 2026:

  • Caution profile. Claude is noticeably more conservative about destructive actions — it will pause and ask before deleting, paying, or sending. That is good in production, occasionally annoying in research.
  • Reasoning depth on multi-step plans. Sonnet 4.5 with extended thinking pulls ahead on tasks that require holding a 10+ step plan in mind across page transitions.

Pick the one whose harness fits your stack. The capability gap is small; the integration cost is not.

Bottom line

Computer use in 2026 is real, useful, and finally reliable enough for narrow well-defined tasks. It is not a replacement for proper APIs and it is not the right tool for high-volume hot paths. But for the long tail of "the system has no API and never will," it has quietly become the most pragmatic option on the table. Sandbox it hard, cap your steps, log everything, and treat it like the unsupervised intern it is — and you will get a lot of value out of a few cents per task.

← Back to all posts