# Computer use API: let a model use a browser through your code

> A computer use API gives your code a way to hand a model a screen. The model looks at a screenshot and asks for a click, a keypress or a scroll. Your program performs it and sends back the new screenshot, and the cycle repeats until the task is done. Anthropic and OpenAI both offer it, and both publish runnable samples.

Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/ai-browser-agents/computer-use-api

## In short

- Run a computer use sample in a container or a disposable browser, never in the session where you are signed in.
- Scale screenshots to the size the vendor recommends and map the clicks back yourself, instead of leaving it to automatic resizing.
- Put a step limit and a time limit on the loop before the first run, because the vendor skeletons leave both to you.
- Open the saved page or file and compare it with your request, because a final answer is not proof.

## The loop, in plain words

The model never touches the machine. Your program does, which is why this lesson is for developers. A screenshot is an image of the current screen, and an action is one structured request, such as a click at an x and y position.

OpenAI's guide counts three steps: send a task, execute the returned actions, and return a screenshot. Then the cycle starts again with the new screen. Anthropic's demo calls the same cycle a sampling loop, a function that keeps calling the model until a reply asks for nothing more.

OpenAI's guide offers two ways to use the API directly. With code execution, the model writes code that drives the browser, for example with Playwright or PyAutoGUI, and your app runs it. With the computer tool, the model returns structured mouse and keyboard actions that your app translates into input. For GPT-6 Astra, OpenAI recommends code execution and says the computer tool remains supported as an alternative. This lesson shows the computer tool loop, because it pictures the screenshot cycle most plainly, and Anthropic's tool has the same shape.

| Step | OpenAI (computer tool, Responses API) | Anthropic (Messages API) |
| --- | --- | --- |
| Turn the tool on | tools=[{"type": "computer"}] | The demo builds the tool list with tool_collection.to_params(). The newest toolset needs no beta header. |
| The model asks | A computer_call item whose actions array can hold several actions | A tool_use block in the reply |
| You answer | A computer_call_output with the call_id and a computer_screenshot image | A tool_result block with the tool_use_id and a base64 PNG image |
| The conversation continues | previous_response_id on the next request | You append the reply and your tool results to the messages list |
| The loop ends | The response holds no computer_call | The reply holds no tool_use block, so the demo has no result to send |

## What the model can ask for

Each action is a request your code must translate into real input. The table lists the action names in OpenAI's integration recipes and in the Anthropic demo's code, side by side.

Anthropic keeps adding to its list with each dated tool version. The newest shape, computer_toolset_20260801, declares one tool for each action instead of one computer tool with an action field. A model supports the toolset or the earlier dated versions it was trained with, so match the tool version to the model.

| What you want | OpenAI computer tool | Anthropic computer tool |
| --- | --- | --- |
| Click | click, with a button field | left_click, right_click, middle_click |
| Double click | double_click | double_click |
| Drag | drag | left_click_drag |
| Move the pointer | move | mouse_move |
| Scroll | scroll | scroll |
| Type text | type | type |
| Press keys | keypress | key, hold_key |
| Wait | wait | wait |
| Look at the screen | screenshot | screenshot |
| Only in one list | Nothing extra is listed | triple_click, left_mouse_down, left_mouse_up, cursor_position, and zoom in computer_20251124 |

## Run OpenAI's booking sample

OpenAI's sample app repository holds a fake hotel booking lab and a short SDK example that runs the Responses loop on it through OpenAI's ComputerToolset helper. A Playwright driver takes the screenshots and applies the mouse and keyboard actions. The repository's two full apps follow the guide's other path, and this lesson does not run them: OpenAI says it builds that loop around models that write code to interact with software. You need Node.js 22.20.0 and an OpenAI API key with access to the configured model. Each run makes real API calls and uses your credits.

The script starts a local server for the lab, drives a new headless Chromium context (a browser with no visible window), blocks requests to other sites and closes both when the run ends. OpenAI says it does not control your desktop or open your own browser session. Add --prompt to try a different booking, or --model to use another model that supports computer use.

````bash
git clone https://github.com/openai/openai-cua-sample-app.git
cd openai-cua-sample-app
corepack enable
pnpm install --frozen-lockfile
cp .env.example .env
export OPENAI_API_KEY="your_api_key_here"
pnpm --dir javascript-app/examples exec playwright install chromium-headless-shell
pnpm --dir javascript-app/examples start
````

1. Clone the repository, enable Corepack, install the dependencies and set your API key, as the first six lines of the code block show.
2. Install the browser the example uses, then start the example, with the last two lines.
3. Read the answer the script prints.
4. Open data/sdk-booking.png, the saved final page, and compare the booking you see with the one you asked for.

> The commands are OpenAI's, copied from the sample app README on 6 November 2026: its First Run commands, then its Node.js example commands without the repeated pnpm install line. Nothing here was run.

## Or run Anthropic's demo in Docker

Anthropic's computer use demo runs a Linux desktop inside a Docker container and puts a chat interface beside it. Once the container is running, open http://localhost:8080 to see the chat and the desktop together. The demo's README also lists http://localhost:8501 for the chat only and http://localhost:6080/vnc.html for the desktop only.

The README says plainly that the pieces are weakly separated. The agent loop runs in the container that Claude controls, serves one session at a time, and needs a restart or reset between sessions. It opens with four precautions: a dedicated virtual machine or container with minimal privileges, no sensitive data such as account logins, an allowlist of internet domains, and a human to confirm consequential decisions. [Lesson 4](https://mindket.com/learn/ai-browser-agents/browser-agent-safety) covers them.

````bash
export ANTHROPIC_API_KEY=%your_api_key%
docker run \
    -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
    -v $HOME/.anthropic:/home/computeruse/.anthropic \
    -p 5900:5900 \
    -p 8501:8501 \
    -p 6080:6080 \
    -p 8080:8080 \
    -it ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest
````

> The command is from the README of Anthropic's computer-use-demo, copied on 6 November 2026 and not run here. Set WIDTH and HEIGHT with -e to change the screen size.

## OpenAI's computer tool loop in code

This is the computer tool loop skeleton from OpenAI's integration recipes. It assumes you wrote handle_computer_actions and capture_screenshot. OpenAI says it illustrates the exchange rather than a complete runtime, and tells you to add permission checks, cancellation, and step and time limits.

Read the first lines for the stop rule: when the response holds no computer_call, the function returns. Everything else is one round trip.

````python
import base64

from openai import OpenAI

client = OpenAI()


def computer_use_loop(target, response):
    while True:
        computer_call = next(
            (item for item in response.output if item.type == "computer_call"),
            None,
        )
        if computer_call is None:
            return response

        handle_computer_actions(target, computer_call.actions)

        screenshot = capture_screenshot(target)
        screenshot_base64 = base64.b64encode(screenshot).decode("utf-8")

        response = client.responses.create(
            model="gpt-6.1-sol",
            tools=[{"type": "computer"}],
            previous_response_id=response.id,
            input=[
                {
                    "type": "computer_call_output",
                    "call_id": computer_call.call_id,
                    "output": {
                        "type": "computer_screenshot",
                        "image_url": f"data:image/png;base64,{screenshot_base64}",
                        "detail": "original",
                    },
                }
            ],
        )
````

> Copied from OpenAI's computer use integration recipes, as written there. The two helper functions are left to you, and the code was not run here.

## Anthropic's loop in code

Anthropic's demo keeps its loop in sampling_loop. The excerpt keeps the call to the API, the assistant message and the end of the loop, and # ... marks code left out. When the reply holds no tool_use block, tool_result_content stays empty and the function returns.

The demo also drops old screenshots. Its helper removes all but the most recent images from the tool results, because a screenshot loses value as the conversation goes on and every one is sent again with each request.

````python
async def sampling_loop(
    # ...
):
    while True:
        # ...
            raw_response = client.beta.messages.with_raw_response.create(
                max_tokens=max_tokens,
                messages=messages,
                model=model,
                system=[system],
                tools=tool_collection.to_params(),
                # The toolset group has no beta flag, so on Vertex/Bedrock the
                # list can be empty; omit the header rather than send it blank.
                betas=betas or omit,
                extra_body=extra_body,
            )
        # ...
        response = raw_response.parse()

        response_params = _response_to_params(response)
        messages.append(
            {
                "role": "assistant",
                "content": response_params,
            }
        )
        # ...
        if not tool_result_content:
            return messages

        messages.append({"content": tool_result_content, "role": "user"})
````

> An excerpt of sampling_loop in loop.py from anthropics/anthropic-quickstarts. The lines are the file's, and # ... marks the code left out. It was not run here.

## Screen size and what to expect

Anthropic does not recommend sending screenshots larger than XGA (1024x768) or WXGA (1280x800), two standard screen sizes. Leaving the resizing to the API gives lower accuracy and slower responses than scaling in your own tool. For your own implementation it recommends XGA: scale a larger screen down and map the model's coordinates back to the original size, and pad a smaller screen with black until it reaches 1024x768.

Accuracy is the other limit. When Anthropic opened computer use in public beta in October 2024 it called the feature experimental, at times cumbersome and error-prone. The figures below are from that post. Two of them come from OSWorld, which Anthropic describes as evaluating AI models' ability to use computers like people do. They describe one model, Claude 3.5 Sonnet, that has since been replaced, and the sources read for this lesson give no newer benchmark, so test on your own task.

**Screen size and the October 2024 benchmark, as the vendors report them**

- **1024x768** screen size Anthropic recommends when you implement computer use yourself (XGA) (Source: [anthropics/anthropic-quickstarts: computer use demo README](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo), read 2026-11-06)
- **14.9%** OSWorld score of Claude 3.5 Sonnet in the screenshot-only category, October 2024 (Source: [Anthropic: Computer use, Claude 3.5 Sonnet and Claude 3.5 Haiku](https://www.anthropic.com/news/3-5-models-and-computer-use), read 2026-11-06)
- **22.0%** the same model on OSWorld when given more steps to finish the task (Source: [Anthropic: Computer use, Claude 3.5 Sonnet and Claude 3.5 Haiku](https://www.anthropic.com/news/3-5-models-and-computer-use), read 2026-11-06)

## Check the result outside the screenshot

OpenAI's guide tells you to bound the run with step, time or cost limits, support cancellation, and check the actual outcome instead of relying on the model's final answer. The sample app README says the same in one line: a final answer does not prove the task succeeded.

OpenAI's recipes show the habit in code. A desktop demo has the model type a note into an editor and save it. The script then reads the saved file itself and raises an error if it does not match, with a comment that saving is asynchronous and the file should be checked independently of the screenshot.

## Watch Anthropic's video

Anthropic put a video on using computer use to automate operations on its YouTube channel.

[Claude | Computer use for automating operations](https://www.youtube.com/watch?v=ODaHJzOyVCQ)

Anthropic's video on computer use for automating operations, on Anthropic's YouTube channel.

## Common mistakes

- Sending full-resolution screenshots and trusting automatic resizing, which Anthropic says lowers accuracy.
- Running the loop with no step or time limit. OpenAI marks its skeleton as a sketch that leaves those to you.
- Running a sample in your everyday desktop session. OpenAI's Python app controls your real mouse and keyboard, and its README says to use a dedicated desktop session.
- Pairing a model with a tool version it was not trained for. Anthropic's README sends you to its docs for each model's supported versions.
- Believing the final message. Open data/sdk-booking.png or the saved file.
- Pointing a sample at a site where you are signed in. The OpenAI sample uses a fake lab and Anthropic recommends a container.

## Try it yourself

- Run OpenAI's booking sample, open data/sdk-booking.png and compare the booking with your request.
- Run it again with a different --prompt and compare again.
- Start Anthropic's Docker demo, ask it to open Firefox, and watch the desktop view at http://localhost:6080/vnc.html.
- Add a counter to OpenAI's loop skeleton that stops after a number of turns you choose, and print how many turns a task used.

## Next: keeping browser agents safe

[Browser agent safety](https://mindket.com/learn/ai-browser-agents/browser-agent-safety) is lesson 4. It covers prompt injection, the controls both vendors tell you to add, and the habits that keep a browser agent from doing damage.

## Check yourself

1. In OpenAI's computer tool loop, what do you send back after you run the model's actions?

   A. A new request that repeats the task
   B. A computer_call_output with the call_id and a screenshot
   C. A tool_result block with an image

   Answer: B. OpenAI's skeleton returns a computer_call_output that carries the call_id and a computer_screenshot, while a tool_result block is Anthropic's name for the same step.

2. How does Anthropic's demo loop know the task is over?

   A. The screenshot turns blank
   B. It counts to a fixed number of turns
   C. The page title changes
   D. The reply holds no tool_use block, so there is no result to send

   Answer: D. In loop.py the function returns when tool_result_content is empty, which happens when the reply asks for no tool.

3. Your screen is larger than XGA. What does Anthropic recommend?

   A. Scale the image down to XGA and map the model's coordinates back to the original size
   B. Send the full resolution and let the API resize it
   C. Crop the screenshot to its top left corner

   Answer: A. The README says relying on the API to resize lowers accuracy and slows the response, and recommends scaling in your own tool.

4. A sample run prints a confident final answer. What do you do before you rely on it?

   A. Trust it, because the model saw the screen
   B. Run it again with a longer prompt
   C. Open the saved final page and compare it with your request
   D. Delete the saved screenshot

   Answer: C. OpenAI's sample app README says a final answer does not prove the task succeeded, so you inspect the result.

## Frequently asked questions

### What is the computer use API?

Both vendors ship it inside their model API, as a tool you declare. The model proposes actions, and your program runs them and reports what the screen shows now. OpenAI's guide describes a second, code execution path in the section above.

### What is the difference between OpenAI's and Anthropic's computer use tools?

The loop is the same, but the names differ. OpenAI's computer tool returns a computer_call with an actions array and expects a computer_call_output with a screenshot. Anthropic returns a tool_use block and expects a tool_result block with an image.

### Do I have to write the loop myself?

No. OpenAI's SDK examples use a ComputerToolset to run the Responses loop, and Anthropic's demo ships its own sampling loop. Writing the loop yourself is for when you need to control every step, and OpenAI's guide covers it.

### Is it safe to run on my own computer?

Not without precautions. Anthropic's README advises running it isolated, with minimal privileges, and OpenAI's sample app README says its code runs with your user permissions and offers no operating-system sandbox.

### How accurate is computer use?

This lesson's sources hold one benchmark, from October 2024, when Claude 3.5 Sonnet scored below 25% on OSWorld, a benchmark of computer tasks, even with extra steps. Anthropic called the feature experimental and error-prone and expected it to improve quickly, so test your own task.

### What does a computer use run cost?

Each model request is billed for input and output tokens, and OpenAI's guide says screenshots count as image input. The cost grows with every step, so put a step limit on the loop.

## Sources

- [OpenAI: Computer use guide](https://developers.openai.com/api/docs/guides/tools-computer-use) (read 2026-11-06)
- [OpenAI: Computer use integration recipes](https://developers.openai.com/api/docs/guides/tools-computer-use-integration) (read 2026-11-06)
- [openai/openai-cua-sample-app README](https://github.com/openai/openai-cua-sample-app) (read 2026-11-06)
- [anthropics/anthropic-quickstarts: computer use demo README](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo) (read 2026-11-06)
- [anthropic-quickstarts: computer-use-demo loop.py](https://github.com/anthropics/anthropic-quickstarts/blob/main/computer-use-demo/computer_use_demo/loop.py) (read 2026-11-06)
- [anthropic-quickstarts: computer-use-demo computer.py](https://github.com/anthropics/anthropic-quickstarts/blob/main/computer-use-demo/computer_use_demo/tools/computer.py) (read 2026-11-06)
- [Anthropic: Computer use, Claude 3.5 Sonnet and Claude 3.5 Haiku](https://www.anthropic.com/news/3-5-models-and-computer-use) (read 2026-11-06)
- [Anthropic on YouTube: Computer use for automating operations](https://www.youtube.com/watch?v=ODaHJzOyVCQ) (read 2026-11-06)

[Browse MCP servers](https://mindket.com/mcp-servers)