Computer use API: let a model use a browser through your code

The loop, in plain words

The model never touches the machine. Your program does, which is why this lesson is for developers. A screenshot is an image of the current screen, and an action is one structured request, such as a click at an x and y position.

OpenAI's guide counts three steps: send a task, execute the returned actions, and return a screenshot. Then the cycle starts again with the new screen. Anthropic's demo calls the same cycle a sampling loop, a function that keeps calling the model until a reply asks for nothing more.

OpenAI's guide offers two ways to use the API directly. With code execution, the model writes code that drives the browser, for example with Playwright or PyAutoGUI, and your app runs it. With the computer tool, the model returns structured mouse and keyboard actions that your app translates into input. For GPT-6 Astra, OpenAI recommends code execution and says the computer tool remains supported as an alternative. This lesson shows the computer tool loop, because it pictures the screenshot cycle most plainly, and Anthropic's tool has the same shape.

What the model can ask for

Each action is a request your code must translate into real input. The table lists the action names in OpenAI's integration recipes and in the Anthropic demo's code, side by side.

Anthropic keeps adding to its list with each dated tool version. The newest shape, computer_toolset_20260801, declares one tool for each action instead of one computer tool with an action field. A model supports the toolset or the earlier dated versions it was trained with, so match the tool version to the model.

Run OpenAI's booking sample

OpenAI's sample app repository holds a fake hotel booking lab and a short SDK example that runs the Responses loop on it through OpenAI's ComputerToolset helper. A Playwright driver takes the screenshots and applies the mouse and keyboard actions. The repository's two full apps follow the guide's other path, and this lesson does not run them: OpenAI says it builds that loop around models that write code to interact with software. You need Node.js 22.20.0 and an OpenAI API key with access to the configured model. Each run makes real API calls and uses your credits.

The script starts a local server for the lab, drives a new headless Chromium context (a browser with no visible window), blocks requests to other sites and closes both when the run ends. OpenAI says it does not control your desktop or open your own browser session. Add --prompt to try a different booking, or --model to use another model that supports computer use.

git clone https://github.com/openai/openai-cua-sample-app.git
cd openai-cua-sample-app
corepack enable
pnpm install --frozen-lockfile
cp .env.example .env
export OPENAI_API_KEY="your_api_key_here"
pnpm --dir javascript-app/examples exec playwright install chromium-headless-shell
pnpm --dir javascript-app/examples start

1. Clone the repository, enable Corepack, install the dependencies and set your API key, as the first six lines of the code block show.
2. Install the browser the example uses, then start the example, with the last two lines.
3. Read the answer the script prints.
4. Open data/sdk-booking.png, the saved final page, and compare the booking you see with the one you asked for.

The commands are OpenAI's, copied from the sample app README on 6 November 2026: its First Run commands, then its Node.js example commands without the repeated pnpm install line. Nothing here was run.

Or run Anthropic's demo in Docker

Anthropic's computer use demo runs a Linux desktop inside a Docker container and puts a chat interface beside it. Once the container is running, open http://localhost:8080 to see the chat and the desktop together. The demo's README also lists http://localhost:8501 for the chat only and http://localhost:6080/vnc.html for the desktop only.

The README says plainly that the pieces are weakly separated. The agent loop runs in the container that Claude controls, serves one session at a time, and needs a restart or reset between sessions. It opens with four precautions: a dedicated virtual machine or container with minimal privileges, no sensitive data such as account logins, an allowlist of internet domains, and a human to confirm consequential decisions. Lesson 4 covers them.

export ANTHROPIC_API_KEY=%your_api_key%
docker run \
    -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
    -v $HOME/.anthropic:/home/computeruse/.anthropic \
    -p 5900:5900 \
    -p 8501:8501 \
    -p 6080:6080 \
    -p 8080:8080 \
    -it ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest

The command is from the README of Anthropic's computer-use-demo, copied on 6 November 2026 and not run here. Set WIDTH and HEIGHT with -e to change the screen size.

OpenAI's computer tool loop in code

This is the computer tool loop skeleton from OpenAI's integration recipes. It assumes you wrote handle_computer_actions and capture_screenshot. OpenAI says it illustrates the exchange rather than a complete runtime, and tells you to add permission checks, cancellation, and step and time limits.

Read the first lines for the stop rule: when the response holds no computer_call, the function returns. Everything else is one round trip.

import base64

from openai import OpenAI

client = OpenAI()


def computer_use_loop(target, response):
    while True:
        computer_call = next(
            (item for item in response.output if item.type == "computer_call"),
            None,
        )
        if computer_call is None:
            return response

        handle_computer_actions(target, computer_call.actions)

        screenshot = capture_screenshot(target)
        screenshot_base64 = base64.b64encode(screenshot).decode("utf-8")

        response = client.responses.create(
            model="gpt-6.1-sol",
            tools=[{"type": "computer"}],
            previous_response_id=response.id,
            input=[
                {
                    "type": "computer_call_output",
                    "call_id": computer_call.call_id,
                    "output": {
                        "type": "computer_screenshot",
                        "image_url": f"data:image/png;base64,{screenshot_base64}",
                        "detail": "original",
                    },
                }
            ],
        )

Copied from OpenAI's computer use integration recipes, as written there. The two helper functions are left to you, and the code was not run here.

Anthropic's loop in code

Anthropic's demo keeps its loop in sampling_loop. The excerpt keeps the call to the API, the assistant message and the end of the loop, and # ... marks code left out. When the reply holds no tool_use block, tool_result_content stays empty and the function returns.

The demo also drops old screenshots. Its helper removes all but the most recent images from the tool results, because a screenshot loses value as the conversation goes on and every one is sent again with each request.

async def sampling_loop(
    # ...
):
    while True:
        # ...
            raw_response = client.beta.messages.with_raw_response.create(
                max_tokens=max_tokens,
                messages=messages,
                model=model,
                system=[system],
                tools=tool_collection.to_params(),
                # The toolset group has no beta flag, so on Vertex/Bedrock the
                # list can be empty; omit the header rather than send it blank.
                betas=betas or omit,
                extra_body=extra_body,
            )
        # ...
        response = raw_response.parse()

        response_params = _response_to_params(response)
        messages.append(
            {
                "role": "assistant",
                "content": response_params,
            }
        )
        # ...
        if not tool_result_content:
            return messages

        messages.append({"content": tool_result_content, "role": "user"})

An excerpt of sampling_loop in loop.py from anthropics/anthropic-quickstarts. The lines are the file's, and # ... marks the code left out. It was not run here.

Screen size and what to expect

Anthropic does not recommend sending screenshots larger than XGA (1024x768) or WXGA (1280x800), two standard screen sizes. Leaving the resizing to the API gives lower accuracy and slower responses than scaling in your own tool. For your own implementation it recommends XGA: scale a larger screen down and map the model's coordinates back to the original size, and pad a smaller screen with black until it reaches 1024x768.

Accuracy is the other limit. When Anthropic opened computer use in public beta in October 2024 it called the feature experimental, at times cumbersome and error-prone. The figures below are from that post. Two of them come from OSWorld, which Anthropic describes as evaluating AI models' ability to use computers like people do. They describe one model, Claude 3.5 Sonnet, that has since been replaced, and the sources read for this lesson give no newer benchmark, so test on your own task.

Check the result outside the screenshot

OpenAI's guide tells you to bound the run with step, time or cost limits, support cancellation, and check the actual outcome instead of relying on the model's final answer. The sample app README says the same in one line: a final answer does not prove the task succeeded.

OpenAI's recipes show the habit in code. A desktop demo has the model type a note into an editor and save it. The script then reads the saved file itself and raises an error if it does not match, with a comment that saving is asynchronous and the file should be checked independently of the screenshot.

Watch Anthropic's video

Anthropic put a video on using computer use to automate operations on its YouTube channel.

Common mistakes

• Sending full-resolution screenshots and trusting automatic resizing, which Anthropic says lowers accuracy.
• Running the loop with no step or time limit. OpenAI marks its skeleton as a sketch that leaves those to you.
• Running a sample in your everyday desktop session. OpenAI's Python app controls your real mouse and keyboard, and its README says to use a dedicated desktop session.
• Pairing a model with a tool version it was not trained for. Anthropic's README sends you to its docs for each model's supported versions.
• Believing the final message. Open data/sdk-booking.png or the saved file.
• Pointing a sample at a site where you are signed in. The OpenAI sample uses a fake lab and Anthropic recommends a container.

Try it yourself

• Run OpenAI's booking sample, open data/sdk-booking.png and compare the booking with your request.
• Run it again with a different --prompt and compare again.
• Start Anthropic's Docker demo, ask it to open Firefox, and watch the desktop view at http://localhost:6080/vnc.html.
• Add a counter to OpenAI's loop skeleton that stops after a number of turns you choose, and print how many turns a task used.

Next: keeping browser agents safe

Browser agent safety is lesson 4. It covers prompt injection, the controls both vendors tell you to add, and the habits that keep a browser agent from doing damage.

Frequently asked questions

Sources

1. OpenAI: Computer use guide (read November 6, 2026)https://developers.openai.com/api/docs/guides/tools-computer-use
2. OpenAI: Computer use integration recipes (read November 6, 2026)https://developers.openai.com/api/docs/guides/tools-computer-use-integration
3. openai/openai-cua-sample-app README (read November 6, 2026)https://github.com/openai/openai-cua-sample-app
4. anthropics/anthropic-quickstarts: computer use demo README (read November 6, 2026)https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo
5. anthropic-quickstarts: computer-use-demo loop.py (read November 6, 2026)https://github.com/anthropics/anthropic-quickstarts/blob/main/computer-use-demo/computer_use_demo/loop.py
6. anthropic-quickstarts: computer-use-demo computer.py (read November 6, 2026)https://github.com/anthropics/anthropic-quickstarts/blob/main/computer-use-demo/computer_use_demo/tools/computer.py
7. Anthropic: Computer use, Claude 3.5 Sonnet and Claude 3.5 Haiku (read November 6, 2026)https://www.anthropic.com/news/3-5-models-and-computer-use
8. Anthropic on YouTube: Computer use for automating operations (read November 6, 2026)https://www.youtube.com/watch?v=ODaHJzOyVCQ