When an agent uses a browser, what is it actually doing?

Products such as Grok Bot say they sign in to your tools, use them as you would, and repeat a workflow after watching you do it once. Yet a language model only reads and writes text (The harness is a program on your machine). So how does text click a button?

The answer is plainer than the pitch. xAI doesn't publish how Grok Bot works inside, so the example here is the way this kind of agent is generally built, with tools anyone can install. It starts from something much older than agents: a script⁵⁰.

A script is the commands you would type, kept in a file (A script is text another program reads). A program can drive a browser the same way. Playwright⁸¹, a library from Microsoft built for testing websites, lets a few lines of Node open Chromium, go to a page and click a button.

Each of those lines was chosen before the script ran. That makes a script free to run again, and it makes a script break when the page changes, because it has no way to write a different line.

An agent keeps the browser and the library, and replaces the list of lines with a loop. Look at the page, ask the model for the next action, carry it out, and go round again. The harness⁰³ that runs this loop is ordinary code, like the script was. The only new part is where each line comes from.

- heading "Sign in to Zendesk"
- textbox "Email" [ref=e12]
- textbox "Password" [ref=e13]
- button "Sign in" [ref=e14]
- link "Forgot password?" [ref=e15]

So the model never touches the browser. It writes a line of text, usually as a tool call⁴¹, and the harness does the rest. That is why the browsers built for agents aren't a new kind of browser.

Chrome could always be scripted. What changed is the author: a model at run time, not a developer in advance.

For this to work, the model has to be given the page in a form it can read. A web page in a browser is a tree of elements, the DOM⁸², and Chromium lets other programs ask for it over a debugging channel, the Chrome DevTools Protocol⁸³. Raw HTML would be enormous and mostly about appearance. So harnesses usually ask for an accessibility snapshot⁸⁴: what each thing is and what it is called, the view a screen reader gets.

- heading "Issues" [level=1]
- link "New issue"
- link "Open (36)"
- link "Closed (18,369)"
- button "Filter by author": Author
- button "Filter by labels": Labels

Where there is no tree to read, as in a native app or a drawing on a canvas, the harness sends a screenshot to a model that can read images, and the reply is a pair of coordinates. Screenshots can use fewer tokens than text, but a click aimed at a coordinate is easier to get wrong than a click on a named element. So many harnesses prefer the text, and some send both.

A page flattened into text, a model that picks from it, and a while loop. That is the whole mechanism.

Here is where the idea pays off. The script is the model's output, and output is tokens⁵⁹. A click is about a dozen of them. So a 40-step job is 40 short replies, billed like any others. Grok Bot's own pricing says so: weekly usage is included, and anything over is "billed based on token cost".

The output is the small part. On every turn the harness also sends the goal, the page and what has happened so far, and all of that is input. A job's cost is roughly the number of turns times the size of the page, which is why a long run on a busy page costs far more than three shell commands, though you'd call both one task.

Harnesses are shaped by this. They flatten pages. They replace old history with a summary (the context window⁶⁰ is finite too). And they prefer a single command to five pages of clicking.

A script is written once and costs nothing to run again. An agent's script is written again on every run, and you pay for the writing.

Now "show it once" has a plain reading. Logging your clicks, keystrokes and navigations gives you a real, replayable script. Test tools have done this for years: npx playwright codegen opens a browser and writes code as you click. Replayed as it stands, that recording is cheap, and it breaks the way any script breaks.

A model can take the same log and write down what you were doing: filter the refund tickets, export them, add them to the sheet. Stored like that, the steps are goals, and the loop can find the button wherever it has moved. Each step then costs tokens again. The middle way is to replay the recorded action and call the model only when it stops matching, which is what action caching in tools such as Stagehand does. xAI doesn't say which of these its routines use. Either way, a timer such as cron⁸⁶ sets it off, the same kind of thing that has run scripts on a schedule since the 1970s.

The script says exactly what. The routine says what for, and the model works out how, each time it has to.

If every turn costs a page, the cheapest job has the fewest turns. Most services offer a way that has none: an API (Why the browser needs an API), a URL that returns data. Use it when it exists and you can get access to it. It takes one step, and its reply keeps its shape when the website is redesigned. Its reply can still be big, so have a script keep the fields you need before the model gets them.

It can also be safer. Many services let a key or an OAuth grant be limited, to reading tickets for example, whereas a logged-in browser session can do whatever your account can. And a page is untrusted text going to the same model that holds your session, so words hidden in a page can be taken as orders. That is prompt injection⁸⁷, and Brave showed it against Perplexity's Comet browser.

The browser earns its cost when there is no API, or when there is one you can't get at: an internal tool from years ago, a supplier's portal, a plan you aren't on. Check the site's terms first, because some ban automation outright, as LinkedIn's do. Most real routines mix them: an API for the data, a plain script for the sums, the browser for the one dashboard nothing else reaches.

Is there a URL that returns JSON for this? If yes, use that. If no, click.