What is a computer-use agent?

A computer-use agent operates a screen the way a person does: it looks at screenshots or the page, then clicks, types and scrolls. It can use any app, and it reads everything on the screen as input.

How it works

The agent gets a view of the screen (a screenshot, or the page's structure in a browser) and a small set of actions: click, type, scroll, press a key, open an address. It loops: look, act, look again. No integration is needed, which is the point. Whatever a person can do in the interface, the agent can.

Why it is harder to secure

  • Everything on screen is input. An email, an ad, a comment or hidden text on a page can address the agent directly. This is indirect prompt injection with a very large surface.
  • The session is the user's. A browser agent is signed in wherever the user is: mail, bank, admin panels.
  • Actions are low level. "Click at 412, 300" says nothing by itself. The meaning is in the page, so a check has to read both.

What helps

  • A separate browser profile with only the accounts the task needs.
  • An allowlist of sites.
  • Confirmation before purchases, sends, deletions and changes to settings.
  • A check on each action in its context, not only on the final answer.

With White Circle

Control enforces policies on what an agent does as well as on what it writes, and can hold an action for confirmation.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL