Computer Use & Browser Automation

Computer use means the agent acts through the same user interface a person would use: reading screens, clicking buttons, filling forms, and navigating browser or desktop applications. This is most useful when no stable API exists or when the workflow spans multiple human-oriented tools.

Good fits include:

• legacy internal systems with no programmatic interface
• cross-application workflows that depend on browser navigation
• operator-assist systems where a human stays nearby and can intervene

Bad fits include stable, high-volume workflows that already have good APIs. UI acting is usually more fragile and slower than direct integrations.

Browser and desktop agents fail differently from API-driven agents:

• UI drift — labels, layouts, or selectors change.
• Ambiguous affordances — multiple similar buttons or fields appear.
• Hidden state — popups, auth modals, and partial page loads interrupt the flow.
• Observability gaps — the system knows what it clicked, but not always why it failed.

Because of this, UI-acting agents require stronger recovery logic and tighter permissions than many API-based workflows.

Safe computer-use systems usually add more constraints than standard tool-calling systems:

• Scope the environment — isolated browser profiles, sandboxed desktops, or limited app sets.
• Limit reachable actions — allow only approved domains, screens, or action classes.
• Insert approval checkpoints — especially before sends, purchases, deletes, or irreversible changes.
• Capture screenshots and action traces — so failures can be audited and debugged.
• Prefer read-only or draft-first modes where possible.

if action.type in {"purchase", "delete", "send"}:
  require_human_approval()

if page.domain not in approved_domains:
  block("Unapproved domain")

A practical decision rule:

• Use APIs first when they exist and are stable.
• Use browser automation when the workflow must operate across human-only surfaces.
• Use computer use with humans nearby when the environment is high-value but brittle.

UI acting is powerful because it reaches places APIs do not. It is risky for the same reason.