Browser Automation — Tool Selection & Usage

Three tools for browser/desktop automation. Each has a different approach; the right choice depends on whether you need the user's session, a clean environment, or native desktop access.

---

1. Decision Tree

Need the user's logged-in browser (cookies, sessions)?
  ├── YES → browser-skill (bsk CLI + extension)
  │          Uses user's real Chromium with their logins.
  └── NO  → Need to drive native desktop apps (Finder, Mail, Figma)?
              ├── YES → computer-use (cua-driver desktop GUI)
              └── NO  → agent-browser (Rust CLI, standalone Chrome via CDP)
                          Fast, isolated, no extension needed.

Quick comparison

Criteria	agent-browser	browser-skill	computer-use
Browser type	Standalone Chrome (CDP)	User's real Chromium	Any desktop app
Needs extension	No	Yes (browser-skill ext)	No
Needs user session	No	Yes (user's cookies)	N/A
Speed	Fast (Rust)	Medium (IPC)	Slow (GUI)
Best for	Scraping, testing, CI	User-account-required sites	Desktop apps, Electron

---

2. agent-browser — Standalone Browser Automation

Fast Rust CLI using Chrome/Chromium via CDP. No user session, no extension needed.

Install

npm i -g agent-browser && agent-browser install

Workflow

# Start, navigate, interact
agent-browser skills get core    # full workflow reference

Key features

• Accessibility-tree snapshots with @eN element refs
• Sessions, auth vault, state persistence, video recording
• Observability dashboard on port 4848
• Electron app support via agent-browser skills get electron

When to use

• Scraping public sites
• Testing web apps in isolation
• CI/CD browser tasks
• Tasks that don't need user login

---

3. browser-skill — User's Real Browser

Uses the bsk CLI + browser-skill extension to drive the user's real Chromium with their logins and cookies. Actions happen in an isolated Agent Window.

Prerequisites

1. bsk on PATH (Rust CLI)
2. browser-skill extension loaded in Chromium and connected
3. bsk doctor if anything fails

Mandatory lifecycle

bsk session start                # → prints 4-letter session id
bsk navigate <url> --session <id>
bsk snapshot --session <id>      # → aria tree with @e1, @e2, … refs
bsk click @e3 --session <id>
bsk session stop <id>            # REQUIRED when done

Core interaction pattern

bsk navigate <url> --session <id>
bsk snapshot --session <id>      # first choice: accessibility tree
bsk get-html --session <id>      # only when snapshot insufficient
bsk screenshot --session <id>    # only when visual needed
bsk click @eN --session <id>     # or fill, select, press
bsk evaluate <js> --session <id> # JS in agent tab

When to use

• Sites behind login (user's cookies available)
• Regression-testing a UI change
• Forms, multi-step flows, smoke tests
• Any page the user can already access

Safety rules

• Never bsk evaluate on banking/SSO/password-manager pages for credential extraction
• Always return borrowed tabs: bsk tab return <tab-id>
• Always stop session when done

---

4. computer-use — Desktop GUI Automation

Drives the user's desktop in the background (no cursor steal, no focus grab) via Hermes computer_use tool. Works for any native app: Finder, Mail, Figma, native chat clients, terminal emulators, games.

Canonical workflow

computer_use(action="capture", mode="som", app="Chrome")
  → screenshot + numbered overlays + AX tree with @1, @2, …

computer_use(action="click", element=7)
  → click by element index (most reliable)

computer_use(action="click", element=7, capture_after=True)
  → click + verify in one call

Available actions

Action	Parameters
capture	`mode=som
click	element=N or coordinate=[x,y], `button=left
double_click	element=N or coordinate=[x,y]
scroll	`direction=up
type	text="…"
key	keys="ctrl+s", "return", "escape"
drag	from_element=N, to_element=M or from_coordinate/to_coordinate
wait	seconds=N
list_apps	Returns running apps
focus_app	app="…", raise_window=false

When to use

• Native desktop apps (Finder/Explorer, Mail/Outlook, Figma, native chat)
• Desktop workflows that need actual GUI interaction
• When you need to interact with non-web content

When NOT to use

• Web automation possible via agent-browser or browser-skill — prefer those
• File edits — use read_file/write_file instead
• Shell commands — use terminal instead
• Never raise_window=True unless user asked

---

5. Quick Reference

Need	Use
Visit a public URL, scrape data	agent-browser
Visit a site behind login	browser-skill (user's browser)
Test a UI change	browser-skill (user's browser)
Click around a desktop app	computer-use
Take a screenshot of a page	agent-browser or browser-skill
Fill a form on a logged-in site	browser-skill
Automated tests in CI	agent-browser