vellum-browser-use

Use this skill to browse the web. All browser operations are executed through the assistant browser CLI, invoked via bash or host_bash. Each operation is a subcommand:

Command	Description
assistant browser navigate	Navigate to a URL
assistant browser snapshot	List interactive elements on the current page
assistant browser screenshot	Take a visual screenshot
assistant browser click	Click an element
assistant browser type	Type text into an input
assistant browser press-key	Press a keyboard key
assistant browser scroll	Scroll the page or a specific element
assistant browser select-option	Select an option from a native <select> element
assistant browser hover	Hover over an element to reveal menus/tooltips
assistant browser wait-for	Wait for a condition
assistant browser extract	Extract page text content
assistant browser wait-for-download	Wait for a file download to complete
assistant browser fill-credential	Fill a stored credential into a form field
assistant browser attach	Attach the Chrome debugger to the active tab
assistant browser detach	Detach the Chrome debugger from the active tab
assistant browser close	Close the browser page
assistant browser status	Diagnose browser backend readiness and setup steps

Getting Started — Check Browser Readiness

Before using any browser commands, run assistant browser --json status first to check which browser backends are available. The status command returns JSON with readiness information for each backend mode:

assistant browser --json status

The response includes:

• recommendedMode — the best available backend (use this)
• modes[] — per-mode status with available, summary, and userActions (remediation steps)

For --virtual-desktop, Chrome and desktop components are included in the assistant image. Browser and computer use start the existing desktop without installing dependencies. If components are missing, report the image problem; do not install packages or launch a separate Chrome process.

Browser Modes

Use --browser-mode <mode> on the assistant browser parent command to pin the browser backend:

Value	Backend	Description
auto	Automatic	Default. Picks the best available backend based on context.
extension	Chrome extension	Routes through the user's Chrome browser via the extension debugger.
cdp-inspect	CDP inspect	Connects to an already-running Chrome instance via DevTools Protocol.
local	Playwright	Drives a dedicated Playwright-managed Chromium instance.

assistant browser --browser-mode extension navigate --url http://www.example.com

Prefer the Chrome Extension

The Chrome extension (extension mode) is the preferred browser backend. It is:

• More secure than Chrome's native remote debugging
• Uses the user's real browser profile (cookies, sessions, saved logins)
• Best experience for interacting with the user's actual browsing context

If the status check shows the extension is not available, encourage the user to install and pair it:

1. Install the Vellum Assistant Chrome Extension from the Chrome Web Store: https://chromewebstore.google.com/detail/vellum-assistant-browser/hphbdmpffeigpcdjkckleobjmhhokpne
2. Open the extension in Chrome and pair it with the assistant.

The status response's userActions array for the extension mode provides these same steps when the extension is not connected.

When a Page Cannot Be Reached

If navigate, curl, or any fetch times out, hits an auth wall, or cannot reach a host (VPN, company login, internal dashboard):

1. Tell the user a connected desktop app or Chrome extension can open the page in a browser where they are already logged in.
2. Give the install links:
   • Desktop app: https://www.vellum.ai/downloads
   • Chrome extension: https://chromewebstore.google.com/detail/vellum-assistant-browser/hphbdmpffeigpcdjkckleobjmhhokpne
3. Offer those first. Only ask for a screenshot or pasted page content if they cannot install either.
4. On iOS or Android there is no in-app browser and no extension to install on the phone. Offer the desktop app or Chrome extension on a computer. Do not describe a browser panel.

Fallback Modes

If the user declines to install the extension:

• cdp-inspect — Connects to an already-running Chrome instance via DevTools Protocol (Chrome 146+). Requires enabling remote debugging in Chrome settings.
• local — Drives a dedicated Playwright-managed Chromium instance. Last resort — does not use the user's browser profile.

Only fall back to these if the user explicitly indicates they do not want to install the extension. Prefer cdp-inspect over local.

Targeting a Specific Client

When multiple clients support host_browser (e.g. two Chrome profiles, a macOS client and a Chrome extension), use --target-client-id <id> on the assistant browser parent command to pin all operations in the invocation to one specific client:

assistant browser --target-client-id <client-id> navigate --url https://example.com

Obtain client IDs from:

assistant clients list --capability host_browser

Omit --target-client-id when only one client is connected — the default interface-preference order (chrome-extension first, then macos) picks the best available client automatically.

Tab Handling (Chrome extension)

On the Chrome extension backend, navigate opens a dedicated tab the first time it runs in a conversation and pins subsequent operations to it, so browsing never disturbs the tab the user is on (often the tab they're chatting with the assistant from). Later navigates reuse that pinned tab.

• --new-tab — force a brand-new tab even when one is already pinned.
• --use-active-tab — navigate the user's currently-active tab instead of a dedicated one.

Both flags are ignored on the local and cdp-inspect backends, which manage their own browser context.

Session Management

Use --session <id> on the assistant browser parent command to group sequential operations so they share browser state (same page, cookies, etc.). Different session IDs create independent browser contexts.

assistant browser --session myflow navigate --url https://example.com
assistant browser --session myflow snapshot
assistant browser --session myflow click --element-id e3

Omitting --session uses the default session.

Machine-Readable Output

Use --json on the assistant browser parent command to get structured JSON output suitable for parsing in scripts:

assistant browser --json navigate --url https://example.com
# {"ok":true,"content":"Page title: Example Domain"}

assistant browser --json snapshot
# {"ok":true,"content":"...element list..."}

assistant browser --json screenshot
# {"ok":true,"content":"...","screenshots":[{"mediaType":"image/jpeg","data":"<base64>"}]}

Error responses use {"ok":false,"error":"..."}.

Screenshots

To save a screenshot to disk, use --output <path>:

assistant browser screenshot --output page.jpg
assistant browser screenshot --full-page --output full.jpg

To receive base64 screenshot data in JSON output:

assistant browser --json screenshot

The response includes a screenshots array with mediaType and data (base64) fields.

Typical Workflow

1. assistant browser --json status to check backend readiness — if the extension is not available, help the user install it
2. (Optional) assistant browser attach to establish the session
3. assistant browser navigate --url <url> to load a page
4. assistant browser snapshot to discover interactive elements
5. Use click, type, press-key, scroll, select-option, or hover to interact
6. assistant browser extract or assistant browser screenshot --output <path> to capture results
7. Always assistant browser detach when you are done — this releases the debugger so the user can browse freely

Human verification

Treat every CAPTCHA and bot-detection challenge as a request for human help, including drag-to-verify sliders, press-and-hold checks, verification checkboxes and image puzzles. Stop before interacting with the challenge, even if its controls look easy to automate. Do not try it yourself, retry it, or script a solution.

In the virtual desktop, request the desktop-help card immediately, using one short sentence for the needed action. Wait for Done or Skip. After Done, take a fresh snapshot; if verification remains, ask for help again. On other browser backends, ask the user to complete verification in their browser and wait for confirmation.

For ordinary logins, use saved credentials or securely prompt for missing credentials, then fill the form yourself. A CAPTCHA on a login page still requires human help.

Interaction Strategies

Date pickers / calendars: Click the date input to open the picker, re-snapshot to see calendar controls, click month navigation arrows to reach the target month, then click the target date. For <input type="date">, use type with YYYY-MM-DD format.

Native <select> elements: Use select-option with --value, --label, or --index. Do not try to click individual <option> elements.

ARIA / custom dropdowns: Click to open, take a new snapshot, then click the desired option by --element-id.

Autocomplete inputs: Type the search text, wait 500-1000ms (wait-for --duration), re-snapshot for suggestions, then click the suggestion or use press-key --key ArrowDown + press-key --key Enter.

Multi-step forms: Complete each step, wait for the next section to load, re-snapshot to discover new elements, then proceed.

Dynamic content: After interactions that change the page, use wait-for (with --selector or --text) or re-snapshot to see updated elements before continuing.

Scrolling: Use scroll --direction down to reveal below-the-fold content before snapshotting. Long pages may require multiple scrolls.

Hover menus / tooltips: Use hover to reveal hidden menus or tooltips, then re-snapshot to see newly revealed elements.

Verification

After critical actions (form submission, booking confirmation, checkout), take a screenshot and then read the saved image to visually verify results before reporting success to the user:

assistant browser screenshot --output /tmp/verify.jpg

Then read the saved image to inspect it before reporting success. Use file_read if the screenshot was taken via bash, or host_file_read if it was taken via host_bash.