macOS Desktop Control (no MCP)

Drive the real macOS GUI through Shell. No MCP, no node. Uses system screencapture /
osascript + cliclick (Homebrew). For browser tasks, use Cursor's browser tools — they're
faster and need no screenshots.

Permissions (one-time)

Cursor needs, in System Settings → Privacy & Security: Screen Recording (screenshots) and
Accessibility (clicks/keys). If a command silently no-ops or the screenshot is empty, a
permission is missing — tell the user to grant it and restart Cursor.

Efficiency rules (fast + low token)

Reading images is the dominant cost. So:

1. One screenshot per checkpoint, not per action. Verify after navigation, before
   irreversible actions, and at the end — not after every click. Trust deterministic steps.
2. Batch a whole sub-task into ONE shell call, ending in one screenshot.sh. E.g.
   app.sh X && click.sh … && <clipboard> && key.sh cmd v && key.sh return && screenshot.sh.
   This is also what keeps focus stable (see Focus below).
3. Click straight from the fraction estimate for big/clear targets. Use the
   move→zoom loop only for small or closely-spaced targets.
4. Prefer keyboard / menu / AppleScript over clicking — zero screenshots, e.g. ⌘K,
   key.sh return, osascript -e 'tell application "Safari" to open location "…"'.
5. Screenshots are auto-downscaled JPEG (~1280px). Coords are fractional so low res is fine;
   bump MACOS_CONTROL_MAXW only to read tiny text, else use zoom.sh.

Coordinate conversion (critical)

click.sh/move.sh take logical points. screenshot.sh prints logical_size=WxH
(e.g. 1470x956) — the conversion basis. Convert by FRACTION, never by raw pixels: the
image you read is downscaled to a variable width, so assuming a fixed width causes a
systematic offset.

fx = target_x_in_image / image_width      # estimate visually, 0..1
fy = target_y_in_image / image_height
click_x = fx * logical_width              # logical_width from logical_size
click_y = fy * logical_height

Precision loop (small targets only)

1. screenshot.sh + Read     -> fraction estimate (X,Y)
2. move.sh X Y              -> move cursor, no click
3. zoom.sh X Y             -> high-detail region WITH cursor; is it on target?
4. off? recompute via zoom's map: line, repeat 2-3
5. click.sh X Y            -> click once confirmed

Focus (avoid wrong-window clicks)

Clicks/keys go to the frontmost app. Anything (notifications, an approval card you must
click, switching back to Cursor) can steal focus, sending your input to the wrong app. So:

• Start every action command with open -a "<App>" (or app.sh) to guarantee the target
  is frontmost, and keep the whole action in ONE shell call so focus can't drift mid-task.

Launching apps (don't hang the shell)

• Launch via open -a "<App>" only. Never run an app's binary directly and never pipe a
  long-running/GUI process (e.g. app & ... | head) — it blocks the shell, gets backgrounded,
  and corrupts the session's later output. To launch, open -a returns immediately; then poll
  readiness with short sleep + context.sh (check frontmost_app) rather than a blocking wait.
• Multi-process (Electron) apps take a few seconds to register and may need a second open -a
  to come to the front; confirm with context.sh before acting, not by guessing.

Reliable input gotchas (learned the hard way)

• Enter/Return: use key.sh return (it uses osascript key code 36). cliclick kp:return
  is frequently ignored by apps.
• Chinese / non-ASCII text: never type it; set the clipboard and paste:
  osascript -e 'set the clipboard to "你好"' && key.sh cmd v.
• "Enter doesn't send" in chat apps: an autocomplete popup may eat it, or the app uses
  ⌘Enter, or the input lost focus. Fallbacks: re-open -a + click input + key.sh return;
  try key.sh cmd return; or click the send button (verify its exact spot with zoom.sh —
  it often sits next to a schedule-send dropdown).
• Electron apps (WeChat, Feishu, Slack, VS Code) expose no usable AX tree, so element-by-name
  location fails — use screenshots + the precision loop.

Scripts

In scripts/ (auto-locate cliclick even if not on PATH).

• context.sh — frontmost app, window title, mouse point, logical_size.
• screenshot.sh [path] — downscaled JPEG of the screen; prints path + logical_size.
• move.sh X Y — move mouse, no click.
• zoom.sh X Y [W H] — high-detail JPEG around a point (cursor shown); prints a map: line.
• click.sh X Y [single|double|right] — click at logical points.
• type.sh "text" — type literal ASCII into the focused field (use clipboard for non-ASCII).
• key.sh [cmd ctrl alt shift] KEY — keys/shortcuts via osascript (return, cmd c, cmd shift 4…).
• app.sh "Name" — activate/focus an app.

Safety

• This controls the real mouse/keyboard. State each action before doing it.
• Confirm before destructive actions (deleting, closing unsaved work) and before messaging real people.
• Never type secrets the user hasn't provided.