locate-anything — visual grounding for exact coordinates

What it is. NVIDIA LocateAnything-3B (Qwen2.5-3B LM + MoonViT + MLP projector)
run through mudler/locate-anything.cpp — an MIT C++/ggml port with prebuilt
GGUFs, no Python at inference. It does grounding, not captioning: give it an image
and an open-vocabulary phrase; it returns labeled boxes. 8sync locate wraps it and
adds a click-center per box so you can drive the browser tool straight to a target.

When to reach for it (vs. the alternatives):

Self-check: if the next action is a click / crop / measure at a pixel, you want a
BOX → 8sync locate. If it's "understand the content", look yourself (GLM-5.3) or zai-vision (GLM-5.2 only).

Setup (once)

8sync locate --setup   # clones + cmake-builds locate-anything.cpp, downloads q8_0 GGUF (~6.3 GB)

Builds with CUDA if the toolkit is present, else CPU (still ~1.7–3× faster than the
official PyTorch path). Artifacts live under ~/.cache/8sync/locate-anything/.

Use

# Ground a UI element → box + click point
8sync locate ui.png "the Sign in button"
#   → 1 detection(s):
#     the Sign in button   box [812, 40, 940, 78]  click≈(876, 59)

# Multi-category (separate with </c>)
8sync locate street.jpg "person</c>car"

# Draw the boxes for a visual check
8sync locate ui.png "the search field" --annotated /tmp/boxed.png

# Decode mode: hybrid (default, Parallel Box Decoding) | slow | fast
8sync locate ui.png "submit" --mode fast

The high-value pipeline: shot → locate → click

8sync shot http://localhost:3000 -o /tmp/ui.png
8sync locate /tmp/ui.png "the primary CTA button"    # → click≈(cx, cy)

Then in the browser tool, click at those coordinates (map back to the live viewport
if the shot used a different width). This closes the loop that pure vision-description
can't: see → locate → act, saving the round-trips of guessing selectors.

Notes / limits

• License: the LocateAnything-3B weights are NVIDIA research / non-commercial use
  only. The locate-anything.cpp code is MIT. Respect the model license.
• Single-image, greedy decoding by design (the port drops stochastic sampling and
  multi-image, which don't have parity-checked detection semantics).
• Coordinates are in the analyzed image's pixel space — rescale if the screenshot
  viewport differs from the live page.
• Prompts are open-vocabulary; a concrete noun phrase ("the red delete icon") grounds
  better than a vague one.