SAM 3 (facebookresearch/sam3)

Overview

Build, integrate, fine-tune, and evaluate Meta SAM 3 with reproducible commands and minimal setup friction.

Quick Routing

Core Workflow

1. Confirm objective and modality.
2. Set up environment and checkpoint access.
3. Run a smoke test.
4. Execute the task path: inference, training, or evaluation.
5. Return reproducible commands and file paths.

1) Confirm objective and modality

• Identify whether the user needs image inference, video inference, fine-tuning, or benchmark evaluation.
• Confirm whether CUDA is available and which GPU memory budget applies.
• Confirm whether Hugging Face access to facebook/sam3 is already approved.

2) Set up environment and checkpoint access

Use a clean environment:

conda create -n sam3 python=3.12 -y
conda activate sam3
pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

Authenticate with Hugging Face before first model load:

hf auth login

Optionally run a preflight check before model download:

python scripts/sam3_preflight_check.py --strict

For full setup and verification commands, read references/setup-and-inference.md.

3) Run a smoke test

From this skill folder, generate a starter script:

python scripts/create_inference_starter.py --mode image --output ./sam3_smoke.py

Edit placeholders and run from a SAM3 checkout.

4) Execute the task path

• Image and video inference:
  Use references/setup-and-inference.md to apply Sam3Processor and build_sam3_video_predictor patterns.
• Fine-tuning:
  Use references/training-and-eval.md and start from a provided config in sam3/train/configs.
• Benchmark evaluation:
  Use references/training-and-eval.md plus upstream dataset notes under scripts/eval/gold, scripts/eval/silver, and scripts/eval/veval.

5) Return reproducible output

• Report exact commands run and any config overrides.
• Include checkpoint source and authentication assumptions.
• Include prompt text, frame index, and confidence threshold when reporting inference outputs.

Guardrails

• Do not assume checkpoint access is granted; verify login and permission first.
• Prefer official sam3.model_builder and predictor APIs over custom re-implementations.
• Keep generated scripts editable and avoid machine-specific absolute paths.
• If running on CPU, explicitly note expected performance limits before large jobs.

Resources

• Setup and inference guide: references/setup-and-inference.md
• Training and evaluation guide: references/training-and-eval.md
• Starter generator: scripts/create_inference_starter.py
• Preflight checker: scripts/sam3_preflight_check.py