LLM evaluation methodology

Distilled from a cover-to-cover reading of Anthropic's "Define success criteria and build
evaluations" (https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) and its linked
evals cookbook (anthropics/claude-cookbooks misc/building_evals.ipynb), fetched 2026-08-08.
Reference files carry per-file source stamps; re-fetch the source page for runnable code or when a
specific must be current.

Routing table

Load the most relevant file first; a second only if the first doesn't fully answer.

Quick decision guide (no file load needed):

• "Where do I start?" → Define measurable success criteria first; evals test against them; only
  then iterate on prompts.
• "Is this criterion good?" → It names a specific quality, a number or defined scale, a realistic
  target, and ties to a user need. "Good performance" fails all four.
• "Which grading method?" → The fastest, most reliable, most scalable that fits: code-based if the
  output can be constrained to allow it; LLM-graded for judgment; human only as a last resort.
• "Can I automate this seemingly subjective eval?" → Usually — constrain the output format,
  reformat to multiple choice, or use an LLM grader with a tight rubric and constrained verdict.
• "How many cases?" → Prefer volume with automated grading over a few hand-graded showpieces;
  generate more from a baseline set with Claude, human-reviewed.
• "Can I trust my LLM grader?" → Only after reading samples of its verdicts against your own
  judgment; and grade with a different model than the one that generated the output.
• "One metric or several?" → Several — most use cases need multidimensional criteria (fidelity +
  safety + latency + cost); a single headline metric hides regressions.

Maintainer update action

/evals:methodology update — maintainer-only drift check: re-fetch the source page (raw markdown)
and the cookbook notebook, diff against the four reference files, apply content corrections, and
refresh every "fetched YYYY-MM-DD" stamp with the new date. Consumers never need this; it exists
because this skill distills a live upstream doc.

Scope boundary

This skill is knowledge (WHY/WHAT of evaluation design), not workflow. It never runs,
scores, or scaffolds evals. To interview for criteria and scaffold an eval suite in your repo, use
/evals:design. To statically validate a Claude Code skill's eval file, use
/skill-quality:check validate-evals when the skill-quality plugin is installed. No
marketplace command executes model-graded evals.

Gotchas

• The reference files are a distillation with fetch-date stamps, not the source: for runnable
  recipe code or any load-bearing specific, re-fetch the source page — its code samples and model
  names move with releases.
• Do not "verify" a claim about the guidance against this skill's own spokes; the spokes ARE the
  derived copy. Verification means fetching the upstream page.