Design an evaluation suite

Guides the consumer from "I want to evaluate X" to committed artifacts: a success-criteria document
and a graded eval suite. Method follows Anthropic's official evaluation guidance — load
/evals:methodology reference files as each phase needs them (they carry the distilled source).

Arguments

$ARGUMENTS names the target. Two shapes:

• app (or a path/description of an LLM-powered feature) — evals for the consumer's own
  LLM-based application behavior.
• skill <name> — evals for a consumer-authored Claude Code skill, emitted as
  evals/evals.json next to that skill.

No argument → ask which target, with one example of each.

Phase 1 — success criteria (before any cases)

Interview until each criterion is specific, measurable, achievable, relevant
(success-criteria.md):

1. What does success look like, concretely? Reject unmeasurable phrasings by proposing a
   measurable rewrite ("good answers" → "≥90% of answers judged correct against their rubric").
2. Which dimensions matter? Walk the eight (fidelity, consistency, relevance/coherence,
   tone/style, privacy, context use, latency, price); keep the ones with a real user need. Most
   targets are multidimensional — press for at least fidelity plus one guardrail dimension.
3. What is achievable? Anchor each target to a baseline (current behavior, prior experiment, or a
   published benchmark); when no baseline exists, record the first run AS the baseline.

Write the result to docs/eval-criteria/<target>.md in the consumer repo (create the directory if
absent; respect an existing consumer convention for criteria docs if one is documented in the
consumer's own CLAUDE.md or rules). Each criterion: dimension, metric, target number/scale,
rationale line.

Phase 2 — eval suite

Per criterion, pick the cheapest reliable grading method
(grading.md, recipes.md):
code-graded where the output can be constrained to allow it; LLM-graded with a tight rubric and
constrained verdict otherwise; human grading only with stated justification.

Case authoring (eval-design.md):

• Mirror the target's real input distribution; include edge cases explicitly — irrelevant or
  nonexistent input, overly long input, poor/harmful/irrelevant user input for chat surfaces,
  ambiguous cases.
• Every case carries a golden answer: an exact answer for code-graded cases, rubric-instructions
  for LLM/human-graded cases.
• Draft a baseline set by hand with the consumer, then offer to generate more cases from it —
  volume over polish — and have the consumer review the generated batch before it lands.

Target = app: scaffold evals/<target>/cases.jsonl (one JSON object per case: id, input,
golden_answer, grading (exact|string_match|llm_rubric|human), optional rubric) plus a
README.md documenting how the consumer's own tooling should run and grade them, with the grader
prompt skeleton from grading.md inlined for llm_rubric
cases. Honor an existing consumer eval layout when one is already present — extend, don't rename.

Target = skill: emit <skills-root>/<skill>/evals/evals.json in this shape — skill_name,
evals[] of {id, name (kebab-case), prompt, expected_output, expectations[]} — covering
trigger/routing, the happy path, at least one refusal/guardrail, and one anti-pattern the skill
must not exhibit. When the skill-quality plugin is installed, validate with
/skill-quality:check validate-evals <skill> (its bundled schema is the contract);
otherwise state that the file follows the marketplace's evals schema and validation was skipped.

Phase 3 — grading hygiene gate

Before finishing, confirm and record in the criteria doc:

• LLM-graded cases name a grader model DIFFERENT from the generator, constrain the verdict format,
  and instruct reasoning-then-discard.
• The consumer's first act is to sample-check grader verdicts against their own judgment before
  trusting the suite at scale.
• Re-run cost is stated (which cases are code-graded and free vs LLM-graded and metered).

What this skill does NOT do

• Does not execute evals. No marketplace command runs model-graded evals; running the suite is
  the consumer's tooling (for Claude Code skills, Anthropic's skill-creator plugin can run skill
  evals when installed).
• Does not overwrite an existing criteria doc or eval suite without showing the diff and getting
  explicit confirmation.
• Does not invent baselines — a target with no anchor is recorded as provisional.

Gotchas

• A consumer saying "just write some tests" still gets Phase 1 — criteria first is the method, not
  a preference; keep it to the few questions that unblock measurable targets.
• Refuse to emit an eval case with no golden answer or rubric — a case that can't be graded is not
  an eval.