Eval Harness

Activate when a behavior, agent, prompt, or workflow needs a repeatable semantic
evaluation. Explaining an existing assertion or running an already-owned test does not
require a new eval suite.

Preflight

Before creating or running an eval, resolve the observable contract, current baseline,
inputs, expected material artifact, failure boundary, evaluation owner, and publication
destination. Missing behavior or success criteria stops before an eval artifact is
created; never invent a baseline.

Minimal suite

• A capability scenario proves the newly required material result.
• A regression scenario proves behavior that must remain unchanged.
• A fail-closed scenario proves the first causal failure and zero effects.
• A should-not-trigger scenario proves absence of activation and fallback.

Use a deterministic grader whenever the result can be checked by code. Use a model or
human grader only for a stated semantic judgment, with an explicit rubric. Keep
definitions, fixtures, graders, and baselines under the project's declared evaluation
owner; do not create a parallel hierarchy.

Execution and evidence

Run each required scenario through the repository's evaluation facade. Preserve child
nonzero status, timeout, signal, exception, and causal chain. Do not retry a failed
required trial, substitute another model or grader, or convert failure to a warning,
skip, neutral result, or later success.

pass@k may describe measured reliability; it never makes a failed required trial
green. pass^k requires every named trial to pass. Publish one complete report only
after all required evidence is available. The report names the exact command, exit
status, decisive output, artifacts, and each scenario result; a partial report or stale
baseline cannot support completion.

Remove temporary inputs and superseded reports through the evaluation owner. Completion
requires the material artifact, regression proof, causal failure proof, zero effects,
and no unowned residue.