Long (Restart-Durable) Eval

Role: worker + supervisor. Runs a checkpointed eval that resumes across
container recycles, and keeps it alive until it finishes.

Why this exists

A cloud session runs in a container that is recycled on a short cadence
(often every few minutes). A recycle kills every running process but keeps
the working tree. So any eval longer than one container lifetime must:

1. Checkpoint to disk at fine granularity, and resume from the checkpoint
   rather than restarting — otherwise every recycle throws the work away.
2. Be relaunched after each recycle. The session itself is the only
   supervisor: each recycle re-invokes it ("Continue from where you left
   off"), which is the relaunch trigger — tighter than any timer.

This skill ships both halves so you don't rebuild them:

File	Role
durable_runner.py	The engine — grades one cell at a time, checkpoints after every sample, resumes at the next unbanked sample. Import it or run via the CLI.
run_eval.py	CLI supervisor — run, status, and ensure-alive (the guard-relaunch primitive).

${CLAUDE_PLUGIN_ROOT} below is the plugin root; from this repo it is
plugins/dev-team.

The eval module contract

An eval module is a plain Python file supplying only the eval-specific
bits. Everything durable is handled for you.

SAMPLES = 5           # optional: trials per cell (majority vote)
WORKERS = 10          # optional: moderate on purpose — see note below

def cells():          # -> list of (key, payload)
    ...               # key = a STABLE string, identical every launch
def sample(payload):  # -> json-serializable; ONE independent trial
    ...
def reduce(samples, key, payload):  # optional -> the cell's record dict
    ...
def finalize(done, out_dir):        # optional -> summary (once all cells bank)
    ...
def flag(record):                   # optional -> bool: force a progress line
    ...

key is the checkpoint identity — it must be byte-identical across relaunches
(e.g. f"{fixture}||{variant}"). payload is passed straight to sample and
is never persisted, so cells() reconstructs it each launch.

Keep WORKERS moderate. When sample collapses a result to a bool, a
transport error (rate-limit → retries exhausted → False) is indistinguishable
from a real negative and gets checkpointed as one. Fewer concurrent dispatches
means fewer such errors. Durability comes from the checkpoint, not the pool
size — there is no reason to max it out.

Steps

1. Write the eval module against the contract above. Choose an --out
   directory under .claude/evals/<slug>/ (checkpoints live there and
   survive recycles).

2. Launch it durably (backgrounded, from the repo root):

   python3 "${CLAUDE_PLUGIN_ROOT}/skills/long-eval/run_eval.py" ensure-alive \
     --module <path-to-your-module.py> \
     --out .claude/evals/<slug>

   ensure-alive starts the run detached only if no live driver already
   owns that --out and it is not already DONE. It is safe to call any number
   of times — it never double-launches. This one command is the whole
   guard-relaunch.

3. On every session wake, re-run the same ensure-alive. A recycle killed
   the driver; this relaunches it, resuming from the checkpoint. Then report
   progress with status (add --json for a machine-readable line):

   python3 "${CLAUDE_PLUGIN_ROOT}/skills/long-eval/run_eval.py" status \
     --out .claude/evals/<slug>

4. Arm a backstop timer as insurance against a missed wake. Schedule a
   self-re-arming check-in (the session's scheduling tool — e.g. send_later /
   ScheduleWakeup / a Routine, ~20 min out) whose body runs the same
   ensure-alive + status and re-arms itself, unless the run is DONE
   or the user said stop. The backstop only matters if a wake is ever missed;
   the recycle-wake is the primary trigger.

   Follow the calling contract in
   `knowledge/long-run-waiting.md`:
   every arm/re-arm call carries the instruction to replay (prompt) and a
   reason — a call with only a delay fails the "prompt is required when stop
   is not true" validation and arms nothing, which silently costs you the
   backstop you thought you had.

5. Report and stop when DONE. When status shows DONE=True, read
   summary.json from the out-dir, report the final result, and stop
   re-arming the backstop — for a wake-up scheduler that means one
   stop: true-only call, not a call that merely explains why you are
   stopping. A flap-flagged cell (mixed pass/fail across
   samples) is low-confidence — surface it for quarantine/rewrite rather than
   trusting its verdict.

Guarantees & limits

• At most one in-flight dispatch is ever lost on a recycle — samples are
  checkpointed individually, not per-cell.
• Checkpoint writes are atomic (os.replace), so a recycle mid-write can
  never truncate cells.json/samples.json.
• A cell whose sample deterministically raises never banks and blocks
  completion across relaunches by design — the run surfaces it and the operator
  fixes the module. ensure-alive will not paper over a broken eval.
• This supervises one eval per --out directory; run independent evals in
  separate out-dirs.