Audit the fleet's health

Per-file inspections (trajectory health, Dockerfile health) tell you each
benchmark and agent is individually sound. They do not tell you the fleet is
sound. A fleet is sound when every documentation claim matches disk, every
benchmark has the tests it is supposed to have, every referenced agent/model
version builds, nothing has silently drifted below the quality bar, and nothing
is stale. This audit is the high-level sanity check run before a release, before
a pitch, or quarterly. It identifies gaps, regressions, and drift across the
whole repo rather than inside a single file.

The audit is the procedural layer over the fleet aggregator. It is
toolchain-agnostic: the mechanical gates are cargo test, and the ten questions
are a checklist any reviewer — human, AI assistant, or script — can walk. The
output format is fixed so reports diff across time.

The full red / yellow / green signal catalog and the three-layer checking model
are bulky reference material; they live in references/checklist.md beside this
skill. Read it before walking the ten questions.

Rules this skill serves

• tests/run/fleet/RULES.md:3 — the report has three sections
  (mechanical gates, procedural audits, verdict); this audit produces the
  procedural section and the verdict.
• tests/run/fleet/RULES.md:4 — the verdict classification (green =
  all green, yellow = some yellows no reds, red = any red); step 4 of this walk
  applies it.
• tests/run/fleet/RULES.md:6 — known-broken aware: a build failure
  inside tests/build/known-broken.md is yellow, not red; this audit honours
  the manifests when judging questions 2 and 6.
• tests/build/RULES.md:6 — failures within the build
  known-broken manifest are yellow; failures outside it are red (question 2).
• .agents/verification/RULES.md:13 — mechanical > procedural > aspirational;
  this audit runs the mechanical gates first, then applies judgment only to the
  drift questions rules cannot reach.

Procedure

Input is the whole working tree at a given commit. Context: the release notes
(what is shipping?), the previous audit report (what was flagged last time?),
and the current RULES.

1. Run all mechanical gates, in order, recording pass/fail for each. WHY:
   the report's mechanical section is the spine; anything failing here is a
   finding before you reach judgment.

   cargo test --test check structure
   cargo test --test check compose -- --ignored
   cargo test --test check dockerfile -- --ignored
   cargo test --test check trajectory -- --ignored
   cargo test --test build -- --ignored      # the slow one
   cargo test --test replay -- --ignored      # end-to-end

2. Reconcile counts. WHY: documentation drift (a README claim that does not
   match disk) is a red signal — the cheapest one to check and the most
   embarrassing to ship.

   • ls benchmarks/ | wc -l vs the README claim (excluding RULES.md /
     TEMPLATE.md).
   • ls agents/ | wc -l vs the README claim (same exclusions).
   • ls tests/run/replay/fixtures/*.traces.jsonl | wc -l vs how many
     benchmarks have fixtures.

3. Walk the ten fleet questions, each yes / no / n.a. with a one-line
   reason. WHY: these are the cross-cutting drift questions no per-file check can
   see.

   #	Question
   1	Does every benchmark in benchmarks/ have both a Dockerfile and a compose.yaml?
   2	Does every committed benchmark and agent actually build?
   3	Does every benchmark labeled eval.benchmark.released="true" have at least one replay fixture under tests/run/replay/fixtures/? (see benchmarks/RULES.md 21a)
   4	Does the README's benchmark/agent/model count match the filesystem?
   5	Does every agent in agents/ have a pinned eval.agent.version label — no unpinned, no latest?
   6	Are there any benchmarks whose Dockerfiles reference upstream images we no longer control or that have gone stale?
   7	Are there any orphan fixtures (files referencing benchmarks / agents no longer in the repo)?
   8	Is every published eval-* version tag in sync — no benchmark at eval-v2 while shared compose is still eval-v1?
   9	Do the RULES principles (esp. 9, 10, 11) hold across every benchmark and agent, or has anything drifted?
   10	Does the CI release workflow reflect reality — has it run recently, is the registry actually populated, does the README Quick Start work from scratch on a clean machine?

4. Classify the verdict. WHY: the classification fixes which gaps block a
   release and which merely document it.

   • Any no on questions 1–5 or 9 is red (do not ship).
   • Any no on questions 6–8 is yellow (ship with the gap documented).
   • Question 10 is informational but mandatory before shipping.
   • Honour the known-broken manifests: a build failure inside
     tests/build/known-broken.md is yellow, not red
     (tests/run/fleet/RULES.md:6).

5. Emit one report in the fixed structure: a Counts section, a Mechanical
   gates section (one row per gate), a Fleet questions table, a Suggested fixes
   list, and a final Verdict line. WHY: the fixed shape lets reports diff across
   releases. Skeleton:

   # Fleet Health Report — YYYY-MM-DD
   
   ## Counts
   - benchmarks: 96 on disk, 96 claimed in README ✓
   - agents: 17 on disk, 17 claimed in README ✓
   - fixtures: 23 ✓
   
   ## Mechanical gates
   - check structure: ✓
   - compose: ✓ (96/96)
   - dockerfile: ✓
   - trajectory: ✓ (23/23 green, 0 yellow)
   - build: ⚠ (4 known failures, all in known-broken.md)
   - replay: ✓ (23/23)
   
   ## Fleet questions
   | # | Q                          | Verdict | Notes |
   |---|----------------------------|---------|-------|
   | 1 | Every bench has both files | ✓       |       |
   | 2 | Every bench builds         | ⚠       | 4 known-broken |
   | ...                                                  |
   
   ## Suggested fixes
   1. ...
   
   ## Verdict
   ⚠ yellow — ship-ready with N known gaps documented

When to run

• Before cutting a release (mandatory) — step 25 of the verify skill.
• Quarterly health check (recommended).
• After a batch of ≥10 new benchmarks / agents lands.
• After a RULES change that touches multiple files.

References

• references/checklist.md — the full red / yellow / green fleet signal
  catalog, classification rules, layered model, and output format.
• tests/run/fleet/RULES.md — the aggregator rules and verdict
  classification this audit produces.
• .agents/verification/audit-dockerfile/SKILL.md and
  .agents/verification/audit-trajectory/SKILL.md — the per-file audits whose
  findings feed the fleet picture.
• .agents/verification/audit-rules-drift/SKILL.md — the companion drift audit
  for question 9.