Gauntlet proof-audit

Overview

Audit every committed gauntlet run record against its own persisted evidence and report the
result. This is the gauntlet-proof-audit instance of docs/patterns/audit-loop.md: a skeptic
pass over docs/verification/gauntlet/*/ROUNDS.md (and AB-ROUNDS.md), the eval corpus
/kit:gauntlet produces and lib/gauntlet/stats.sh projects into numbers. Neither of those
checks that a record's CLAIMS match its own EVIDENCE: a "clean=true" that never reconciles with
checker-output.txt, a findings count with no matching finding, a scrubbed key that leaked
anyway, a run-dir that drifted from the contract's naming. This skill never re-runs a probe and
never rewrites a historical record; it reports discrepancies for the operator.

The four slots (per the audit-loop pattern)

Process

1. Branch first. All edits ride an isolated branch. Auditing on master is the failure this
   skill exists to prevent.

2. Enumerate. Run the item-set command above. Write the list down before judging anything.
   The list is the queue; a resumed or scheduled run picks up where the last one stopped.

3. Tier 1, mechanical pass, zero model cost, every record. For each record, extract and
   check the mechanical claims:

   • marker well-formedness is DELEGATED, never restated here: run bash lib/gauntlet/stats.sh
     first (like doc-drift runs test-meta.sh first). Its Pass-1 sweep is the single enforcer of
     the [[QL-VERDICT ...]] / [[AB-VERDICT ...]] grammar across the whole corpus; a non-zero
     exit naming a file is a pre-confirmed marker finding. Do not re-implement the regex here, so
     the grammar cannot desync from its enforcer,
   • for a record whose rounds carry a per-round checker-output.txt (round-N/submission/ or
     round-N/, or a campaign row's own dir), the recorded verdict (the round table's
     GREEN/RED cell, or the row's Checker column) matches that file's own SUBMISSION: GREEN|RED tail line, AND the marker's own clean=true|false agrees with that verdict
     (clean=true only when every scored round is GREEN); a marker that disagrees with the
     checker output is a finding even when the table cell happens to agree,
   • the marker's findings=K reconciles with the round's named findings (an inline F1..FK
     list, or round-N/findings.md's own heading count); K=0 means no F-item is named,
   • scrub: no resolved credential VALUE in the record dir's COMMITTED files, an
     ANTHROPIC_API_KEY=<hex/base64 value> or any recognizable token shape (sk-ant-,
     sk-, ghp_, an assigned <TOKEN>=<20+ opaque chars>); a real value is DANGER, it is
     an actual leak. A bare op://... reference is NOT a leak: it is a pointer, allowed by
     estate secret-handling policy exactly as a path containing an ID is; do not flag it. Note
     it only if the operator asked for a pointer inventory, never as a finding on its own,
   • run-dir grammar: the directory name carries a <YYYY-MM-DD>-<preset>-<slug> shape for any
     record dated 2026-09 or later (pre-2026-09 dirs are grandfathered per commands/gauntlet.md,
     never flagged for the old shape alone).
     A Tier 1 failure is a finding with both sides quoted (the claim, the evidence). A record with
     no committed checker-output.txt for a round it claims a verdict on is UNTESTABLE on that
     axis, not OK and not a fabrication either: downgrade that axis to UNSURE, evidence quoted as
     "no checker-output.txt committed for round-N".

4. Tier 2, judgment pass, model-read, only where it earns its cost. Dispatch kit:audit-scanner
   (its tools roster physically cannot write) for two cases only: a record Tier 1 flagged (judge
   how deep the drift goes), or a quoted finding whose exact wording needs matching against the
   transcript it cites (transcript.jsonl / transcript.md). Skip Tier 2 for every record Tier 1
   cleared with no quote-matching need; most records never need it.

5. Verdict each record with the audit-loop grammar: OK / FLAG / UNSURE / DANGER. REMOVE is
   never used: a run record is evidence of what happened, not a doc that can go stale and get
   deleted. A verdict with no checkable evidence downgrades to UNSURE (a pruned/gitignored
   transcript, a room artifact never committed). DANGER is reserved for a record that actively
   misrepresents its own evidence (a claimed GREEN whose checker-output.txt says RED) where an
   operator reading ROUNDS.md at face value would be misled about the artifact's real state; a
   plain reference leak or an untestable axis is FLAG or UNSURE, not DANGER.

6. Apply. Report-first: every FLAG and UNSURE is listed in the PR body with its quoted
   evidence, for the operator to read. Only a DANGER verdict gets a correction note, appended to
   the record (never a rewrite of the original claim): a dated ## Audit correction (YYYY-MM-DD)
   block naming the discrepancy and citing the contradicting evidence file. The original round
   table, verdict line, and markers are left untouched; the correction sits beside them.

7. Verify. Re-run the item-set command; it must still enumerate the same records (an audit
   pass reads, it never adds or removes a record). If any correction note was appended, confirm
   it parses as prose (no marker collision with QL-VERDICT/AB-VERDICT, lib/gauntlet/stats.sh
   must still run clean). Run bash tests/test-meta.sh: green, or the run is not done.

8. Ship. Commit, push, open a PR whose body lists every FLAG/DANGER with its evidence and
   every UNSURE for the operator. If every record verdicts OK, create no branch and report CLEAN
   with the enumeration list.

Cadence

On-demand first: run after a gauntlet campaign lands, before citing corpus numbers in a report,
or when a record's honesty is in question. For a recurring pass, wire /loop or a schedule the
same way doc-drift does; there is no default cron, cadence is the operator's to wire.

Cadence

This instance has a declared cadence in the ## Cadence table of docs/patterns/audit-loop.md.
Record a finished pass with bin/audit ran gauntlet-proof-audit "<one line>", including a CLEAN one, so
bin/audit due stops naming it. The kit ships no scheduler: due is what a human or a job reads.