CI Failure Triage — Read Before You Conclude

The failure this prevents

Calling a red check "transient / flaky / orphaned / stale" WITHOUT reading its
body. This session: a CodeQL check failed; it was dismissed as a "transient 4s
orphaned check-run" — twice — when it was reporting 3 real security
vulnerabilities (2 critical command-injection + 1 high XSS). The conclusion came
before the evidence. (2026 failure-triage practice = taxonomy → read → cluster →
gate, a repeatable detection system, not vibes:
https://latitude.so/blog/ai-agent-failure-modes-detection-playbook)

Hard rule

You may not use the words "transient", "flaky", "stale", "orphaned", or
"unrelated" about a check until you have read its failure body and quoted the
actual error. A duration (e.g. "4s") is a hint, never proof.

Protocol (in order — do not skip)

1. Identify the exact failing check + its commit.

   head=$(gh pr view <N> --json headRefOid -q .headRefOid)
   gh pr checks <N> | grep -viP "\t(pass|skipping)\t"

2. Read the failure body. This is the step that gets skipped.

   • GitHub Actions job: gh run view --job <id> --log-failed | tail -40
   • CodeQL / code-scanning: read the ALERTS, not just the check:

     gh api "repos/<owner>/<repo>/code-scanning/alerts?state=open&per_page=100" \
       --jq '.[] | "\(.rule.id) | \(.rule.security_severity_level) | \(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line) | ref=\(.most_recent_instance.ref)"'

   • Also fetch the check-run's output.title / output.summary — it usually
     states exactly what failed ("2 new alerts including 2 high severity…").

3. Classify against branch protection. Map each REQUIRED context to its real
   state on the head commit, so you know what actually blocks:

   gh api repos/<owner>/<repo>/branches/main/protection/required_status_checks -q '.contexts[]'
   gh api "repos/<owner>/<repo>/commits/$head/check-runs" --paginate \
     -q '.check_runs[] | "\(.name)=\(.conclusion)"'

4. THEN form a hypothesis, and only with one of these supported verdicts:

   • REAL — the body shows a genuine defect → fix the code, add a test.
   • PRE-EXISTING — same alert exists on main independent of your diff
     (prove it: query alerts on refs/heads/main). Diff-attributed but not yours;
     still must be fixed to unblock — fix it or get an explicit dismissal decision.
   • INFRA — the log shows a runner/network/quota error (quote the line).
     Only now may you re-run. "It was fast" is not an INFRA verdict.

5. Re-run only after an INFRA verdict with a quoted log line. Re-running a
   green-workflow-but-red-check loop forever does nothing if the real blocker is a
   pinned failed check-run from a prior commit — push a fresh commit instead.

"Are you sure?" interaction

If the CEO asks "are you sure?" about a CI claim, your triage was superficial.
Re-run this protocol from step 2 (read the body) before answering. See
evidence-first-answer.

Anti-patterns

Exit criteria

Every red required check is classified with quoted evidence and either fixed
(REAL/PRE-EXISTING) or re-run after a quoted INFRA log line. Never declare CI green
without gh pr checks showing it.