Audit a Dockerfile's health

Structural validation tells you a Dockerfile is present. docker build
tells you it compiles. Neither tells you it is sane — no hardcoded
secrets, no version drift, no convention violations, no TODOs, no subtle
mistakes a reviewer would flag. This audit is the procedural layer that closes
that gap. It applies to a single Dockerfile, a batch, or the whole fleet, and it
is toolchain-agnostic: a human reads the file in their editor, or a sub-agent
reads this checklist and executes it — one sub-agent per Dockerfile, verdicts
collected. The output format is fixed so findings are comparable across
releases.

The full red / yellow / green signal catalog and the layered (mechanical →
procedural → external-linter) model are bulky reference material; they live in
references/checklist.md beside this skill. Read it before walking the seven
questions so you know what the mechanical layer already covers.

Rules this skill serves

• tests/static/RULES.md:5 — the data-driven Dockerfile rule
  catalog (one ID, severity, why, and predicate per rule) is the mechanical
  layer this audit sits on top of; the audit finds what those rules miss.
• tests/static/RULES.md:9 — prefer mechanical over procedural:
  any pattern this walk surfaces that could be a text check should be proposed
  as a new catalog rule, not left as a recurring manual finding.
• .agents/verification/RULES.md:13 — mechanical > procedural > aspirational;
  this is the procedural tier, run in release verification, not a substitute for
  the mechanical catalog.
• tests/run/agents/RULES.md and benchmarks/RULES.md — the
  required-label sets (agent eval.agent.version, benchmark eval.benchmark.*)
  that question 5 checks the image against.

Procedure

For each Dockerfile under benchmarks/*/ or agents/*/:

1. Gather context. Note the parent directory name (for label-consistency
   checks), its expected type (benchmark or agent), and its sibling files
   (compose.yaml, install.sh, entrypoint.sh). WHY: most findings — label
   drift, a version that does not match the install command, a heavy package no
   sibling script uses — are only visible against this context.

2. Run the mechanical catalog first. cargo test --test check dockerfile.
   Note what it found. WHY: the audit's job is to find what the rules missed,
   not to duplicate them (tests/static/RULES.md:5).

3. Read the Dockerfile end to end and answer the seven questions, marking
   each yes / no / n.a. with a one-line reason. WHY: each question targets a
   class of defect the regex catalog cannot judge.

   #	Question
   1	Does the install sequence make sense? (base → system deps → language runtime → app → entrypoint)
   2	Are comments sufficient — could a new maintainer understand WHY each layer exists?
   3	Is there dead code (unused ARGs, dangling COPY destinations, commented-out blocks)?
   4	Does the image include anything the runtime does not need (build toolchains, docs, sample data)?
   5	Does the label set correctly describe the image content? (agent version matches the install command, dataset revision matches the fetch URL, etc.)
   6	Is the entrypoint sane — reads the right env vars, handles missing defaults, exits cleanly?
   7	Any subtle smells a reviewer would flag but the rules did not catch?

4. Classify the verdict. WHY: questions 5 and 6 are correctness — a wrong
   answer there means the image is actively wrong, not merely untidy.

   • healthy — all seven answers are yes or n.a.
   • needs attention — any question is no but the image still works.
   • broken — question 5 or 6 is no (the image is wrong).

5. Emit one report entry per Dockerfile in the fixed format, then a summary
   count and the top suggested fixes. WHY: the fixed shape lets findings diff
   cleanly across releases.

   ## benchmarks/aime/Dockerfile
   - Mechanical rules: ✓ (0 findings)
   - Q1 (install order): ✓
   - Q2 (comments): ✓
   - Q3 (dead code): ✓
   - Q4 (bloat): ⚠ ships pyarrow for build-time parquet parse, uninstalled at L23 — good
   - Q5 (labels): ✓ data_revision matches URL sha
   - Q6 (entrypoint): ✓
   - Q7 (smells): ✓
   - Verdict: healthy

6. Propose catalog rules for any mechanical-shaped finding. If a smell
   recurs and could be a text check (e.g. a "TODO" string literal inside a
   RUN, a silent pip install ... || true, a duplicate post-FROM ARG, a
   phantom pip uninstall in a separate layer, an ARG/LABEL version
   mismatch, a missing data_revision when fetching a mutable ref), record it
   as a proposed new rule for the catalog. WHY: this is the mechanical >
   procedural escalation (tests/static/RULES.md:9) — a manual
   finding that keeps recurring belongs in code.

When to run

• Before cutting a release (whole fleet) — step 23 of the verify skill.
• When the mechanical Dockerfile catalog flags a yellow that needs judgment.
• When a new benchmark or agent batch lands.
• When a RULES.md changes (old Dockerfiles may have drifted).
• Quarterly, as a health check.

References

• references/checklist.md — the full red / yellow / green signal catalog,
  classification rules, layered-checking model, and output format.
• tests/static/RULES.md — the mechanical Dockerfile rule
  catalog this audit complements.
• .agents/verification/audit-trajectory/SKILL.md — the parallel per-fixture
  runtime-health audit.