Produce or refresh one benchmark's AUDIT.md — the standing record of what has actually been checked about it (validity, safety, size, speed, cost, distribution). Use it when adding a benchmark, after changing its Dockerfile or grader, when its AUDIT.md is stale, or as the per-benchmark step of a fleet audit. It fills the machine-measurable fields from real tool runs and leaves human-judgment fields (running, traces-reviewed, replicate-official, cost) explicitly unchecked until a human does them. For the whole-repo sweep use audit-fleet; this skill drives the oracle (core/oracle/README.md) for the grading-soundness rung.
Recursos
1Instal·la
npx skillscat add exgentic/eval-containers/audit-benchmark Instal·la des del registre de SkillsCat.
Audit a benchmark — write its AUDIT.md
An AUDIT.md is only worth trusting if a ✓ means a check actually ran. This
procedure fills the auto-measurable fields from real runs and marks everything
else honestly unchecked, so the report never drifts into false confidence.
Serves .agents/verification/audit/RULES.md:1–8.
Steps
Start from the template. If
benchmarks/<name>/AUDIT.mdis absent, copyreferences/template.mdto it. Starting from the template guarantees every
dimension is present and every status starts?(RULES.md:3, :7).Stamp provenance. Set the frontmatter
benchmark,host(e.g.local podman+Rosetta), andcommit(git rev-parse --short HEAD). The commit
anchors what code was audited; its date and any staleness derive from git, so
no separate date is stored (RULES.md:2).Build it on Rosetta. Build the image per
docs/guides/podman-on-apple-silicon.md§6 — single-FROMviaDOCKER_BUILDKIT=0 docker build, multi-stage via nativepodman build --platform linux/amd64 --pull=never. Set building✓/✗with the build
command as evidence (RULES.md:5, :6, :8).Oracle it. Run
eval-containers oracle <name> [--task-id <id>]. Set
oracle✓iff gold=1.0 and no-op<1.0, recording both numbers as evidence;n/afor an LLM-judge or stub grader. This is the grading-soundness rung —
it proves the grader is neither always-pass nor always-fail (core/oracle).Read isolation + safety off the artifact. Confirm from the Dockerfile and
compose that the benchmark does not undermine the framework's isolation:/tasksstays root-only and the task id never reaches the agent →
isolation✓; the agent runs non-root → agent-nonroot✓; secrets
enter only via--mount=type=secret→ secrets-isolated✓; no agent
egress is opened → egress-blocked✓. Each is inspectable without a run.Measure size + speed. From the built image and the build/grade runs:
image size (and the per-task multiplier for per-task benchmarks) → Size;
build, grade, and end-to-end wall-clock → Speed. Copy the measured numbers
in; never hand-type an estimate (RULES.md:8).Check distribution.
docker manifest inspect ghcr.io/exgentic/benchmarks/<name>:latest— set published✓if it
resolves,✗if not. Record pull size as the sum of the manifest's layer
sizes: the compressed download, distinct from the local image size (RULES.md:6).Leave the rest honestly unchecked. running (a live agent run),
traces-reviewed (human reads a trajectory), replicate-official (a
known model reproduces the published score), resource-limited, and
cost need work this skill does not do. Mark each?(or✗), never✓,
until that work actually happens (RULES.md:5, :7).Commit it, then refresh the rollup. Commit
benchmarks/<name>/AUDIT.md,
then regenerate the root project table (theaudit-rollupskill) so it
reflects this change. The report is a tracked artifact, refreshed whenever the
benchmark or its grader changes so its provenance never goes stale
(RULES.md:1, :2, :10).