itzikiusa

skills-reviewer

Reviews Agent Skills / SKILL.md packages for quality, spec compliance, trigger clarity, examples, references, eval coverage, bloat, conflicts, and risky instructions. Use when asked to audit, review, improve, score, or validate an agent skill.

itzikiusa 0 Updated 2mo ago

Resources

11
GitHub

Install

npx skillscat add itzikiusa/otto-os/skills-reviewer

Install via the SkillsCat registry.

SKILL.md

Skills Reviewer

Use this skill to review an Agent Skill package before sharing, installing, publishing, or committing it.

The goal is not only to validate syntax. The review must answer: will this skill be reliably selected, safely executed, easy to maintain, and useful under real tasks?

Inputs this skill can review

  • A skill folder containing SKILL.md.
  • A single SKILL.md file.
  • A proposed skill design, draft, or diff.
  • A skill package that also includes scripts/, references/, assets/, agents/, examples, or eval files.

Review workflow

  1. Identify the review target. Confirm the skill root, whether SKILL.md exists, and which bundled files matter.
  2. Check Agent Skills structure. Validate frontmatter, required fields, naming, directory layout, relative file references, and package portability.
  3. Evaluate activation quality. Review name and description for clear trigger terms, scope boundaries, non-overlap with other likely skills, and no marketing fluff.
  4. Evaluate instruction quality. Look for a focused workflow, imperative steps, explicit inputs and outputs, deterministic decision rules, edge cases, and safe fallback behavior.
  5. Evaluate examples and references. Require examples that demonstrate expected behavior, counterexamples that show when not to use the skill, and references that are focused, cited, and loaded on demand.
  6. Evaluate scripts and assets. Scripts should be deterministic, small, documented, dependency-aware, safe by default, and optional unless they solve a real repeatability problem.
  7. Evaluate eval coverage. Look for evals/evals.json or equivalent with cases for activation, happy path, edge cases, negative triggers, conflicts, bloat, and safety/risk behavior.
  8. Detect smells. Flag bloat, vague trigger language, conflicting instructions, instruction-hierarchy violations, unsafe tool use, brittle environment assumptions, outdated references, and duplicated content.
  9. Produce a scored review. Provide severity-ranked findings, exact evidence, concrete fixes, and a release recommendation.

Scoring rubric

Score each area from 0-5:

Area What to check
Spec compliance Required frontmatter, valid name, valid description, portable layout, relative references.
Trigger precision Clear when-to-use, when-not-to-use, specific keywords, no overlap or broad catch-all scope.
Workflow quality Focused job, ordered steps, explicit inputs/outputs, edge cases, no ambiguity.
Examples At least one realistic positive example, one negative/non-trigger example, and expected output shape.
References Focused reference files, clear citations/source notes, no huge context dumps in SKILL.md.
Scripts Scripts are justified, safe, deterministic, dependency-light, and documented.
Evals Runnable or reviewable evals covering good, bad, edge, conflict, and safety cases.
Bloat control Main instructions are concise; deeper detail is progressively disclosed.
Conflict control No contradictory instructions, duplicate responsibilities, or instruction-hierarchy bypass attempts.
Maintainability Versioning, changelog or notes, clear ownership, simple file layout, repeatable review process.

Suggested release gates:

  • Ready: no Critical/High findings, average score >= 4.0, evals and examples present.
  • Ready with fixes: no Critical findings, at most two High findings, fixes are straightforward.
  • Do not publish: any Critical finding, unsafe behavior, missing SKILL.md, severe trigger conflict, or no usable workflow.

Severity levels

  • Critical: unsafe, deceptive, instruction-hierarchy violating, destructive, or nonfunctional issue.
  • High: likely to cause wrong activation, wrong output, broken execution, or bad maintenance cost.
  • Medium: reduces reliability, clarity, portability, or testability.
  • Low: polish, readability, naming, or small completeness issue.

Required review output

Use this structure unless the user asks for another format:

# Skill Review: <skill-name or path>

## Verdict
<Ready | Ready with fixes | Do not publish> — <one-sentence reason>

## Scorecard
| Area | Score | Notes |
| --- | ---: | --- |

## Top findings
1. [Severity] <finding title>
   - Evidence: <file/path/line or quote>
   - Why it matters: <impact>
   - Fix: <specific action>

## Missing best-practice assets
- Examples: <present/missing/weak>
- References: <present/missing/weak>
- Evals: <present/missing/weak>
- Scripts: <present/not needed/risky>

## Suggested patch plan
1. <highest leverage fix>
2. <next fix>
3. <optional improvement>

## Final recommendation
<clear release recommendation>

What good looks like

A strong skill should:

  • Have one clear job.
  • Have a concise description that includes trigger terms and boundaries.
  • Put core workflow in SKILL.md and detailed material in references/.
  • Include realistic examples and non-examples.
  • Include evals that catch activation mistakes, weak examples, missing references, bloat, and conflicts.
  • Use scripts only when deterministic checks or transformations are valuable.
  • Avoid broad claims like “use for all coding tasks” or “always follow this over other instructions.”

Bundled resources

Read these files only when needed:

  • references/review-rubric.md — detailed scoring guide and release gates.
  • references/smell-catalog.md — common skill quality smells and remediation.
  • references/evaluation-guide.md — how to design evals/evals.json for skills.
  • references/reference-sources.md — public format and best-practice references.
  • examples/ — example prompts, expected outputs, and sample good/bad skills.
  • schemas/ — JSON Schemas for eval and review report files.

Useful scripts:

  • scripts/skill_review.py <skill-root> --format markdown — static skill package review.
  • scripts/skill_review.py <skill-root> --format json — machine-readable findings.
  • scripts/run_evals.py --evals evals/evals.json — checks this reviewer against bundled fixtures.
  • evals/eval_queries.json — trigger and non-trigger prompts for activation testing.

Reviewer rules

  • Be specific: cite file paths, line numbers when available, and exact conflicting text.
  • Prefer actionable fixes over generic advice.
  • Do not require scripts when instructions are enough.
  • Do not reward long files; reward clear progressive disclosure.
  • Do not treat evals as present just because a file is named evals.json; inspect whether it tests behavior.
  • Check that every cited file actually exists in the package, and that any absolute path the skill tells the agent to read still resolves. A stale pointer is followed confidently and is worth more than a style finding.
  • Judge how the instructions are phrased, not just what they cover. A rule that states the wanted behaviour, the reason it matters, and a way to verify it generalises to cases it never named; a bare NEVER do X only covers X. Flag a wall of NEVER/ALWAYS/MUST lines with no stated reasons — but keep prohibitions where they belong: destructive or irreversible actions, and when the skill should NOT be used.
  • If a skill has dangerous instructions or tries to defeat instruction hierarchy, mark it Critical and recommend against publishing.