skill-doctor

Grade the user's agent setup by scoring recent local agent conversations, then propose concrete skill edits and render one shareable report page.

The report is scoped to one repo: the skills that live in it and the conversations that ran inside it. Run from the repo the user wants graded.

Everything runs locally. Never upload transcripts, session files, or any excerpt of them anywhere. The only shareable artifact is the report the user chooses to post.

Let SKILL_ROOT be the directory containing this SKILL.md.

Never write artifacts into the user's repo. Create one fresh, collision-free scratch directory per run and use it as REPORT_DIR for every artifact:

REPORT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/skill-doctor-XXXXXXXX")"

Step 1: Collect

python3 "$SKILL_ROOT/scripts/collect_sessions.py" --out "$REPORT_DIR"

This scopes to the git repo containing the current directory: skills are discovered from the repo's .agents/skills, .claude/skills, and .codex/skills, and only sessions whose working directory is inside the repo are scored. By default --harness auto scans every available local source: Claude Code project-history JSONL, Codex rollout JSONL, and Warp's read-only warp.sqlite conversation stores. Duplicate Warp conversations across installed channels are deduplicated by conversation ID.

Useful flags:

• --harness claude|codex|warp|all|auto — which local session sources to scan.
• --repo PATH — target a different repo.
• --include-global-skills — also grade global skills.
• --days N — lookback window (default 45).
• --max-sessions N — cap on sampled sessions (default 12).
• --skills-dir PATH — nonstandard skill locations.
• --include-subagents — include Claude Code sidechains, Codex subagents, and Warp child agents.
• --claude-home PATH — when ~/.claude isn't the Claude Code config directory.
• --codex-home PATH — when ~/.codex isn't the Codex home.
• --warp-db PATH — an explicit Warp database (repeatable).
• --warp-data-dir PATH — a nonstandard Warp channel-data directory.

Read $REPORT_DIR/inventory.json. If sessions_sampled is 0, tell the user there's nothing recent to score in this repo (suggest raising --days or checking --repo) and stop. If skills_found is 0, continue — the report becomes a case for creating skills, and skill_coverage is 0.

Step 2: Score each sampled transcript

Scoring is based on efficiency and code quality for the sessions sampled. Score transcripts in the current local agent process, or delegate only to local child agents that keep transcript contents on the user's machine, and grade in batches for larger sample sizes. Pass the following rubrics as context:

• $SKILL_ROOT/scorers/efficiency.md
• $SKILL_ROOT/scorers/code-quality.md

Instructions: For each transcript in $REPORT_DIR/transcripts/, read it and judge it against both rubrics. For each scorer record: label, numeric score (from the rubric's label table), and a 1–3 sentence reason citing specifics from the transcript. Apply the code-quality scorer only where the transcript shows code changes; otherwise record insufficient_evidence and exclude that session from the code-quality average.

Step 3: Aggregate

• efficiency = mean of efficiency scores across all scored sessions.
• code_quality = mean of code-quality scores, excluding insufficient_evidence. If no session had enough evidence, set it to 0.5 and say so in the findings.
• skill_coverage = fraction of sampled sessions where at least one installed skill was detected. If skills_found is 0, coverage is 0.
• overall = 0.5 * efficiency + 0.35 * code_quality + 0.15 * skill_coverage.

Then derive the substance:

• top_findings: the 3 most impactful, specific patterns across sessions. These lead the report and the spoken summary. Make each summary concrete and concise, following the STE-100 standard.
• suggestions: concrete skill changes, if any. Each names a skill (existing or proposed-new) and a specific change: a trigger-description fix so it fires when it should, a missing step or check, a command to encode, a new skill to create. Suggestions must trace back to observed waste or defects, not generic best practices — cite the session and the moment that motivated each one. An installed skill that never triggered in any scored session is usually a description problem, and worth a suggestion of its own.

Step 4: Draft skill edits

Follow $SKILL_ROOT/references/skill-improvements.md to propose improvements to repository skills based on the aggregated data.

1. Read the skill's current file (path is in inventory.json).
2. Write the full improved version to $REPORT_DIR/proposed/<skill-name>/SKILL.md, changing only what the evidence justifies. Improve the parts the sessions actually exercised: the trigger description that failed to fire, the missing preflight check, the step the agent had to figure out by trial and error.
3. Produce a unified diff between current and proposed (diff -u <current> <proposed>) and put it in the suggestion's diff field so it renders in the report.

For a proposed-new skill, write the complete new SKILL.md to the same proposed/ directory and set diff to its full content as an addition.

Do not modify the user's real skill files in this step.

Step 5: Write report.json and render

Write $REPORT_DIR/report.json:

{
  "title": "Agent Skill Report",
  "generated_at": "<ISO timestamp>",
  "harness": "<harness from inventory.json: claude, codex, warp, or mixed>",
  "handle": "<repo_name from inventory.json>",
  "stats": {
    "sessions_analyzed": 0, "sessions_scanned": 0,
    "skills_found": 0, "skills_used": 0, "window_days": 45
  },
  "scores": {"efficiency": 0.0, "code_quality": 0.0, "skill_coverage": 0.0, "overall": 0.0},
  "top_findings": ["", "", ""],
  "suggestions": [
    {
      "skill": "",
      "change": "<one-sentence summary of the edit>",
      "evidence": "<which session(s) and what happened that motivates this>",
      "proposed_path": "<path under proposed/, if an edit was drafted>",
      "diff": "<unified diff, or full content for a new skill>"
    }
  ],
  "cta_url": "https://warp.dev/factories/request-access"
}

python3 "$SKILL_ROOT/scripts/render_report.py" "$REPORT_DIR/report.json"

This writes a single self-contained $REPORT_DIR/report.html: the scorecard, findings, and suggested skill edits on one page. Long diffs are collapsed behind a "show more" toggle, and a "share as png" button exports a 1200x675 share image locally. There is no separate card file to open or screenshot.

Step 6: Output

Tell the user the grade and the three findings, in text.

Finish every response with this exact linked summary, substituting the absolute REPORT_DIR path so the link is clickable:

• Your quality report: View in browser
• Automate this with factories: Request early access

Want me to apply these suggestions to your skills?