breethomas

calibrate

Post-launch AI feature calibration workflow. Document error patterns, review eval performance, and decide on agency promotion. Based on CC/CD framework for continuous calibration of AI products.

breethomas 16 3 Updated 7mo ago

Resources

1
GitHub

Install

npx skillscat add breethomas/pm-thought-partner/calibrate

Install via the SkillsCat registry.

SKILL.md

Calibrate - Post-Launch AI Feature Calibration

Core Philosophy

Calibration happens after launch, not before.

The mistake: Building elaborate systems to perfectly calibrate AI behavior before launch.
The reality: You learn what quality means by shipping to users and seeing what they actually need.

The Calibration Loop:

  1. Deploy at current agency level
  2. Monitor performance in prod
  3. Analyze and learn
  4. Calibrate system
  5. Test changes
  6. Consider agency increase
  7. Repeat

Entry Point

When this skill is invoked, start with:

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 CALIBRATE
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Calibration happens after launch, not before.

What do you need?

  1. Document error patterns
     โ†’ Analyze failures, categorize, plan fixes

  2. Review eval performance
     โ†’ Are evals catching real issues? Missing patterns?

  3. Agency promotion decision
     โ†’ Is this feature ready for more autonomy?

  4. Quick calibration check
     โ†’ Is the system behaving as expected?

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Parse intent from context:

  • If user mentions "errors" or "failures" or "bugs" โ†’ Flow 1
  • If user mentions "evals" or "tests" or "coverage" โ†’ Flow 2
  • If user mentions "promote" or "increase" or "V2" โ†’ Flow 3
  • If user mentions "check" or "status" or "quick" โ†’ Flow 4

Command-line shortcuts:

  • /calibrate โ†’ Show entry menu
  • /calibrate --errors โ†’ Flow 1 (error patterns)
  • /calibrate --evals โ†’ Flow 2 (eval review)
  • /calibrate --promote โ†’ Flow 3 (agency promotion)
  • /calibrate --quick โ†’ Flow 4 (quick check)

Flow 1: Document Error Patterns

Step 1: Gather Error Data

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 ERROR PATTERN DOCUMENTATION
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Let's catalog what's going wrong.

Where are you seeing errors?
  โ€ข User feedback / complaints
  โ€ข Support tickets
  โ€ข Monitoring alerts
  โ€ข Manual review
  โ€ข User corrections / overrides

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Questions to ask:

  • "What specific failures have you observed?"
  • "How often does this happen? (rare, occasional, frequent)"
  • "What's the impact when it fails?"
  • "Are there patterns in WHEN it fails?"

Step 2: Categorize Errors

Categories must emerge from the errors you actually observe โ€” not from a pre-defined list. Read the failures first, then group similar ones together. Pre-defined categories cause confirmation bias.

For the systematic process (reading traces, emergent categorization, failure rates), run /upgrade-evals. Flow 1 here is for quick ad-hoc error documentation when you don't need the full analysis.

Step 3: Analyze Root Causes

For each error pattern, determine:

  • Likely reason: Why is this happening?
  • Potential fix: Prompt change? Context? Guardrail? Training data?
  • Priority: P1 (critical), P2 (important), P3 (nice to fix)

Output: Error Pattern Table

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 ERROR PATTERN ANALYSIS
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Feature: [name]
Analysis Date: [date]
Data Source: [where errors were observed]

| Error Pattern | Category | Likely Reason | Potential Fix | Priority |
|---------------|----------|---------------|---------------|----------|
| [description] | [type] | [why] | [how to fix] | P1 |
| [description] | [type] | [why] | [how to fix] | P2 |
| [description] | [type] | [why] | [how to fix] | P3 |

PATTERN ANALYSIS:
- Most common category: [X]
- Emerging pattern: [Y]
- Regression from last period: [Z]

RECOMMENDED ACTIONS:
1. [P1 action]
2. [P2 action]
3. [P3 action]

ADD TO EVALS:
- [ ] Add test case for [error pattern 1]
- [ ] Add test case for [error pattern 2]

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Flow 2: Review Eval Performance

Step 1: Current Eval State

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 EVAL PERFORMANCE REVIEW
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Let's see if your evals are working.

Current state:
  โ€ข How many test cases do you have?
  โ€ข What's your pass rate?
  โ€ข When did you last update evals?
  โ€ข Are you seeing failures in prod that evals missed?

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Questions to ask:

  • "Are current evals catching real issues?"
  • "What new patterns have emerged since launch?"
  • "Are you passing 100%?" (If yes, evals may be too easy)
  • "How often do you run evals?"

Step 2: Gap Analysis

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 EVAL GAP ANALYSIS
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Check coverage across categories:

  โ–ก Happy path - Common successful scenarios
  โ–ก Edge cases - Unusual but valid inputs
  โ–ก Adversarial - Intentional misuse attempts
  โ–ก Boundary - Out of scope handling
  โ–ก Regression - Previously fixed issues
  โ–ก Production errors - Real failures observed

Missing categories = gaps in coverage
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Output: Eval Assessment

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 EVAL ASSESSMENT
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Feature: [name]
Test Cases: [count]
Pass Rate: [%]
Last Updated: [date]

COVERAGE ASSESSMENT:
| Category | Coverage | Status |
|----------|----------|--------|
| Happy path | [X] cases | โœ… Good |
| Edge cases | [X] cases | โš ๏ธ Needs work |
| Adversarial | [X] cases | โŒ Missing |
| Boundary | [X] cases | โœ… Good |
| Regression | [X] cases | โš ๏ธ Needs work |

EFFECTIVENESS:
- Catching real issues? [Yes/No/Partially]
- False positive rate: [%]
- Prod errors missed: [list]

RECOMMENDATIONS:
1. Add [X] test cases for [gap]
2. Update [Y] tests that are stale
3. Remove [Z] tests that are redundant

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Flow 3: Agency Promotion Decision

Promotion Checklist

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 AGENCY PROMOTION CHECK
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Considering: V[current] โ†’ V[target]

Let's verify readiness.

QUALITY METRICS
  โ–ก Accuracy/quality stable for 4+ weeks?
  โ–ก No new error patterns emerging?
  โ–ก User corrections decreasing?

SAFETY & TRUST
  โ–ก Confident in known failure modes?
  โ–ก Override mechanism working well?
  โ–ก User feedback positive?

OPERATIONAL READINESS
  โ–ก Monitoring in place for new level?
  โ–ก Rollback plan ready?
  โ–ก Team aligned on promotion?

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

For each item, ask:

  • "What's your evidence?"
  • "How long have you observed this?"
  • "What would change your answer?"

Output: Promotion Verdict

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 PROMOTION VERDICT
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Feature: [name]
Current: V[n] | Target: V[n+1]

VERDICT: [READY โœ… / NOT READY โŒ / NEEDS WORK โš ๏ธ]

โœ… PASSING:
- [criteria met with evidence]
- [criteria met with evidence]

โŒ BLOCKING:
- [criteria not met + what's needed]
- [criteria not met + what's needed]

โš ๏ธ RISKS IF PROMOTED NOW:
- [risk + mitigation needed]

RECOMMENDATION:
[Clear recommendation with reasoning]

NEXT STEPS:
1. [action if ready]
2. [action if not ready]

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Flow 4: Quick Calibration Check

Health Check

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 QUICK CALIBRATION CHECK
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Fast health check for [feature name]

  โ€ข Current quality metric: [X%]
  โ€ข Trend: [โ†‘ improving / โ†’ stable / โ†“ degrading]
  โ€ข Any alerts triggered? [Y/N]
  โ€ข User feedback signals: [positive/neutral/negative]
  โ€ข Override rate: [X%] (is this expected?)

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Output: Health Status

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”
 CALIBRATION STATUS
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Feature: [name]
Agency Level: V[n]
Check Date: [date]

STATUS: [HEALTHY โœ… / ATTENTION โš ๏ธ / DEGRADED โŒ]

METRICS:
| Metric | Value | Trend | Status |
|--------|-------|-------|--------|
| Quality | [X%] | [โ†’] | โœ… |
| Override rate | [X%] | [โ†“] | โœ… |
| User satisfaction | [X] | [โ†’] | โš ๏ธ |

ALERTS:
- [any triggered alerts]

ACTION NEEDED:
- [none / specific action]

NEXT CHECK: [recommended cadence]

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”

Calibration Cadence

Weekly: Quick check (Flow 4)

  • Is quality stable?
  • Any new alerts?
  • User feedback signals?

Monthly: Eval review (Flow 2)

  • Are evals catching real issues?
  • New patterns to add?
  • Stale tests to update?

Quarterly: Deep calibration

  • Error pattern analysis (Flow 1)
  • Agency promotion consideration (Flow 3)
  • Strategic calibration review

Challenge Patterns

"Quality looks fine"
โ†’ "What's your correction rate? What's the trend? Fine compared to what baseline?"

"No complaints"
โ†’ "Are users actually using it? Are they silently working around it?"

"Ready to promote"
โ†’ "Show me the data. How long has quality been stable? What failure modes have you validated?"

"Evals are passing"
โ†’ "100% pass rate? That might mean evals are too easy. When did you last add new test cases?"

Integration with Other Commands

Before /calibrate:

  • /agency-ladder - Define the ladder first
  • /spec --ai - Ensure spec includes calibration plan

Related:

  • /ai-health-check - Pre-launch validation
  • /start-evals - Set up eval infrastructure
  • /upgrade-evals - Systematic error analysis on real traces
  • /build-judge - LLM-as-Judge for automating subjective evals
  • /eval-rag - RAG-specific retrieval + generation evaluation

Attribution

Framework: CC/CD (Continuous Calibration/Continuous Development)
Source: Aishwarya Naresh Reganti & Kiriti Badam (Lenny's Newsletter)
Adaptation: Post-launch calibration workflows

Remember

  • Calibration is continuous, not one-time
  • Error patterns are the richest signal
  • User corrections show where AI fails
  • Evals must evolve with the product
  • Promotion requires proven reliability
  • 100% pass rate means evals are too easy

Categories

Code Review Debugging Testing
GitHub

Install

npx skillscat add breethomas/pm-thought-partner/calibrate

Install via the SkillsCat registry.

Recommended Skills

morluto
rea-ci-tests-audit by morluto
370.7K 1w ago
cloudflare
security-audit by cloudflare
41.4K 1mo ago
cursor
fix-merge-conflicts by cursor
13.7K 8mo ago
cursor
run-smoke-tests by cursor
13.7K 8mo ago
mastra-ai
find-skills by mastra-ai
29K 3w ago
davepoon
coding-agent-pm by davepoon
3.7K 4mo ago
bex-co
routine-logic-bugfixer by bex-co
312 5d ago
bex-co
routine-logic-simplifier by bex-co
312 5d ago
HMS091
vp-cpo-readiness-advisor by HMS091
0 6d ago
tin-computer
small-reviewable-site-health-fix by tin-computer
264 1mo ago