Harness review

Ground rules:

• This review runs only because the user asked, and the user decides at every step.
• Never change the kit or any project. The only output is a draft proposal, for the user to annotate.
• Read only the projects the user picks. Not every project here uses this harness.

1. Choose the data (with the user)

1. From the workspace root, list the subfolders that contain docs/metrics/milestones.csv. Check that the file exists only; don't open anything yet.
2. Ask the user (AskUserQuestion, multi-select) which of those projects to include.
3. Find the kit: use $CLAUDE_KIT_DIR if it's set, otherwise ask for the path.

2. Aggregate (no model tokens)

Run python3 ${CLAUDE_SKILL_DIR}/aggregate.py <chosen project dirs>. It reads only those projects' milestones.csv and retro.md, and prints:

• every milestone;
• medians by kind and by kit version;
• causes of rework, token share by role, guardrail blocks by rule, anomaly frequency;
• every harness: line from the retros.

Then list the kit's existing docs/proposals/ by title and status, so you don't re-propose something already planned or rejected.

3. Diagnose

Match the data against this playbook. Evidence rule: a pattern needs at least 2 milestones of the same kind, or 1 if it's severe (lost work, a bypassed guardrail, a security issue). Anything weaker is a hypothesis to watch, not a change to make. Never compare milestones of different kinds, and say plainly when the aggregate reports too little data.

4. Present findings (in chat)

For each finding: the data, the pattern, the candidate change, and your confidence (firm / watch). Ask the user which, if any, to turn into a proposal. An empty result is fine: "no change warranted yet" is a valid outcome.

5. Draft the proposal (only for items the user picked)

Write <kit>/docs/proposals/NNNN-<slug>.md, using the next free number:

• Status: Draft, awaiting annotation.
• Evidence: the relevant tables and harness: lines. Anonymize them, because the kit is public: "a domain milestone in project A", with no project names, paths, people or code.
• Hypothesis, and the exact change (which kit files change, and how).
• Predicted effect: the metric, its direction and rough size, and the milestone kind it applies to, e.g. "−20% tokens/step on api milestones".
• Quality guard: which quality metric must not get worse (must-fix findings, escaped defects, test files added).
• How it will be tested: hook or lint tests where they apply; plugin evals for prompt changes; the benchmark for token-saving changes once it exists.
• Rollback: how to revert it.

Don't commit anything. Tell the user where the proposal is, so they can annotate it the usual way. Later milestones' metrics check whether the predicted effect happened.