Harness review

Ground rules:

• This review runs only because the user asked, and the user decides at every step.
• Never change the kit or any project. The only output is a draft proposal, for the user to annotate.
• Read only the projects the user picks. Not every project here uses this harness.

1. Choose the data (with the user)

1. From the workspace root, list the subfolders that contain docs/metrics/milestones.csv. Check that the file exists only; don't open anything yet.
2. Ask the user (AskUserQuestion, multi-select) which of those projects to include.
3. Find the kit: use $CLAUDE_KIT_DIR if it's set, otherwise ask for the path.

2. Aggregate (no model tokens)

Run python3 ${CLAUDE_SKILL_DIR}/aggregate.py <chosen project dirs>. It reads only those projects' milestones.csv and retro.md, and prints:

• every milestone;
• medians by kind and by kit version;
• causes of rework, token share by role, guardrail blocks by rule, anomaly frequency;
• every harness: line from the retros.

Then list the kit's existing docs/proposals/ by title and status, so you don't re-propose something already planned or rejected.

3. Diagnose

Match the data against this playbook. Evidence rule: a pattern needs at least 2 milestones of the same kind, or 1 if it's severe (lost work, a bypassed guardrail, a security issue). Anything weaker is a hypothesis to watch, not a change to make. Never compare milestones of different kinds, and say plainly when the aggregate reports too little data.

Pattern	Likely cause	Candidate change
High tokens/step and high failing runs/step	Plan steps too coarse, or fixtures unclear	Smaller steps; a fixture table in each step; sharper verification commands
spec-gap is the top cause	The interview misses an area	Add that area to spec-interview's questions
design-gap recurring	The brief misses states or data	Extend the design brief checklist
plan-gap recurring	Plans miss or misorder work	Extend the milestone plan checklist
impl-error dominant	The developer misreads clear specs	Developer prompt; a model escalation for that kind
test-bug recurring	The tester misreads the spec	Tester prompt; independence rules
environment recurring	Setup gaps	bootstrap/machine-setup.md; the M0 checklist
No reviewer must-fix items across 3 or more low-risk milestones	Review is over-provisioned	Reviewer on sonnet when risk: low
Malformed tool calls concentrated in one role	Prompt or tool guidance	Fix that role's prompt
The same guardrail fires repeatedly	A prompt conflicts with a rule	Fix the prompt, not the guardrail
Interventions recur	Missing permissions or unclear prompts	The task prompt template; settings
High architect token share	Reading scrollback, or long reports	Tighter report formats
Recurring harness: observations	As stated in the lines	As stated, made concrete

4. Present findings (in chat)

For each finding: the data, the pattern, the candidate change, and your confidence (firm / watch). Ask the user which, if any, to turn into a proposal. An empty result is fine: "no change warranted yet" is a valid outcome.

5. Draft the proposal (only for items the user picked)

Write <kit>/docs/proposals/NNNN-<slug>.md, using the next free number:

• Status: Draft, awaiting annotation.
• Evidence: the relevant tables and harness: lines. Anonymize them, because the kit is public: "a domain milestone in project A", with no project names, paths, people or code.
• Hypothesis, and the exact change (which kit files change, and how).
• Predicted effect: the metric, its direction and rough size, and the milestone kind it applies to, e.g. "−20% tokens/step on api milestones".
• Quality guard: which quality metric must not get worse (must-fix findings, escaped defects, test files added).
• How it will be tested: hook or lint tests where they apply; plugin evals for prompt changes; the benchmark for token-saving changes once it exists.
• Rollback: how to revert it.

Don't commit anything. Tell the user where the proposal is, so they can annotate it the usual way. Later milestones' metrics check whether the predicted effect happened.