Orchestration Benchmark (#1095)

Role: orchestrator. This command dispatches benchmark runs and aggregates
their results — it never performs the benchmark tasks itself and never edits
the code under test.

You have been invoked with the /orchestration-benchmark command.

Not /benchmark. /benchmark measures runtime performance of web
pages (Core Web Vitals, resource sizes, load times). This skill measures
orchestration value: whether the dev-team pipeline's delegation earns
its overhead against a solo session on the same task at the same rigor.
Different subject, different instruments, different report.

Orchestrator constraints

1. Matched rigor is non-negotiable. Every arm runs the identical task
   with identical acceptance checks. An arm may not skip or weaken
   verification — a cheap worker that reads less and passes weaker checks is
   a different product, not a cheaper version of the same one. A run that
   weakened its checks is a rigor violation: discard it, do not average it in.
2. The protocol is pre-registered. The decision rule below is fixed
   before the first run. Do not adjust thresholds, arms, or the task matrix
   after seeing results — report against the rule as written.
3. Never report a single run. Report median + spread per arm per task
   class. A difference smaller than the spread is a null result.
4. Write results to files. Per-run records and the final report go to
   files; present only the summary tables and verdict in chat.
5. Be concise. Tables and short sentences; no narration of each run.

Parse Arguments

Arguments: $ARGUMENTS

• --task-class <trivial|standard|complex>: Run only one task-matrix class
  (bucket names from knowledge/task-size-classifier.md). Default: all three.
• --runs <n>: Runs per arm per task class (default: 5). Fewer than 5 makes
  the result exploratory, not decidable — say so in the report and do not
  apply the decision rule to it.
• --dry-run: Print the run plan — interleaved schedule, selected fixtures,
  each arm's agents' declared model:/effort: frontmatter, and a cost
  estimate — without dispatching anything.

Arms (3, matched rigor)

Arm	Configuration
A — Solo	Single session, no subagent dispatch; the session model does all reading and all work itself.
B — Current orchestration	The dev-team pipeline exactly as shipped.
C — Delegation-only sweep	Arm B plus the #1093 sweep rule: the orchestrator never reads beyond classification-sized probes; all bulk reading goes to low-band workers with scoped briefs per the #1092 batching rules.

All three arms run the identical task with identical acceptance checks (the
fixture's eval assertions). See constraint 1: unequal rigor invalidates the
run.

Task matrix

Sample across the knowledge/task-size-classifier.md buckets
(trivial | standard | complex) — one task class per bucket:

Class	Bucket	Example	Expectation
Small single-file	trivial	One-file bug fix from an eval fixture	Expected worst case for delegation
Medium few-file	standard	Feature slice touching 3–5 files	Near the predicted crossover
Large sweep-heavy	complex	/code-review over a 20+ file diff; repo-wide audit	Expected best case for delegation

Reuse existing /agent-eval fixtures under evals/fixtures/ wherever
possible; add fixtures only where a class has none.

Repetitions and isolation

• ≥5 runs per arm per task class (see --runs). Report median + spread
  (min/max or IQR) — never a single run. A difference smaller than the spread
  is a null result.
• Isolation via /headless-run (skills/headless-run/SKILL.md): each run
  dispatches through
  skills/headless-run/scripts/isolated_dispatch.py — fresh session id,
  clean HOME and config dir, scrubbed env — so runs are independent and no
  parent-session identity or state carries over.
• Cache parity: interleave arm order — A, B, C, A, B, C, … Never batch
  all runs of one arm together; batching gives later arms warmer caches.
• Model pinning: record the exact model ID(s) per run. All arms dispatch
  the same agents, each resolved by the harness from its own model:/
  effort: frontmatter (ADR 0026) — record that frontmatter before the
  first run and include it in the report. If any dispatched agent's
  frontmatter changes mid-experiment, abort and restart; do not mix runs
  across frontmatter versions.

Metrics (instrument named per metric)

Metric	Instrument
Dollar cost & tokens by model/thread	python3 ${CLAUDE_PLUGIN_ROOT}/hooks/lib/cost_meter.py report --transcript <path>
Tokens moved to low band (mechanism check)	The cost meter's by_model / by_thread split from the same report
Quality	The fixture's eval assertions — identical across arms
Rework	Fix-loop iterations; .claude/metrics/review-value.jsonl where review checkpoints ran
Wall-clock	Harness transcript timestamps

Mechanism check (mandatory)

Arm C's cost reduction must be produced by the claimed mechanism. If Arm C's
cost drops but reading tokens did not shift to low-band subagent threads
(the cost meter's by_model/by_thread split shows the movement), treat the
result as a rigor violation, not a win — the savings came from reading
less, not from delegating the reading.

Decision rule (pre-registered — fixed before the first run)

Apply per task class:

1. Adopt C over B only if all three hold:
   • median cost reduction ≥ 25%;
   • no quality regression — every run passes the fixture's acceptance
     checks;
   • the mechanism check passes.
2. Retain B over A only if B beats A on cost or quality. If A
   dominates B on a class, report it — that finding feeds
   /harness-audit — do not suppress it.
3. Crossover output: report the task-size threshold below which A/B
   beats C. That number feeds the #1093 sweep rule.

Threats to validity

Each of these must be explicitly addressed in the report — not silently
skipped:

1. Unmatched rigor — an arm passed weaker checks; the comparison is void.
2. Fixed-overhead asymmetry — orchestration pays a constant dispatch tax
   that dominates small tasks and vanishes on large ones.
3. Cache warmth — runs batched by arm give later arms warmer caches;
   interleaving (above) is the control, state whether it held.
4. Model drift — a dispatched agent's model:/effort: frontmatter or an underlying model changed
   mid-experiment; the abort/restart rule (above) is the control.
5. Fixture overfitting — the crossover threshold generalizes only as far
   as the task matrix; say so wherever the threshold is quoted.

Report

Write the report to .dev-team-reports/orchestration-benchmark-<date>.md.

For the header block and closing Provenance section, follow
knowledge/report-template.md; the sections below are this skill's own
body.

Body sections, in order:

1. Per-class results — one table per task class: median + spread per arm
   for each metric (cost, tokens by band, quality pass rate, rework,
   wall-clock).
2. Crossover threshold — the task-size threshold below which A/B beats
   C, with the fixture-overfitting caveat attached.
3. Decision-rule outcomes — per class: adopt/retain verdicts against the
   pre-registered rule, including the mechanism-check result.
4. Threats to validity — all five items above, each with how this run
   addressed it.
5. Run log — each dispatched agent's declared model:/effort:
   frontmatter, exact model IDs per run, interleaved schedule as executed,
   discarded runs and why.

Relationship to other tooling

• /benchmark is runtime-performance-only (web pages); this skill
  measures orchestration value. They share nothing but the name shape.
• /headless-run is the isolation mechanism for every dispatched run;
  /agent-eval fixtures supply the tasks and acceptance checks;
  /cost-report and hooks/lib/cost_meter.py are the cost instruments;
  each agent's model:/effort: frontmatter is the model-pinning source of
  truth (ADR 0026); /harness-audit consumes any "A dominates B" finding.
• Instrumentation flip belongs to the run, not to this skill. After the
  first full documented protocol run (#1099), the instrumentation list in
  plugins/dev-team/CLAUDE.md moves "efficiency gains" from Not yet to
  Instrumented, naming this skill as the instrument. Shipping this skill
  does not perform that flip.