Gate-Result Benchmark Re-baseline

Operational procedure for updating tests/crew/benchmark_baseline.json after a legitimate perf change on main.

When to re-baseline

Re-baseline only when a deliberate perf change lands on main. Examples:

• Validator hardening (added / removed gate-result schema checks)
• Cache eviction tuning (changes to gate-result ingestion memoization)
• Schema expansion (new required fields in gate-result.json)
• Sanitizer pattern changes (scripts/qe/content_sanitizer.py)

Do not re-baseline to silence a regression. If the benchmark fails on a PR that didn't intend perf work, treat it as a real regression and find the cause.

How the lane works

• Workflow: .github/workflows/benchmark.yml
• Trigger: PRs touching scripts/crew/phase_manager.py, the gate-result schema/sanitizer (scripts/qe/content_sanitizer.py), or the benchmark test/baseline themselves. Other PRs skip the lane to keep default CI cost flat.
• Test: tests/crew/test_gate_result_benchmark.py::test_load_gate_result_p95_within_2x_baseline
• Marker: benchmark — opt-in. Local uv run pytest deselects it (see pyproject.toml addopts = "-m 'not benchmark'").
• Explicit invocation: uv run pytest -m benchmark
• Fixtures: 50 deterministic gate-result.json files cycling through 1 KB / 4 KB / 16 KB / 60 KB (bounded by MAX_SUMMARY_BYTES). Each round clears the memoization cache so the full validate + sanitize + cache-insert path is measured. Cache-hit timing is not part of the SLO.
• Baseline: tests/crew/benchmark_baseline.json — p95_ns in nanoseconds.
• Assertion: p95_current ≤ slo_multiplier × p95_baseline (default slo_multiplier = 2.0). The workflow comments the delta on every triggered PR.
• Missing-baseline behavior: if benchmark_baseline.json is deleted or malformed, the test soft-skips with a directive to record one. The SLO is not enforced until a valid baseline is present — so a re-baseline PR can merge without a circular dependency.

Re-baseline procedure

1. Check out the target main commit.

   git checkout main && git pull

2. Run the benchmark three times. The SLO is against p95, and CI noise varies — taking the highest of three local runs adds margin.

   uv run pytest -m benchmark tests/crew/test_gate_result_benchmark.py -s
   uv run pytest -m benchmark tests/crew/test_gate_result_benchmark.py -s
   uv run pytest -m benchmark tests/crew/test_gate_result_benchmark.py -s

3. Record the highest p95_current reading from the three runs.

4. Update tests/crew/benchmark_baseline.json:

   Field	New value
   p95_ns	The highest p95_current from step 3, rounded up
   recorded_on	Today's date (YYYY-MM-DD)
   recorded_from_commit	Short SHA of the target main commit
   slo_multiplier	Leave at 2.0 unless the AC-11 contract changes
   rebaseline_procedure	Leave at wicked-garden:platform:gate-benchmark-rebaseline

5. Commit the baseline update in a dedicated PR. Title convention:

   chore(benchmark): re-baseline AC-11 after {change-summary}

6. The benchmark workflow runs on the PR and must pass (p95 should be well under 2× the new baseline since you just measured it).

Pre-flip dependency (semantic-gate condition)

Strict mode for gate-result ingestion (WG_GATE_RESULT_STRICT_AFTER, default 2026-06-18) requires the AC-11 benchmark lane to be active in CI. Rationale: once strict-mode activates, a silent 2×+ perf regression would push every approve_phase call over the SLO without a guard.

If the benchmark lane is broken or disabled on main:

• Before the flip date: push WG_GATE_RESULT_STRICT_AFTER out (env var or default-constant update). Do not let strict-mode activate without benchmark enforcement.
• After the flip date: fix the benchmark lane before touching the ingestion path.

Env-var rollback (preferred over revert)

For production rollback of a specific ingestion check, prefer env-var soft-disable over git-revert:

Variable	Effect
WG_GATE_RESULT_SCHEMA_VALIDATION=off	Skip schema validator
WG_GATE_RESULT_CONTENT_SANITIZATION=off	Skip content sanitizer
WG_GATE_RESULT_DISPATCH_CHECK=off	Skip dispatch-log orphan check

All flags auto-expire at WG_GATE_RESULT_STRICT_AFTER.

References

• Test: tests/crew/test_gate_result_benchmark.py
• Baseline: tests/crew/benchmark_baseline.json
• Workflow: .github/workflows/benchmark.yml
• Ingestion path: scripts/crew/phase_manager.py (gate-result load + memoization)
• Sanitizer: scripts/qe/content_sanitizer.py