Retrain AI Evaluation Weights

Use when the user wants to retrain AI weights from 17Lands data, add new training datasets, update learned weight values in Rust, or run CMA-ES optimization.

Architecture Overview

The AI weight system has 4 layers:

1. Base weights (EvalWeightSet in crates/phase-ai/src/eval.rs) — 9 weights × 3 game phases (early T1-3, mid T4-7, late T8+). Learned from 17Lands replay data.
2. Archetype multipliers (ArchetypeMultipliers in crates/phase-ai/src/deck_profile.rs) — 5 archetypes × 9 multipliers. Scale base weights per deck type.
3. Keyword bonuses (KeywordBonuses in crates/phase-ai/src/eval.rs) — 10 params for creature evaluation.
4. Policy penalties (PolicyPenalties in crates/phase-ai/src/config.rs) — tactical policy score knobs.
5. AiProfile (crates/phase-ai/src/config.rs) — 3 params (risk_tolerance, interaction_patience, stabilize_bias).

All stored in AiConfig. The CMA-ES optimizer tunes one parameter group per
run via --group eval|penalties|keywords|archetype:

• eval: 9 late-game EvalWeights plus 3 AiProfile values. Early/mid weights are derived from the 17Lands phase ratios.
• penalties: every field listed in ACTIVE_POLICY_PENALTY_FIELDS.
• keywords: all KeywordBonuses fields.
• archetype: 5 archetypes x 9 ArchetypeMultipliers.

Do not mix groups in one run. Compare and validate one group artifact at a
time so regressions can be attributed to a specific surface.

Training Data Setup

Data location: data/17lands/ (gitignored)

Required files from 17Lands (https://www.17lands.com/public_datasets):

• replay_data_public.{SET}.PremierDraft.csv — Per-turn board state snapshots. Premier Draft (Bo1) is best: largest dataset, no sideboard confounds, human-drafted decks.
• cards.csv — Arena card ID to mana value mapping.

To add new sets: Download CSVs and symlink or copy into data/17lands/:

ln -s ~/Downloads/replay_data_public.FDN.PremierDraft.csv data/17lands/
ln -s ~/Downloads/replay_data_public.DSK.PremierDraft.csv data/17lands/
# cards.csv only needed once (shared across sets)
ln -s ~/Downloads/cards.csv data/17lands/

The script auto-discovers all replay_data_public.*.PremierDraft.csv files.

Retraining Steps

Step 1: Run the training script

rtk python3 scripts/train_eval_weights.py --data-dir data/17lands --output data/learned-weights.json

Dependencies: pip3 install -r scripts/requirements-training.txt (pandas, scikit-learn, numpy)

What it does:

• Streams all replay CSVs with skill filter (win_rate >= 0.55, games >= 50)
• Splits samples into 3 turn-phase buckets (early T1-3, mid T4-7, late T8+)
• Trains separate logistic regression per phase
• Maps 5 features to EvalWeights fields: life_diff→life, creature_count_diff→board_presence, creature_mv_diff→board_power, hand_diff→hand_size, non_creature_diff→card_advantage
• Scales so max coefficient = 2.5
• 4 weights stay hand-tuned: board_toughness=1.0, aggression=0.5, zone_quality=0.3, synergy=0.5

Output: data/learned-weights.json with per-phase weights and accuracy metrics.

Step 2: Update Rust with new values

Read data/learned-weights.json and update EvalWeightSet::learned() in crates/phase-ai/src/eval.rs:

pub fn learned() -> Self {
    EvalWeightSet {
        early: EvalWeights {
            life: /* phases.early.weights.life */,
            aggression: /* phases.early.weights.aggression */,
            board_presence: /* phases.early.weights.board_presence */,
            board_power: /* phases.early.weights.board_power */,
            board_toughness: /* phases.early.weights.board_toughness */,
            hand_size: /* phases.early.weights.hand_size */,
            zone_quality: /* phases.early.weights.zone_quality */,
            card_advantage: /* phases.early.weights.card_advantage */,
            synergy: /* phases.early.weights.synergy */,
        },
        mid: EvalWeights { /* same pattern from phases.mid.weights */ },
        late: EvalWeights { /* same pattern from phases.late.weights */ },
    }
}

Step 3: Verify

rtk cargo fmt --all
rtk ./scripts/tilt-wait.sh --timeout 420 clippy test-ai

If Tilt is not running (rtk tilt get uiresource clippy fails), use the
project-reference skill before falling back to direct cargo commands.

Step 4 (optional): CMA-ES optimization

# Smoke test (fast, verifies binary works)
rtk cargo tune-ai data/ --group eval --generations 2 --population 5 --games 3 --seed 42

# Full eval/profile run
rtk cargo tune-ai data/ --group eval --generations 100 --population 50 --games 20 --output data/cma-tuned-eval.json

# Full policy-penalty run
rtk cargo tune-ai data/ --group penalties --generations 100 --population 50 --games 20 --output data/cma-tuned-penalties.json

# Full keyword-bonus run
rtk cargo tune-ai data/ --group keywords --generations 100 --population 50 --games 20 --output data/cma-tuned-keywords.json

# Full archetype-multiplier run
rtk cargo tune-ai data/ --group archetype --generations 100 --population 50 --games 20 --output data/cma-tuned-archetype.json

# Validate a tuned artifact against paired holdout matchups and Easy/Medium/Hard opponents
rtk cargo tune-ai data/ --validate --games 500 --output data/cma-tuned-eval.json

CMA-ES writes the requested artifact plus a sibling *-manifest.json
containing the git SHA, seed, group, parameter names, fitness decks, holdout
decks, opponent pool, games/eval, paired-seed flag, draw-exclusion flag, and
baseline config hash. Do not write CMA output to data/learned-weights.json;
that path is reserved for the 17Lands logistic-regression artifact
(kind: 17lands_phase_weights). After a full run, review the validation
output before manually updating Rust defaults.

Fitness uses registered duel_suite matchup IDs from the fitness split and
paired mirrored seeds. Drawn games are excluded from the fitness denominator.
Holdout validation uses a separate registered split and compares baseline vs
learned on the same seeds against Easy, Medium, and Hard opponent configs.

Commander sanity measurement lives in ai-duel, not ai-tune:

rtk cargo run --release --bin ai-duel -- client/public --commander-suite --games 8 --seed 42 \
  --difficulty Hard --baseline-difficulty Medium \
  --output target/commander-suite-results.json

This runs the candidate seat through four seat rotations against three
baseline seats and reports win rate, survival turns, and elimination order.

Key Files

File	Purpose
scripts/train_eval_weights.py	Python training pipeline
scripts/requirements-training.txt	Python deps (pandas, scikit-learn, numpy)
data/learned-weights.json	Trained weight artifact (committed)
data/cma-tuned-weights.json	CMA-ES tuned artifact (generated, not a 17Lands artifact)
data/cma-tuned-weights-manifest.json	CMA-ES reproducibility manifest
data/17lands/	Raw 17Lands CSVs (gitignored)
crates/phase-ai/src/eval.rs	EvalWeights, EvalWeightSet, KeywordBonuses, evaluation functions
crates/phase-ai/src/deck_profile.rs	ArchetypeMultipliers, deck classification
crates/phase-ai/src/config.rs	AiConfig with all tunable params
crates/phase-ai/src/bin/ai_tune.rs	CMA-ES optimizer binary
crates/phase-ai/src/bin/ai_duel.rs	Duel-suite, compare, and Commander measurement binary

EvalWeights Fields (9 total)

Field	17Lands Feature	Measures
life	life_diff	Life total differential
board_presence	creature_count_diff	Creature count differential
board_power	creature_mv_diff	Total mana value of creatures
hand_size	hand_diff	Cards in hand differential
card_advantage	non_creature_diff	Non-creature, non-land permanents
board_toughness	—	Total toughness (hand-tuned)
aggression	—	Power bonus when ahead on life (hand-tuned)
zone_quality	—	Hand quality + graveyard value (hand-tuned)
synergy	—	Board synergy bonus (hand-tuned)