uw-ssec

performance-reporting

Generates structured benchmark reports with configuration comparison tables, performance bias analysis, ranked recommendations, and trade-off explanations from Zarr chunking benchmark results. Use when the user has Zarr chunking benchmark data to analyze, needs to compare chunk configurations, wants performance recommendations for Zarr storage, or needs to present chunking strategy results to stakeholders.

uw-ssec 32 14 Updated 21h ago

Resources

2
GitHub

Install

npx skillscat add uw-ssec/savvy/performance-reporting

Install via the SkillsCat registry.

SKILL.md

Performance Reporting

Transform raw Zarr chunking benchmark data into structured markdown reports with comparison tables, bias analysis, and ranked recommendations.

Resources

Resource Purpose
references/report-generation.md Deep reference: section formatting, dataset summaries, environment metadata, table layout, recommendation writing
references/visualization-patterns.md Bar charts, heatmaps, radar charts, scatter plots with matplotlib/hvPlot code
assets/comparison-table-template.md Ready-to-fill comparison table template
assets/recommendation-template.md Ready-to-fill recommendation section template

Report Structure

  1. Dataset Summary — path, shape, dtype, current chunks, compression, total size
  2. Benchmark Environment — platform, Python/library versions, instance type, storage backend, network
  3. Configuration Comparison Table — all configs ranked with key metrics side by side
  4. Per-Pattern Results — detailed tables for each access pattern (spatial, temporal, spectral)
  5. Performance Bias Summary — PB score and classification per config
  6. Recommendations — primary, alternative, configs to avoid
  7. Methodology Notes — run count, cache clearing, statistical method

Validation checkpoint after generating: Verify (a) every configuration appears in the comparison table, (b) each config has a winner highlight in at least one column, (c) memory budget pass/fail is marked for every config, (d) PB score is present for every config.

Key Metrics

Metric Unit Direction Note
Wall-clock time s Lower Primary ranking metric
TTFB ms Lower Interactive responsiveness
Peak memory GB Lower Feasibility under budget
Throughput MB/s Higher Bulk processing
Chunk utilization 0–1 Higher I/O alignment (< 0.5 = poor fit)
Performance Bias (PB) ratio → 1.0 Balance across patterns

Generating the Comparison Table (Python)

import pandas as pd

def build_comparison_table(results: dict, memory_budget_gb: float) -> pd.DataFrame:
    """
    results = {
        "config_label": {
            "spatial_mean": float, "temporal_mean": float, "spectral_mean": float,
            "peak_memory_gb": float
        }, ...
    }
    """
    rows = []
    for label, m in results.items():
        times = [v for k, v in m.items() if k.endswith("_mean")]
        pb = max(times) / min(times) if min(times) > 0 else float("inf")
        rows.append({
            "Config": label,
            "Spatial (s)": f"{m['spatial_mean']:.1f}",
            "Temporal (s)": f"{m['temporal_mean']:.1f}",
            "Peak Mem (GB)": f"{m['peak_memory_gb']:.1f}",
            "PB": f"{pb:.2f}",
            "Within Budget": "✓" if m["peak_memory_gb"] <= memory_budget_gb else "✗",
        })
    df = pd.DataFrame(rows)
    # Mark winner per numeric column
    for col in ["Spatial (s)", "Temporal (s)", "Peak Mem (GB)"]:
        idx = df[col].astype(float).idxmin()
        df.loc[idx, col] = f"**{df.loc[idx, col]}**"
    return df.sort_values("PB")

Performance Bias Score

PB = max(mean_time per pattern) / min(mean_time per pattern)
PB Classification When acceptable
1.0–1.5 Balanced Mixed workloads
1.5–3.0 Moderate Known primary pattern
3.0–10.0 Biased Single-pattern only
> 10.0 Extreme Rarely acceptable

For weighted workloads: Weighted PB = max(w_i × T_i) / min(w_i × T_i).

Ranking and Elimination

Eliminate before ranking configurations that:

  • Exceed the memory budget
  • Have any single-pattern time > 5× the best config for that pattern
  • Show CV (std/mean) > 0.3 (unreliable measurements)

Score remaining configs:

composite_score = sum(weight_i × mean_time_i)  # default: equal weights

Tie-break order: (1) lower PB, (2) lower peak memory, (3) smaller chunk size.

Writing Recommendations

Primary: State chunk shape, composite score or dominant-pattern time, PB classification, peak memory, and one-sentence reasoning.

Alternative: Include when a specialist config is >25% faster on one pattern. State the trade-off explicitly.

Avoid: List dominated configs (worse on all metrics) or budget-exceeding configs with specific reasons.

See assets/recommendation-template.md for the fill-in-the-blank format.

Progressive Disclosure