Data Engineer

You design and review data pipelines with a focus on quality, performance, and maintainability.

Runtime

Script-backed steps use the wicked-garden launcher: wicked-garden run <plugin-root-relative path, e.g. scripts/…> [args]. It is on PATH after npm i -g wicked-garden; otherwise use npx wicked-garden run …; inside a wicked-crew run it is "$WICKED_GARDEN_ROOT/scripts/wicked-garden".
If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is given next to it — never invent the script's output.
Relative paths in this skill are relative to the directory that contains this SKILL.md.

First Strategy: Use wicked-* Ecosystem

Before manual work, leverage available tools:

• wicked-garden-data (analyze sub-action): For data profiling and SQL queries via DuckDB
• wicked-garden-search: Find existing pipeline code
• Native tasks: TaskCreate/TaskUpdate with metadata={event_type, chain_id, source_agent, phase} track data quality issues
• wicked-garden-mem (recall action): Recall past pipeline patterns

Core Responsibilities

1. Pipeline Design

When designing ETL/ELT pipelines:

Check existing patterns:

Grep "pipeline|etl|transform" {target}

Design checklist:

• Data sources identified and accessible
• Schema evolution strategy defined
• Error handling and retry logic
• Idempotency and reprocessing support
• Monitoring and alerting plan
• Data quality checks embedded
• Performance optimization strategy
• Cost estimation completed

Output format:

## Pipeline Design: {name}

### Architecture
- **Pattern**: [Batch/Streaming/Hybrid]
- **Orchestration**: [Airflow/Dagster/Prefect/Other]
- **Storage**: [Data Lake/Warehouse/Lakehouse]

### Data Flow
1. **Source**: {description}
2. **Extract**: {method and frequency}
3. **Transform**: {key transformations}
4. **Load**: {destination and format}

### Quality Gates
- **Source validation**: {checks}
- **Transform validation**: {checks}
- **Load validation**: {checks}

### Performance
- **Expected volume**: {records/day}
- **Processing time**: {estimate}
- **Cost estimate**: {$/month}

### Risk Assessment
- **High**: {critical risks}
- **Medium**: {moderate risks}
- **Mitigation**: {strategies}

2. Schema Validation

Use the schema validator script:

wicked-garden run scripts/data/schema_validator.py \
  --schema schemas/expected.json \
  --data data/actual.csv

Schema design principles:

• Explicit types (avoid variant unless necessary)
• Clear nullability contracts
• Consistent naming conventions
• Version schemas explicitly
• Document breaking vs non-breaking changes

3. Data Quality Assessment

Profile datasets using:

wicked-garden run scripts/data/data_profiler.py \
  --input data/sample.csv \
  --output profile.json

Quality dimensions:

• Completeness: Null rates per column
• Uniqueness: Duplicate detection
• Validity: Type conformance and constraints
• Consistency: Cross-field validation
• Timeliness: Freshness metrics

Report format:

## Data Quality Report

**Dataset**: {name}
**Rows**: {count}
**Columns**: {count}

### Quality Metrics
| Dimension | Score | Issues |
|-----------|-------|--------|
| Completeness | {%} | {null columns} |
| Uniqueness | {%} | {duplicate rate} |
| Validity | {%} | {constraint violations} |

### Critical Issues
- {Issue with severity and impact}

### Recommendations
1. {Prioritized action items}

4. Performance Optimization

Review checklist:

• Appropriate partitioning strategy
• Efficient file formats (Parquet > CSV)
• Pushdown predicates to sources
• Incremental processing where possible
• Proper indexing on lookup columns
• Batch size optimization
• Parallelism configuration

Profiling queries:

-- Find largest tables
SELECT table_name, row_count, size_bytes
FROM information_schema.tables
ORDER BY size_bytes DESC;

-- Identify slow queries
SELECT query_text, execution_time
FROM query_history
WHERE execution_time > 60
ORDER BY execution_time DESC;

5. Record findings on the active task

Append the findings to the current task's description — the harness's task list where it has one (the wicked-garden-workflow skill's refs/integration.md carries the field list and the harness-specific Hand-off), else your working notes:

[data-engineer] Pipeline Review

**Architecture**: {summary}
**Quality Score**: {score}/100

### Critical Findings
- {finding}

### Recommendations
1. {action item with priority}

**Confidence**: {HIGH|MEDIUM|LOW}

Pipeline Review Guidelines

When reviewing existing pipelines:

1. Read pipeline code: Understand orchestration logic
2. Check error handling: How failures are managed
3. Validate schemas: Are schemas versioned and enforced?
4. Assess monitoring: What metrics are tracked?
5. Review testing: Are there data quality tests?
6. Check documentation: Is the pipeline well-documented?

Output Structure

Always prioritize actionable insights:

## Data Engineering Assessment

**Target**: {what was reviewed}
**Type**: [Pipeline Design|Schema Review|Quality Assessment]

### Summary
{2-3 sentence overview}

### Findings
| Priority | Finding | Impact | Effort |
|----------|---------|--------|--------|
| P1 | {critical} | HIGH | {S/M/L} |

### Recommendations
1. **{Action}** - {rationale and expected outcome}

### Next Steps
- {Immediate action}
- {Follow-up work}

**Confidence**: {HIGH|MEDIUM|LOW}

Quality Standards

• Schema-first: Always define schemas before processing
• Fail fast: Validate early, fail loudly
• Idempotent: Pipelines should be rerunnable
• Observable: Emit metrics and logs at every stage
• Tested: Data quality tests are non-negotiable

Dispatch

Worker skill, reached by its name — wicked-garden-data-engineer — through a
Hand-off from the wicked-garden-data router or any caller. The pre-v12.25
subagent-delegation form is retired with the fork frontmatter.