airflow
Apache Airflow workflow orchestration. Use for data pipelines.
Install
npx skillscat add g1joshi/agent-skills/airflow Install via the SkillsCat registry.
Apache Airflow orchestrates data pipeline workflows by defining directed acyclic graphs of tasks that can be scheduled or triggered by events. It solves the problem of managing complex, interdependent data processing jobs such as ETL, ELT, and model retraining. Use it when you need reliable scheduling, dependency handling, or event‑driven execution of data engineering tasks.
Airflow
Apache Airflow is the standard for data engineering pipelines. v3.0 (2025) introduces Event-driven Triggers and a modern React UI.
When to Use
- ETL/ELT: Scheduling nightly data warehouse loads.
- ML Ops: Retraining models when new data arrives.
- Dependency Management: "Run Task B only if Task A succeeds".
Core Concepts
DAGs (Directed Acyclic Graphs)
Defined in Python.
Task SDK
New in v3.0. Allows writing tasks in any language, not just Python.
Edge Executor
Run tasks on remote edge devices.
Best Practices (2025)
Do:
- Use the TaskFlow API:
@taskdecorators are cleaner thanPythonOperator. - Use Datasets: Define data-aware scheduling (
schedule=[Dataset("s3://bucket/file")]).
Don't:
- Don't put top-level code in DAG files: It runs every scheduler heartbeat.
References
Categories
Install
npx skillscat add g1joshi/agent-skills/airflow Install via the SkillsCat registry.