← Data Engineering & ETL learning pathData Engineering & ETL · Module 6 of 10
Orchestration with Airflow
Schedule and coordinate pipeline tasks.
📖 Key ideas
- Orchestrators run tasks in the right order on schedule.
- Apache Airflow defines workflows as DAGs in Python.
- Tasks have dependencies: extract >> transform >> load.
- Retries, alerts and backfills handle failures and history.
- Alternatives: Dagster, Prefect, cloud workflow tools.
💡 Real-life example: An orchestrator is a conductor ensuring each musician (task) plays at the right time.
DAGDirected Acyclic Graph of tasks
OperatorAirflow task template
BackfillRunning past dates
RetryRerunning a failed task