Workflow Orchestration
In the context of Data Engineering, workflow orchestration refers to the process of scheduling and arranging tasks that form your [[Data Pipeline|data pipeline]]. A workflow orchestration tool allows you to schedule, run, and observe the entire process.
[[Apache Airflow]] [[Dagster]] [[Prefect]]
- Create complex custom workflows
- Makes it easier to create [[Idempotence|idempotent]] workflows
- Alert you if something fails
- Allows you to gracefully retry and recover from failures
- Adds complexity in scheduling
- Requires additional infrastructure and maintenance costs