Workflow Tools
Introduction to workflow tools and their benefits.
We'll cover the following
Cron is useful for simple pipelines but runs into challenges when tasks have dependencies on other tasks that can fail. To help resolve this issue, where tasks have dependencies and only portions of a pipeline need to be rerun, we can leverage workflow tools. Apache Airflow is currently the most popular tool, but other open-source projects are available and provide similar functionality including Luigi and MLflow.
Get hands-on with 1200+ tech skills courses.