- Apache Airflow
Using airflow and creating DAGs.
We'll cover the following...
Airflow
Airflow is an open-source workflow tool that was originally developed by Airbnb and publically released in 2015. It helps solve scheduling tasks that have many dependencies, which is a challenge many companies face. One of the core concepts in this tool is a graph that defines which tasks to perform and the relationships between these tasks.
DAG
In Airflow, a graph is referred to as a DAG, which is an acronym for directed acyclic graph. A DAG is a set of tasks to perform, where each task has zero or more upstream dependencies. One of the constraints is that cycles, where two tasks have upstream dependencies on each other, are not allowed.
In our pre-configured execution environment, the airflow app is already installed. To skip local installation instructions and learn about creating a DAG, click here.
Getting started #
DAGs are set up using Python code, which is one of the differences from other ...