airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

It is common to abstract and modularize when writing software. We create classes and libraries for other teams and developers to consume. In the Airflow world, as workflows get complex and multiple teams work on pipelines, we may need to abstract away some portions of the pipeline, which are wholly contained or maintained by other teams using SubDAGs.

# SubDAGs
Consider the workflow below.


<br/>

<div style="margin:auto;max-length:18em; max-width:25em;">

![](/api/collection/5352985413550080/6113974358638592/page/5770663905198080/image/6278580703592448)


<br/>

<h5 style="text-align:center"><b>DAG of Example8</h5>

</div>
<br/>


<br/>

<div>

![](/api/collection/5352985413550080/6113974358638592/page/5770663905198080/image/6278580703592448)


<br/>

<h5><b>DAG of Example8</h5>

</div>
<br/>


This lesson explains how complex workflows can be simplified and modularized using SubDAGs.