airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

Tasks get triggered when tasks upstream to them succeed. This is the default trigger behavior for tasks, but there are different variations that Airflow offers. Consider the following workflow:

<br/>

<div style="margin:auto;max-length:20em; max-width:30em;">

![](/api/collection/5352985413550080/6113974358638592/page/6400557097943040/image/5190142705270784)

<h5 style="text-align:center"><b>DAG of Example11</h5>
</div>
<br/>

In the workflow above, `task5` has the `trigger_rule` set to `all_success` by default. This implies that `task5` will be executed only when tasks directly upstream to it succeed. Due to branching, only one of the tasks, `task3` or `task4`, will be executed, which is why `task5` is never executed and appears skipped. When the `trigger_rule` is set to `all_success` or `all_failed` for a task, the task is skipped if any of its direct parents is skipped. The skipped tasks are said to *cascade* through for `all_success` and `all_failed` trigger rules. Here is the official documentation for the various values for `trigger_rule` and their behavior:

1. `all_success`: (default) All parents have succeeded.

2. `all_failed`: All parents are in a failed or upstream_failed state.

3. `all_done`: All parents are done with their execution.

4. `one_failed`: Fires as soon as at least one parent has failed. It does not wait for all parents to be done.

5. `one_success`: Fires as soon as at least one parent succeeds. It does not wait for all parents to be done.

6. `none_failed`: All parents have not failed (failed or upstream_failed), i.e., all parents have succeeded or been skipped.

7. `none_failed_or_skipped`: All parents have not failed (failed or upstream_failed), and at least one parent has succeeded.

8. `none_skipped`: No parent is in a skipped state, i.e., all parents are in a success, failed, or upstream_failed state.

9. `dummy`: Dependencies are just for show, trigger at will.

# Example 2

As an example, we'll change the `trigger_rule` for `task5` to `one_failed` and then observe how it affects `task5`'s execution. Furthermore, we'll raise `AirflowFailException` in `task2`, which will cause the task to fail. Given the trigger rule for `task5`, it should execute since `task2` fails with an exception. The code for the Example12 DAG appears below, which is the same for the Example11 DAG except that on line 60, Example11 has the trigger rule set to `all_success`.


<br/>

<div>

![](/api/collection/5352985413550080/6113974358638592/page/6400557097943040/image/5190142705270784)

<h5><b>DAG of Example11</h5>
</div>
<br/>

This lesson explains how the trigger behavior of tasks changes with the different settings for the trigger_rule parameter. Additionally, the lesson also covers unexpected task failures.

__default

Task Triggers and Failures

DAG of Example11