airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

In the previous lesson, we created an operator that allowed us to execute any workload as part of Airflow’s DAG execution. Now, we’ll look into a variant of the operator, called a sensor. A sensor can be used to monitor for a condition to become true or a criterion to be met. A sensor is created by deriving from the base class, `BaseSensorOperator`, and overriding its `poke` function. The `poke` function is repeatedly invoked every `poke_interval` seconds until one of the following occurs:

1. The `poke` method returns true. If it returns false, it will be invoked again.

2. The `poke` method raises an `AirflowSkipException`, in which case, the sensor task instance’s status will be set to `Skipped`.

3. The `poke` raises another exception, in which case it will be retried until the maximum number of `retries` have been attempted.

Let's write an example sensor that polls for the existence of a file. Let's call our sensor the `HelloWorldSensor`. Here is the code for the sensor:

This lesson delves into creating an Airflow sensor.

__default

Creating a Sensor