Creating a Sensor

In the previous lesson, we created an operator that allowed us to execute any workload as part of Airflow’s DAG execution. Now, we’ll look into a variant of the operator, called a sensor. A sensor can be used to monitor for a condition to become true or a criterion to be met. A sensor is created by deriving from the base class, BaseSensorOperator, and overriding its poke function. The poke function is repeatedly invoked every poke_interval seconds until one of the following occurs:

1. The poke method returns true. If it returns false, it will be invoked again.

2. The poke method raises an AirflowSkipException, in which case, the sensor task instance’s status will be set to Skipped.

3. The poke raises another exception, in which case it will be retried until the maximum number of retries have been attempted.

Let’s write an example sensor that polls for the existence of a file. Let’s call our sensor the HelloWorldSensor. Here is the code for the sensor: