airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

This lesson explains how cross-communication between tasks takes place using XCom.
XCom is short for “cross-communication,” a feature that allows tasks to communicate between themselves. Cross-communication utilizes Python’s pickling functionality. Python’s pickle module is used to serialize a Python object’s structure into a character stream and to deserialize it back. Objects that can’t be pickled can’t be shared using `xcom`. The object to be shared is pickled and stored in the database with an associated execution date, task instance, and DAG run by the sending task and then retrieved from the database by the intended recipient task. The `xcom_push()` and `xcom_pull()` methods can be used to send and retrieve objects, respectively.

We’ll demonstrate the use of `xcom` by creating a DAG that stitches the Hello World example operator and sensor from the previous lessons. We’ll make slight changes to the operator so that, instead of printing a message on the console, it creates a file on a given path. The operator will run first and then the sensor. The sensor will poll for the path of the file, but, instead of passing the path to the constructor of the sensor, we’ll program the sensor to receive the file path using `xcom`.

Let’s examine the code for the `HelloWorldOperatorXcom` operator first. We pass in the variable name and the associated value in the `xcom_push()` method.

```python
task_instance.xcom_push("pathToCheck", self.path)
```

We also return an integer value,`101`, from the `execute()` method. We’ll see later how the return value can be retrieved by the recipient task.

This lesson explains how cross-communication between tasks takes place using XCom.