XCom

This lesson explains how cross-communication between tasks takes place using XCom. XCom is short for “cross-communication,” a feature that allows tasks to communicate between themselves. Cross-communication utilizes Python’s pickling functionality. Python’s pickle module is used to serialize a Python object’s structure into a character stream and to deserialize it back. Objects that can’t be pickled can’t be shared using xcom. The object to be shared is pickled and stored in the database with an associated execution date, task instance, and DAG run by the sending task and then retrieved from the database by the intended recipient task. The xcom_push() and xcom_pull() methods can be used to send and retrieve objects, respectively.

We’ll demonstrate the use of xcom by creating a DAG that stitches the Hello World example operator and sensor from the previous lessons. We’ll make slight changes to the operator so that, instead of printing a message on the console, it creates a file on a given path. The operator will run first and then the sensor. The sensor will poll for the path of the file, but, instead of passing the path to the constructor of the sensor, we’ll program the sensor to receive the file path using xcom.

Let’s examine the code for the HelloWorldOperatorXcom operator first. We pass in the variable name and the associated value in the xcom_push() method.

task_instance.xcom_push("pathToCheck", self.path)

We also return an integer value,101, from the execute() method. We’ll see later how the return value can be retrieved by the recipient task.