Hooks and Connections

Airflow operators and sensors don’t operate in isolation; rather, they can connect to systems outside of Airflow, such as databases, cloud systems, remote machines, etc., using hooks and connections, which contain the necessary information required to connect to external systems.

Hooks

All hooks derive from the base class, BaseHook, defined here. Hooks are meant as an interface to interact with external systems. Popular hook implementations, such as MySqlHook, HiveHook, and PigHook return objects, can handle the connection and interaction to specific instances of these systems and expose consistent methods to interact with them. Hooks can also be used to retrieve authentication and hostname information using the airflow.models.connection.Connection model. This saves the developer from hardcoding sensitive information inside the DAG code.

You can explore all the hooks that come with Airflow under this directory. As an example, here’s the MySQL hook from the Airflow code base. Note that the hook doesn’t contain the logic on how queries are executed against the MySQL server. Instead, it contains the logic on how to connect and interface with the MySQL server. A hook is a mechanism to communicate with external shared resources in a DAG. For example, multiple tasks in a DAG can require access to a MySQL database. Instead of creating a connection per task, you can retrieve a connection from the hook and utilize it. With hooks, we don’t need to store authentication parameters within the DAG.

Connections

Hooks allow us to connect to external systems, and the information needed to make the connection, such as hostname, port, credentials, etc., is stored in a connection. Within the UI, you can access the connections in the Admin->Connections section. Later, the pipeline code that you write can reference a connection object using the conn_id parameter. Below is a screenshot of the web server’s UI, listing various connections.