airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

Airflow operators and sensors don't operate in isolation; rather, they can connect to systems outside of Airflow, such as databases, cloud systems, remote machines, etc., using _hooks_ and _connections_, which contain the necessary information required to connect to external systems.

# Hooks
All hooks derive from the base class, `BaseHook`, defined [here](https://github.com/apache/airflow/blob/main/airflow/hooks/base.py). Hooks are meant as an interface to interact with external systems. Popular hook implementations, such as `MySqlHook`, `HiveHook`, and `PigHook` return objects, can handle the connection and interaction to specific instances of these systems and expose consistent methods to interact with them. Hooks can also be used to retrieve authentication and hostname information using the `airflow.models.connection.Connection` model. This saves the developer from hardcoding sensitive information inside the DAG code.

You can explore all the hooks that come with Airflow under <a href="https://github.com/apache/airflow/tree/master/airflow/providers" target="_blank">this</a> directory. As an example, here's the <a href="https://airflow.readthedocs.io/en/stable/_modules/airflow/hooks/mysql_hook.html" target="_blank">MySQL hook</a> from the Airflow code base. Note that the hook doesn't contain the logic on how queries are executed against the MySQL server. Instead, it contains the logic on how to connect and interface with the MySQL server. A hook is a mechanism to communicate with external shared resources in a DAG. For example, multiple tasks in a DAG can require access to a MySQL database. Instead of creating a connection per task, you can retrieve a connection from the hook and utilize it. With hooks, we don't need to store authentication parameters within the DAG.

# Connections
Hooks allow us to connect to external systems, and the information needed to make the connection, such as hostname, port, credentials, etc., is stored in a connection. Within the UI, you can access the connections in the `Admin->Connections` section. Later, the pipeline code that you write can reference a connection object using the `conn_id` parameter. Below is a screenshot of the web server's UI, listing various connections.

This lesson explores the use of hooks and connections.

__default

Hooks and Connections

Hooks