airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

In this lesson, we'll explore the working of the browser UI of Airflow's web server and interact with Airflow through it. You may click on the widget below and wait for the UI to be loaded. Once loaded, click on the tiny arrow-in-a-box icon on the right, which will open the UI for you in a new browser tab. You can then follow these exercises:

- The landing page for the web server will take you to a list of DAGs. All the examples we ran through the command line interface will also appear with the owner column set to DataJek. In contrast, Airflow also comes with its own set of examples and should be identifiable with the owner column set to airflow or Airflow.

- Feel free to run any example that we covered in the earlier lessons by clicking on the button next to the DAG name. There are also other UI actions you can perform, such as refreshing the status, checking the logs, editing the metadata about a workflow, etc. Since these actions are self-explanatory, we won't delve into them.

- The DAG names are clickable, and you can click on them to get fine-grained information about a DAG. You'll also see a visual representation of the DAG in graph and tree views, which are really helpful to understand complex workflows.

- Click on `Admin -> Pool` and you'll be taken to the Pools UI page. You should find a single pool by the name of `default_pool` with 128 worker slots.

- Click on `Admin -> Connections` and you should see a long list of predefined connections that come with Airflow’s installation. The connection we made, `conn_to_datajek`, should also be on the list. Click on the pencil icon beside the connection name and you’ll see the connection form that we discussed earlier. On the main connections UI page, you should also see a `Create` tab that allows for creating new connections.

- Turn on the `Example3` DAG and wait for it to complete. Once completed, you can navigate to `Admin -> XComs` and observe the return value of 101 from the task and the `pathToCheck` variable listed. Here is a screenshot

![](/api/collection/5352985413550080/6113974358638592/page/4638098813091840/image/5485518364606464)


- Navigate to `Admin -> Configuration`, which should show you the contents of `airflow.cfg`. By default,  the web server doesn’t display the config in the browser unless the `expose_config` setting is set to true.

- Navigate to `Admin -> Users` and you'll find an empty list since we haven't created any users. You can create new users using the `Create` tab.

- Airflow allows for arbitrary content to be saved as key-value pairs, called variables. You may want to expose some key-value pairs outside of your code using Airflow variables while accessing them inside your DAG. This would allow you to modify the variable from the UI. Navigate to `Admin -> Variables` and you'll find no listed variables. You can create one using the `Create` tab. If you have several dozen variables, you can upload them as a JSON file rather than adding each one by hand. Variables are especially useful when working with templates, which we have not covered in this course. Additionally, Airflow also allows for creating and managing variables using environment variables.

- Enable the `Example10` DAG and wait for it to complete. Next, navigate to `Browse -> SLA Misses` and you should see `Example10` listed there. A screenshot is presented below. Since `Example10` is scheduled to run after every two minutes and designed to miss SLA every time, you should see multiple DAG runs for `Example10`.

![](/api/collection/5352985413550080/6113974358638592/page/4638098813091840/image/6541299658784768)

- After you have enabled a few DAGs and some of them have run, navigate to `Browse -> TaskInstances`. You should see a list of task instances (the basic unit of work in Airflow) from different DAG runs along with their status. Here is a screenshot:


![](/api/collection/5352985413550080/6113974358638592/page/4638098813091840/image/5091004999270400)

- The other three links, Logs, Jobs, and DAG runs, under `Browse` are self-explanatory, and you should take a minute to examine each.

- Recall that Airflow stores all metadata information about DAGs, tasks, pools, etc. in a database. Behind the scenes, we are running MySQL as the backing database. You can execute queries against the database by navigating to `Data_Profiling -> Ad Hoc Query`. Select `airflow_db` from the dropdown if it's not already selected. Execute the following query in the query editor and you shall see all the tables in the database.

```mysql
SHOW TABLES;
```

- We can also grab the task instances' metadata using the query editor and issuing the following query:

```mysql
select * from task_instance;
```

- In fact, we can run queries against any database from the ad hoc query editor for which we have created a connection object.

- The  `Data_Profiling -> Charts` link can be used to graph results to create charts and data visualizations from SQL queries, but we don’t cover the advanced details in this course.

This lesson gives a practical exposure to Airflow's web server UI.

__default

Working With the UI