airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow

When initially working with Airflow, it is common to get confused with how all the scheduling parameters work together. In this lesson, we'll explore the differences between the various parameters.

When creating a DAG, we can specify the `start_date` and a `schedule_interval` as parameters to the constructor. Let's see an example:

```python
dag = DAG(
    'Example9',
    default_args=default_args,
    description='Example DAG 9',
    schedule_interval='@daily',
    start_date=datetime(2020, 9, 5))
``` 

In the `Example9` DAG, we set `start_date` to `5th Sept 2020`, and the `schedule_interval` is set to `@daily`. Note that `@daily` is an alias for the `0 0 * * *` crontab expression. There are other aliases for commonly used schedules, such as `@weekly`, `@monthly`, and `@yearly`, which all translate to crontab expressions under the hood. You can provide a crontab expression for the `schedule_interval` parameter for complex schedules. A good resource to work with crontab expressions is <a href="https://crontab.guru/" taget="_blank">crontab.guru</a>. Remember that Airflow works with UTC by default but can be configured to work with your local time too. Airflow will also schedule DAG runs for the previous days even though we are running the DAG now. Each DAG run will be associated with an execution date and a start date. The execution date is the date that the DAG should have run, and the start date is when Airflow actually runs it. **Please don't confuse the `start_date` that we pass into the DAG constructor with the start date associated with a DAG run; both are distinct.**


The combination of `start_date` and `schedule_interval` implies that the DAG `Example9` should have run starting from `5th Sept 2020` up until now. Let's further assume that now is `9th Sept 2020` i.e., you are running the DAG on 9th of Sept 2020. Airflow is smart enough to run the missing DAGs for 5th, 6th, 7th, and 8th of Sept for you too. All these four runs will have a start date of Sept. 9th (since all of them were kicked off on Sept. 9th) but will all have different timestamps on the 9th. Additionally, the execution dates for these DAGs will span from Sept. 5th to Sept. 8th.
  

![](/api/collection/5352985413550080/6113974358638592/page/5770521762332672/image/5752921328189440)

As you can see from the screenshot above, Airflow created DAG runs for each of the prior days. The execution date for each DAG run is distinct, but the start dates for all of the runs is the 9th of Sept. Sure, the hours, minutes, etc., differ because they are run at different times on the 9th of Sept. The astute reader would observe that the run for the 9th of Sept. is missing from the list in the screenshot above. This is because Airflow has its roots in ETL, which involves running batch jobs at the end of the day for that day. For instance, the data for the 4th of July will be collected for the entire day, and the ETL job for the 4th of July will actually run at 12 a.m. on the 5th of July. This makes sense because the job for a day should wait to have all the data for that day before running. In our example, the DAG run for the 9th of Sept. isn’t executed until 12 a.m. on the 10th of Sept., which is why it doesn’t show up in the listing in the screenshot. Thinking on the same lines, if the schedule for a DAG is 3 p.m. every day, then the DAG run for the previous day will run immediately after 3 p.m. the next day (and not 12 a.m.). There will be a full 24 hour delay before the DAG run for the previous day executes.

It is interesting to consider what happens if you set the `schedule_interval` to run every Monday and Friday. Let’s say we set the `start_date` for the DAG as August 15th, 2020.

 

|Day   |Date   | DAG Execution Date  | DAG Start Date  |
| - | - | - | - |
| Sunday  | Aug. 15th, 2020  | -  | -  |
| Monday  | Aug. 16th, 2020  | -  | -  |
| Tuesday  | Aug. 17th, 2020  | -  | -  |
| Wednesday  | Aug. 18th, 2020  | -  | -  |
| Thursday  | Aug. 19th, 2020  |  - | -  |
| Friday  | Aug. 20th, 2020 | Aug. 16th, 2020 | Aug. 20th, 2020  |
| Saturday  | Aug. 21st, 2020 | - | -  |
| Sunday  | Aug. 22nd, 2020  |  - | -  |
| Monday  | Aug. 23rd, 2020  | Aug. 20th, 2020 | Aug. 23rd, 2020 |
| Tuesday  | Aug. 24th, 2020  |  - | -  |
| Wednesday  | Aug. 25th, 2020  |  - | -  |
| Thursday  | Aug. 26th, 2020  |  - | -  |
| Friday  | Aug. 27th, 2020 | Aug. 20th, 2020  | Aug. 27th, 2020 |
| Saturday  | Aug. 28th, 2020 |  - | -  |

The DAG’s start date is set to Aug. 15th, 2020 (not to be confused with a DAG run’s start date, as shown in the table above). The first DAG run is for Aug. 16th, 2020, but it’ll start on Friday, Aug. 20th, 2020! This is an idiosyncrasy of Airflow that can confuse seasoned engineers. The DAG runs for Aug. 16th, 2020 actually runs on Aug. 20th, 2020; the execution date for the DAG is Aug. 16th, 2020, but the start date is Aug. 20th, 2020. The DAG run with the execution date of Aug. 20th, 2020 will have a start date of next Monday, i.e., Aug. 23rd, 2020, and so on and so forth.


Finally, you may note that a DAG doesn’t execute at exactly the time it is supposed to run, e.g., a DAG run may start at 10:01 p.m. when its schedule asks it to run at 10:00 p.m. There may be a delay of a few seconds or so. There’s a configuration parameter, `scheduler_heartbeat_sec`, defined in `airflow.cfg` that controls how often the Airflow scheduler runs. The scheduler runs and looks for tasks to trigger, and there may be a delay in when a task becomes due and when the scheduler is able to run it. Making the scheduler run at a higher frequency can put pressure on the database, so any tweaks should be done cautiously.

This lesson clarifies the working of schedule_interval and start_date, which can be confusing for complex crontab expressions.

__default

Scheduling