ETL Pipeline Example: Airflow Extraction Task

After completing the previous task, we now have a function called extract for extracting the latest batch of data from the production database in a file called helper.py. We’ll add this function as a task in an airflow DAG to complete the extract step.

This is how we build our pipelines using airflow, by adding more and more tasks to a particular DAG.

Airflow operators

Before showing how to do that, let’s discuss airflow operators. In Airflow, operators are the building blocks that define tasks in a pipeline. Each operator is different and can be used to perform a particular type of task, such as executing a SQL query, transferring files, running a Python script, etc. Some common operators are: ...