Interacting with Airflow

After implementing our ETL pipeline, Airflow will execute it continuously according to our DAG’s configuration. During the DAG's lifetime, we might want to interact with Airflow and gather some information about it. A great way to do this is by using the terminal.

Interacting with Airflow

Here are some popular commands we can use to interact with Airflow:

• To generate an airflow cheat sheet:

  • airflow cheat-sheet

• To generate a list of all current DAGS in airflow:

  • airflow dags list

• To check if airflow is able to locate a specific DAG:

  • airflow dags list | grep social_media_pipeline

• To test a single task:

  • airflow tasks test social_media_pipeline extract 2023-08-29

• To test the entire DAG:

  • airflow dags test social_media_pipeline 2023-08-29

• To pause a DAG:

  • airflow dags pause social_media_pipeline

• To unpausing a DAG:

  • airflow dags unpause social_media_pipeline

• To trigger the DAG:

  • airflow dags trigger social_media_pipeline

• To trigger the DAG in the future:

  • airflow dags trigger social_media_pipeline -e <FutureDate>

• To check the tasks in a DAG:

  • airflow tasks list social_media_pipeline

• To check the next execution date of a DAG:

  • airflow dags next-execution social_media_pipeline