Data Engineering Foundations in Python/

...

/

Solution: Build an FXRate Reporting Pipeline with Airflow

Solution: Build an FXRate Reporting Pipeline with Airflow

Discuss the solution to the Airflow challenge.

We'll cover the following...

Solution

Let's check out the suggested solution to this challenge. Here are the tasks we need to perform:

  • Create a task to fetch the exchange rate from the endpoint every minute from Monday to Friday. Make sure that rerunning the DAG on a different date makes the same API request.

  • Create a task to insert data into the BigQuery exchange_rate table. We must ensure idempotency, meaning that running the same Dag Run multiple times will not result in duplicated records. Using the MERGE statement in BigQuery is a good option.

  • Create a task to retrieve data points of the day from the exchange_rate table, ...