Challenge: Build an FXRate Reporting Pipeline with Airflow

Problem statement

In this challenge, we will create a foreign exchange rate reporting pipeline in Airflow. The pipeline fetches the EUR-to-USD exchange rate of the current day from Frankfurter API every minute from Monday to Friday and ingests it into a BigQuery table. Every five minutes from Monday to Friday, an analysis is performed which calculates the average rate of the current day so far and saves the result into another BigQuery table. The latest analysis of the day will overwrite any previous analysis conducted on the same day.

Note:

• For the sack of quick testing, we make the job frequency to every minute and every five minutes. But the rate normally doesn't change within five minutes.

• It's an open question, so there is no definite answer. We should encourage ourselves to come up with different solutions and think about their pros and cons.

Sample data from Frankfurter API

To give us an idea of what the API response looks like, here's an example below. The endpoint always returns the latest exchange rate for a specific date.