Conclusion : Workflow Tools for Model Pipelines

In this chapter, we explored a batch model pipeline for applying a machine learning model to a set of users and storing the results in BigQuery. To make the pipeline portable, so that we can execute it in different environments, we created a Docker image to define the required libraries and credentials for the pipeline.

We then ran the pipeline on an EC2 instance using batch commands, cron, and Airflow. We also used GKE and Cloud Composer to run the container via Kubernetes.