Search⌘ K
Home
Courses
Data Science in Production: Building Scalable Model Pipelines

- Cloud Cron

Explore how to schedule model pipeline tasks on Google Kubernetes Engine using cron expressions. Learn to push container images to Container Registry, create and connect to GKE clusters, and configure scheduled workflows using YAML files and kubectl commands. This lesson empowers you to automate your model deployment pipelines within a cloud environment effectively.

We'll cover the following...

Introduction

Both AWS and GCP provide managed cron options. With AWS it’s possible to schedule services to run on ECS using cron expressions, and the GCP Kubernetes Engine provides scheduling support for containers.

Cloud data

Crons in GKE

In this lesson, we’ll explore the GKE option because it is simpler to set up. The first step is to push our model pipeline image to Container Registry using the following commands:

Shell
cat dsdemo.json | sudo docker login -u _json_key --password-stdin https://us.gcr.io
sudo docker tag sklearn_pipeline us.gcr.io/[project_id]/sklearn_pipeline
sudo docker push us.gcr.io/[project_id]/sklearn_pipeline

We will be performing these steps in the code widget at the end of the lesson.

Creating a cluster

Next, we’ll set up Kubernetes to schedule the image to run on a managed cluster. Unlike in ...