Kubeflow: An End-to-end Machine Learning Lab

As we described earlier in this chapter, TensorFlow was designed to utilize distributed resources for training. To leverage this capability, we will use the Kubeflow projects. Built on top of Kubernetes, Kubeflow has several components that are useful in the end-to-end process of managing machine learning applications. To install Kubeflow, we need to have an existing Kubernetes control plane instance and use kubectl to launch Kubeflow’s various components. The steps for setup differ slightly depending on whether we are using a local instance or one of the major cloud providers.

Running Kubeflow locally with MiniKF

If we want to get started quickly or prototype our application locally, we can avoid setting up a cloud account and instead use virtual machines to simulate the kind of resources we would provision in the cloud. To set up Kubeflow locally, we first need to install VirtualBox to run virtual machines and Vagrant to run configurations for setting up a Kubernetes control plane and Kubeflow on VirtualBox VMs.

Once you have these two dependencies installed, create a new directory, change into it, and run:

This initializes the VirtualBox configuration and brings up the application. You can now navigate to http://10.10.10.10/ and follow the instructions to launch Kubeflow and Rok (a storage volume for data used in experiments on Kubeflow created by Arrikto). Once these have been provisioned, you should see a screen like this:

Log in to Kubeflow to see the dashboard with the various components:

We'll return to these components later and go through the various functionalities available on Kubeflow, but first, let’s walk through how to install Kubeflow in the cloud.

Installing Kubeflow in AWS

To run Kubeflow in AWS, we need a Kubernetes control plane available in the cloud. Fortunately, Amazon provides a managed service called EKS, which provides an easy way to provision a control plane to deploy Kubeflow. Follow the following steps to deploy Kubeflow on AWS:

1. Register for an AWS account and install the AWS Command Line Interface: This is needed to interact with the various AWS services, following the instructions ...