Orchestration

Container orchestration

Container orchestration systems are responsible for managing the life cycles of containers in a cluster. They provide services including provisioning, scaling, failover, load balancing, and service discovery between containers. AWS has multiple orchestration solutions, but the general trend has been moving towards Kubernetes for this functionality, which is an open-source platform originally designed by Google.

Container orchestration in data science

One of the main reasons for using container orchestration as a data scientist is to be able to deploy models as containers, where you can:

• scale up infrastructure to match demand,
• have a fault-tolerant system that can recover from errors,
• have a static URL for your service managed by a load balancer.

It’s a bit of work to get one of these solutions up and running, but the end result is a robust model deployment. Serverless functions provide a similar capability, but using orchestration solutions means that you can use any programming language and runtime is necessary to serve your models. It provides flexibility at the cost of operational overhead, but as these tools evolve, the overhead should be reduced.