Design of Distributed Task Scheduler

We identified requirements for a distributed task scheduler in the previous lesson. In this lesson, we will map those requirements to a design.

Components

We can consider scheduling at many levels. It could be scheduling that is done internally by an organization to run their tasks on their own cluster of machines, and they have to find ample resources and need to decide which task to run first. On the other hand, It could be scheduling that a cloud provider uses to schedule tasks coming from multiple clients. Cloud providers need to decide which task to run first and which clients to handle first to provide appropriate isolation between different tenants. So, in general, the big components of the system that we have are:

• Clients that initiate task execution.
• Resources on which the task actually executes.
• A scheduler between clients and resources that decides which task should get the resource first.

As shown in the above illustration, it is necessary to put the incoming tasks into a queue. It is because of the following reasons:

• We might not have sufficient resources available right now.
• There is task dependency, and some tasks need to wait for others.
• We need to de-couple the clients from the task execution so that they can hand off work to our system. Our system then queues it for execution.

Let’s design a task scheduling system that should be able to schedule any tasks. Often many tasks are relatively short-lived (from seconds to minutes). For long-running tasks, we might need the ability of periodic checksumming and restoring at the application level to recover from possible failures. We assume that each task’s computational needs can be met by some single server in our fleet. For tasks that need many servers, ...