Pods and Scheduling

Kubernetes guarantees to schedule all containers in the same Pod to the same cluster node. Despite this, you should only put containers in the same Pod if they need to share resources such as memory, volumes, and networking. If your only requirement is to schedule two workloads to the same node, you should put them in their own Pods and use one of the following options to schedule them together.

Terminology: Before going any further, remember that nodes are host servers that can be physical servers, virtual machines, or cloud instances. Pods wrap containers and execute on nodes.

Pods provide a lot of advanced scheduling features, including all of the following:

• nodeSelectors

• Affinity and anti-affinity

• Topology spread constraints

• Resource requests and resource limits

nodeSelectors are the simplest way of running Pods on specific nodes. You give the nodeSelector a list of labels, and the scheduler will only assign the Pod to a node with all the labels.

Affinity and anti-affinity rules are like a more powerful nodeSelector.

As the names suggest, they support affinity and anti-affinity rules. But they also support hard and soft rules, and they can select on nodes as well as Pods:

• Affinity rules attract

• Anti-affinity rules repel

• Hard rules must be obeyed

• Soft rules are only suggestions

Selecting on nodes is common and works like a nodeSelector where you supply a list of labels, and the scheduler will assign the Pod to nodes possessing the labels.

To select on Pods, the scheduler takes a similar list of labels and schedules the Pod to nodes running other Pods possessing the labels.

Consider a couple of examples.

A hard node affinity rule specifying the project=qsk label tells the scheduler it can only run the Pod on nodes with the project=qsk label. It won’t schedule the Pod if it can’t find a node with that label. If it was a soft rule, the scheduler would try to find a node with the label, but if it can’t find one, it’ll still schedule it. If it was an anti-affinity rule, the scheduler would look for nodes that don’t have the label. The logic works the same for Pod-based rules.

Topology spread constraints are a flexible way of intelligently spreading Pods across your infrastructure for availability, performance, locality, or any other requirements. A typical example is spreading Pods across your cloud or datacenter’s underlying availability zones for high availability (HA). However, you can create custom domains for almost anything, such as scheduling Pods closer to data sources, closer to clients for improved network latency, and many more reasons.

Resource requests and resource limits are very important, and every Pod should use them. They tell the scheduler how much CPU and memory a Pod needs, and the scheduler uses them to select nodes with enough resources. If you don’t specify them, the scheduler cannot know what resources a Pod requires and may schedule it to a node with insufficient resources.

Deploying Pods

Deploying a Pod includes the following steps:

1. Define the Pod in a YAML manifest file

2. Post the manifest to the API server

3. The request is authenticated and authorized

4. The Pod spec is validated

5. The scheduler filters nodes based on nodeSelectors, affinity and anti-affinity rules, topology spread constraints, resource requirements and limits, and more

6. The Pod is assigned to a healthy node meeting all requirements

7. The kubelet on the node watches the API server and notices the Pod assignment

8. The kubelet downloads the Pod spec and asks the local runtime to start it

9. The kubelet monitors the Pod status and reports status changes to the API server

If the scheduler can’t find a suitable node, it marks it as pending.

Deploying a Pod is an atomic operation. This means a Pod only starts servicing requests when all its containers are up and running.