Delve into Kubernetes Chaos Engineering to discover system limits, simulate adverse effects, and improve resiliency. Learn about fault tolerance, high availability, and running experiments in a Kubernetes cluster.

Let’s face it, bad things happen in software engineering, often when you least expect it. Servers go down, there are spikes in traffic, you receive networking errors. Luckily, you can avoid these problems. Enter chaos engineering.
Chaos engineering is where you try to find the limits of a system and simulate adverse effects that might happen. These simulations then help you to determine the best way to improve your system and build up its resiliency.
This course is divided into various sections to get you used to the different types of experiments you can run in chaos engineering. You will start by learning how to destroy application instances in order to show you the importance of scaling, fault tolerance, and high availability.
You will then dive deep into other types of experiments such as destroying a network, draining and deleting nodes, and testing availability. Towards the end of the course, you will learn how to run experiments in a Kubernetes cluster.

The DevOps Toolkit: Kubernetes Chaos Engineering

# Assignment

As I mentioned before, nothing happens if we destroy only one out of three nodes (again, ignoring the DB). But, if we continue destroying nodes, bad things could happen.

Here comes yet another assignment: Figure out what would happen if we destroy a whole data center.


# Critical issues with our clusters

There are at least two critical things that we didn’t do in the previous section, but we should.

## Not using Cluster Autoscaler

First of all, we should have created our Kubernetes cluster with Cluster Autoscaler so that it automatically scales up and down depending on the traffic. Not  only would our cluster scale up and down to accommodate an increase and decrease in the workload, but when a node goes down, it would be recreated by Cluster Autoscaler. The cluster would also figure out that there is not enough capacity. Cluster Autoscaler itself would solve fully (or partly) the problems that we could have encountered if we continued running the previous experiment and continued deleting nodes.

## Running a zonal cluster

The second issue is that we are running a zonal cluster. If you followed my Gists, your cluster is running in a single zone, which means that it is not fault-tolerant. If that zone (data center) goes down, we’d be lost. So, the second change we should have done to our cluster is to make it regional. It should span multiple zones within the same region. It shouldn’t run in different regions because that would increase latency unnecessarily. Every cloud provider, at least the big three, has a concept of a region, even though it might be named differently. By region, I mean a group of zones (data centers) that are close enough to each other so that there is no high latency, while they still operate as entirely separate entities. Failure of one should not affect the other. At least, that’s how it should be in theory.

# Making a regional cluster

Therefore, we should make our cluster regional, and we should make it scalable. Since the steps differ from one provider to another, I will not show you how to create such a cluster, but I will provide the Gists. Please consult them to see how to create a better Kubernetes cluster in Google, Azure, or AWS.

* [Regional and scalable GKE](https://gist.github.com/88e810413e2519932b61d81217072daf)
* [Regional and scalable EKS](https://gist.github.com/d73fb6f4ff490f7e56963ca543481c09)
* [Regional and scalable AKS](https://gist.github.com/b068c3eadbc4140aed14b49141790940)

We'll start the next section by creating the cluster in that way. It will scale automatically, and it will be regional.

That's about it. There are many other things we could do with nodes, but we explored just enough for you to have at least the basic understanding of how chaos engineering applied to Kubernetes nodes works.

---

In the next lesson, we will remove the resources that we have created.


This lesson contains your next self-assignment and discusses some critical things that we have not done yet.

Introduction To Kubernetes Chaos Engineering

Defining Requirements

Destroying Application Instances

Experimenting with Application Availability

Obstructing and Destroying Network

Draining and Deleting Nodes

Creating Chaos Experiment Reports

Running Chaos Experiments Inside a Kubernetes Cluster

Executing Random Chaos

What’s Next?

Destroying Cluster Zones

Assignment

Critical issues with our clusters

Not using Cluster Autoscaler