Pausing After Actions

Why didn’t the experiment fail?

In our previous experiment, we validated the state before and after actions. We checked whether the Pod exists, we terminated the Pod, and then we verified whether the Pod still exists. The experiment should have failed, but it didn’t. The reason why it didn’t fail is that all those probes and actions were executed immediately one after another.

When Chaos Toolkit sent an instruction to Kube API to destroy the Pod, it received an acknowledgment of that action. After that, it immediately validated whether the Pod was still there, and it was. Kubernetes did not have enough time to remove it entirely. Maybe the Pod was still running at that time, and perhaps, we were too fast. Or else, maybe the Pod was terminating. That would probably explain the strange outcome. The Pod was not gone right away. It was still there, with the state terminating. So, the thing that we are probably missing to make that experiment more useful is a pause.

Let’s see how we can pause after or before actions to give enough time for the system to perform whichever tasks it needs to perform before we validate the state again.

Inspecting the definition of terminate-pod-pause.yaml

We’re going to take a look at yet another YAML.