Production Environment Problems

What is chaos engineering?

Imagine a conversation that starts like this:

“Hey boss, I’m going to log into production and shut down a few instances of the system. Just a few here and there. Shouldn’t hurt anything,” you say.

How do you think the rest of that conversation will go? Normally, not well! Killing instances turns out to be a radical idea, but not a crazy one. It’s one technique in an emerging discipline called chaos engineering. $^{1}$, chaos engineering is “the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” That means it’s empirical rather than formal. We don’t use models to understand what the system should do. We run experiments to learn what it does.