Testing a Circuit Breaker

Understanding the system

Circuit breakers are one of the most interesting concepts in fault isolation for reliable systems. They are used to account for errors that happen over time in a part of the system. If the frequency at which the errors happen is considered too high compared to the successful cases, the breaker is tripped.

Once tripped, further calls automatically fail before having a chance to reach the subsystem gated by the breaker. The idea is that failures tend to be costly and take a lot of time, and a failing system under heavy stress is even harder to get back into a usable state. The circuit breaker helps us prevent the client from doing work that would result in expensive failures. Instead, we get cheap failures that the failing subsystem recuperate under less stress.

For this example, we’ll send tests to the circuit_breaker library from Klarna. But first, let’s see how it works.

Understanding circuit_breaker

Let’s start by taking a look at how the library works. After that, we can move on to writing tests for it.

The library can be used as a wrapper over a given function call. This wrapper looks at the return values and duration of each function call, and efficiently tracks various failure rates to figure out if the breaker should be tripped or not. Once tripped, the breaker prevents function calls from taking place. It can be untripped after either a cooldown period or a manual intervention.

As a user of the library, we can define a tolerance of how many errors per period of time are required to break the circuit, including acceptable error values and timeout thresholds. If the defined rate is crossed, the breaker trips. We can also trip the breaker manually to prevent any calls from taking place. This lets operators of a system selectively enable or disable features.

The circuit_breaker call

Here’s what a circuit_breaker call looks like: