Building Systems With an Emergency Stop

Systems are going to run amok. This is a simple truth that we need to come to terms with early in infrastructure tooling development.

When we are a small company, there is usually a very small group of people who understand the systems well and watch over any changes to handle problems. If those people are good, they can quickly respond to a problem. Usually, these people are the developers of the software.

As companies start to grow, jobs begin to become more specialized. The larger the company, the more specialized the jobs. As that happens, the first responders to major issues don't have the access or knowledge to deal with these problems.

This can create a critical gap between recognition of a major problem and stopping the problem from getting worse.

This is where the ability to allow first responders to stop changes comes into play. We call this an emergency-stop ability.

Understanding emergency stops

There are multiple ways to build an emergency-stop system, but the basics are the same. The software will check some data store that contains the name of the workflow we are executing and what the emergency-stop state is.

The most simplistic version of an emergency-stop system has two modes, as follows:

• Go

• Stop

The software that does any type of work would need to reference the system at intervals. If it cannot find itself listed or ...