Physical Fundamentals

Controlling the running system

In the last chapter, the operations team, my client, and I narrowly avoided a financial disaster. It was a difficult situation, and the “solution” was not exactly ideal. All of us would have been happier if it’d never happened. My team couldn’t fix the underlying problem. The delivery scheduling servers were outside our control. But I was able to diagnose the problem, and the operations center partially mitigated its effects. That was only possible because we already had good visibility into the running system. There certainly wasn’t time to add a bunch of logging calls inside the application. With runtime visibility, though, new logging wasn’t necessary. The applications revealed their problems. To apply the solution, we exercised control over the running system. There’s no way we could have recovered if we’d had to reboot the servers after every configuration change.

Design for production

The next few chapters cover those key ingredients, leading us to a concept of design for production. Designing for production means thinking about production issues as first-class concerns. That includes the production network, which might be considerably different from your development environment. It also includes logging and monitoring, runtime control, and security. Design for production also means designing for the people who do operations, whether they are a dedicated ops team or integrated with development.