If Patching Hurts, Do It More Often

Why patch frequently

Martin Fowler has written about the saying, “If it hurts, do it more often” as it pertains to activities like deployments and integration. This idea fits wonderfully into a discussion on patching. Fowler gives three main reasons why doing painful things more often makes them less painful over time:

1. It breaks work into small, manageable pieces.
2. It adds opportunities for feedback.
3. It provides practice and the potential for automation.

Let’s see how breaking the patching process up into smaller pieces can help us. There’s generally a long period of time between patches that fix truly critical vulnerabilities. If we wait and only apply these critical patches, the upgrade will be a bigger job and there will be a higher chance that we’ll have to deal with a breaking change. However, if we break up this work over time and apply a lot of small patches all along the way, we’ll be fairly current when that critical patch inevitably rolls along. This makes it less likely that we’ll have to deal with a breaking change during the tight time constraints around deploying a critical patch. Also, when a less critical patch comes along, we can be more flexible and give ourselves more time; we don’t have to drop everything and deploy right away. We can use this extra time to build out reliable automation and work through any bugs or breaking changes.

Feedback is important because this is how we learn what we need to improve. Unless we patch often, we won’t know where to focus our efforts. Maybe we need to focus on testing, maybe we need to focus on minimizing downtime, maybe on eliminating race conditions during deployment. If we don’t deploy often, we’re just guessing.

Finally, practice and automation are the keys to reliable, uneventful patching. Even if we don’t have an automated deployment yet, doing frequent manual deployments should motivate us to spend the time to automate deployments. We would want to spend time on automation of any frequently occurring, repetitive task anyway. Improving our security posture along the way is a bonus. Hopefully, we’ve been using the many lower-impact patches as an opportunity to streamline our deployment process. Most of the time, improving the ability to patch reliably is more important than the patch itself.

A practical application of fear

When you approach the issue of patching production systems from the security point of view, you’ll want to patch as many things as possible as often as possible.

Your teammates who are in an operations role such as sysadmins or DevOps will want to make as few changes as possible to a working system. “If it ain’t broke, don’t patch it,” might be their slogan. Don’t let these differing viewpoints turn into conflict. The ops folks have a fear of breaking things. Fear isn’t always bad; fear can keep us out of danger. So when you encounter resistance from ops about patching, don’t try to out-shout them.

First aim to understand and resolve the fear. Their fear might be telling you something. Maybe the fear is telling you that you need an automated test suite because you need to roll out patches frequently. Maybe the fear is telling you that you need to build out a performance testing environment, or that you have unresolved technical debt that makes deployments difficult and time-consuming, or that your vendor’s patches aren’t reliable and you should move to a new vendor. This fear is a valuable resource. Learn from it.

What’s next?

We’ve learned how to take inventory of our third-party code, and how to search it for published vulnerabilities. But what makes software vulnerable? What kinds of software vulnerabilities are there? How do we find them in our own code, and how do we prevent introducing them in the first place? We’ll cover all that in the next chapter.