Testing Your Patches

So we’ve found out-of-date software on our network. We know we want to patch it as soon as possible. But how soon is possible?

The answer is going to be a little different for every organization. You’ll have to decide how comfortable you are with a given vendor’s track record of providing stable patches. The answer will also depend on the criticality of the system to be patched, the severity of the vulnerability, and the availability of workarounds for the vulnerability. These variables are outside of your control. The only thing that you can do to speed up the deployment of a patch is to have a set of tests ready to give you a quick yes or no on the question of whether the patch will break things.

Creating comprehensive tests

A set of tests that covers every piece of software in your organization will always be a work in progress. There is so much third-party software, and there are so many demands on our time other than patching. But even partial test coverage is valuable. If dedicated tests for each piece of third-party code aren’t an option right now, you can still get value from integration tests that run against a fully running instance of your program. You may be able to test multiple libraries as well as your own code in a single integration test. Integration tests are coarser-grained than unit tests. This is nice because you can cover more at once. But this is also problematic because when a test breaks you’ll have more places to look to find the culprit.

Manual testing for time crunches

If time is really tight, you may be stuck with manual tests. There’s nothing wrong with manual tests. Just make sure that the tests are documented so that anyone on your team can perform the test. Just like we saw with searching for patches, there’s great value in spreading the knowledge around the team, breaking down knowledge silos, and gaining multiple perspectives into this work. You may find that tests “mature” from manual to automated over time. It’s entirely reasonable to start with manual testing and only automate if a software product needs to be updated often.

If your tests let problems into production, make the tests more detailed. If they break too often, make them less detailed. If you have to perform the tests frequently, automate them.

What is a breaking change?

What makes one version of a piece of software different from the version that came before it? Sometimes it’s things like improved performance or brand-new functionality. These are changes that improve the system but don’t change the way users interact with it. If a user of the software wants to use the new functionality, they can. If they don’t, then they don’t need to change anything: the new version is a drop-in replacement for the old version. Other kinds of changes are things like renamed methods and method signature changes. These kinds of changes are breaking changes. Breaking changes require that the users change the way they use the software or it breaks.

Breaking changes are different than bugs. A bug is a problem where software does something the authors and users of the software didn’t intend or expect. A breaking change is a problem where the authors of the software intended for the software to behave differently than it used to and for users to change the way they use the software in order to adapt to the change.

                                                 Q U I Z