Painless Releases

The pain of unique releases

The release process described in the Case Study: Waiting for Godotchapter, rivals that of NASA’s mission control. It starts in the afternoon and runs until the wee hours of the morning. In the early days, more than twenty people had active roles to play during the release. As we might imagine, any process involving that many people requires detailed planning and coordination. Because each release is arduous, they don’t do many a year. Because there are so few releases, each one tends to be unique. That uniqueness requires additional planning with each release, making the release a bit more painful, further discouraging more frequent releases.

The benefit of frequent releases

Releases should be about as big an event as getting a haircut (or compiling a new kernel, for the gray-ponytailed UNIX hackers who don’t require haircuts). The literature on agile methods, lean development, continuous delivery, and incremental funding all make a powerful case for frequent releases in terms of user delight and business value. With respect to production operations, however, there’s an added benefit of frequent releases. It forces you to get really good at doing releases and deployments.

Making releases and change automates

A closed feedback loop is essential to improvement. The faster that feedback loop operates, the more accurate those improvements will be. This demands frequent releases. Frequent releases with incremental functionality also allow your company to outpace its competitors and set the agenda in the marketplace.

As commonly practiced, releases cost too much and introduce too much risk. The kind of manual effort and coordination we described previously is barely sustainable for three or four releases a year. It could never work for 20 a year. One solution, the easy but harmful one, is to slow down the release calendar. Like going to the dentist less frequently because it hurts, this response to the problem can only exacerbate the issue. The right response is to reduce the effort needed, remove people from the process, and make the whole thing more automated and standardized.

Canary deploy

In Continuous Delivery HF10, Jez Humble and Dave Farley describe a number of ways to deliver software continuously and at low risk. The patterns let us enforce quality even as we crank the release frequency up to 11. A Canary Deploy pushes the new code to just one instance, under scrutiny. If it looks good, then the code is cleared for release to the remaining machines. With a Blue/Green Deploy,”machines are divided into two pools. One pool is active in production. The other pool gets the new deployment. That leaves time to test it out before exposing it to customers. Once the new pool looks good, we shift production traffic over to it. (Software-controlled load balancers help here). For really large environments, the traffic might be too heavy for a small pool of machines to handle. In that case, deploying in waves lets us manage how fast you expose customers to the new code.

These patterns all have a couple of things in common. First, they all act as governors (see Governor) to limit the rate of dangerous actions. Second, they all limit the number of customers who might be exposed to a bug, either by restricting the time a bug might be visible or by restricting the number of people who can reach the new code. That helps reduce the impact and cost of anything that slipped past the unit tests.