Deployment strategy: blue/green deployment vs. canary release
Just-in-time feature releases and bug fixes are essential for maintaining an edge over competitors. Previously, the process of rolling out new features was a slow process—grindingly slow by today’s standards. Bug patches, although more urgent, also faced delays due to infrastructure constraints. However, the turn of the millennium saw the advent of cloud computing. Infrastructure management now became the responsibility of the service provider. Businesses could now focus only on their product while managing the required resources and scaling as per business requirements could be outsourced. This led to various development paradigms appearing on the tech horizon.
In this blog, we will discuss two strategies for quickly deploying software updates, namely blue/green deployment and canary release. We will also look at their pros and cons.
Let’s get started.
Continuous integration/continuous delivery#
To make a case for the two deployment strategies, let’s first understand continuous integration/continuous delivery (CI/CD). CI/CD is a software development paradigm consisting of automated integration, testing, and deployment of code changes. Unlike the older techniques that were constrained by scheduled launch dates or relied on integrating multiple updates into a single batch before version launch, this method aims to be more frequent. The CI (continuous integration) in this strategy focuses on the regular integration of code changes from multiple developers into a shared repository. The CD (continuous delivery) part is responsible for automatically deploying successfully tested code changes to production environments.
Another term that we frequently encounter together with CI/CD is DevOps, which is a broader organizational culture developed to improve the software development and delivery process. It is an evolution of Agile development practices. CI/CD is one strategy of DevOps—focusing on tools that automate integration, testing, and deployment in a manner that is in line with the DevOps philosophy.
The shift to cloud computing in recent years and the advent of container-based computing using software such as Kubernetes and Docker have been great enablers of CI/CD. People can now deploy their software in a virtual environment and migrate the entire environment as needed.
To get the maximum benefit from this approach, it is necessary to have a strategy in place to eliminate deployment downtime and enable quick rollbacks, thereby ensuring a smooth running of the CI/CD pipeline. Several approaches have been proposed in this regard. Blue/green deployment is one of the popular ones that we will see next.
To dive deep into DevOps and CI/CD concepts, explore our courses: CI/CD Using Native Tools Available on AWS, The DevOps Toolkit Catalogue, and The DevOps Toolkit: Working with Jenkins X.
Blue/green deployments: Instant replacement of services#
In a blue/green deployment, we create two separate hosting environments that are identical to each other. The environment running the current, fully tested version of the application is labeled blue. The new version of the application with the latest and as-yet-not-fully-tested updates, runs in the green environment. Note that the environment can be a server serving user requests or a group of servers running various microservices to serve the end user.
The way this strategy works is that at any given time, only one environment can handle requests, i.e., only the blue environment is live. Meanwhile, the development team works on the new release of the application in the green environment. Once the development work is done and initial testing has been completed, the incoming traffic is now routed to the green environment.
There are multiple ways in which this routing is carried out. For instance, the DNS server can be updated, the resource discovery service can point to the new server, or the user-facing server can route the incoming requests to the new environment. Also, it is not necessary to completely separate everything into a blue and green environment. It is possible, for example, to have a single database that is used by both environments. In that case, only the application software would switch and not the database.
Once the green environment is live and seems stable, the roles of each environment switch. The blue environment becomes our staging environment for the next version while the green environment becomes the live app. In this way, we alternate between the blue and green setups.
Once the green environment goes live, there is a cutoff time before which we do not modify the previous version. This is done so that if we’re not satisfied with the new version, some bugs are detected, or if we receive negative customer feedback, we can simply roll back to the previous version. However, suppose we roll back and still, there is a period of time in which everyone is accessing the bad version.
This begs a very important question: Is there a more efficient way of deployment so that the impact of a bad release is contained? Yes, there is. We will look at canary release next.
Backstories behind names usually give insights into design choices. So why was this approach called blue/green? The creators, Dan North and Jez Humble, wanted to avoid names that had an obvious ordering or negative connotations. For instance, naming it A/B could be confusing since it might imply that A is better than B. Similarly, using “red” in the name was avoided since red is associated with danger. Interestingly, blue/green deployment is sometimes referred to as red/black deployment because Netflix developers use these colors to match their brand colors.
Canary release—phased rollout#
As early as 1896, coal miners would bring canaries into coal mines as an early warning signal for the accumulation of gasses, such as carbon monoxide, that are lethal for humans. The birds, being more sensitive, would become sick before the miners, alerting them to toxic gasses. In some, more humane, cases, the canary cages had dedicated oxygen tanks so the birds could be saved after the alert. In computer science, canaries refer to a method for early detection of a problem. We encounter the concept in diverse fields ranging from software versioning and deployment to cybersecurity and operating systems.
A canary release extends the idea behind blue/green deployment. The core idea is the same—we separate the stable version from the under-development version and route the traffic to the appropriate environment. It differs in the scale of deployment. In blue/green deployment, the switch to the latest version is instantaneous. Assuming the time required for propagating the changes is negligible, it is expected that everyone would see the latest version.
A canary release takes a more conservative and cautious approach: Incremental rollout (also known as phased rollout). The latest version is visible only to a small percentage of the users. During that time, both the stable and the potentially buggy latest versions exist in parallel. A large percentage of the users continue to see the former, while a small percentage are routed to the experimental version. This allows for thorough testing without the threat of bugs bringing down the entire system for all users. Once the development team is satisfied, the remaining users’ traffic is also routed to the latest version.
A question arises: How do we decide who gets to see the latest version? There are multiple strategies for doing this. The sample can be selected randomly. The traffic can be routed to the relevant version based on the geographical location of the source of the request. A more complex approach would be to decide based on demographics. This can be achieved via browser cookies or, for more accuracy, using signups. A safer bet is to create a private network of company employees and route only their requests to the latest version.
It is important to note that deployment is different from testing, in that it focuses on how to roll out features while minimizing disruption in case of any bugs in the application or dissatisfied users. Deployment is followed by monitoring to detect issues and rolling back if needed. Testing, on the other hand, focuses on detecting bugs and user issues before deployment. So for instance, beta testing involves external users testing a prerelease version of the entire software product to provide feedback and identify issues before the official launch, while a canary release gradually rolls out specific changes or features to a small subset of users to validate their stability and impact in a real-world environment, typically before wider deployment.
Pros, cons, and a comparison#
The blue/green deployment design pattern offers many benefits. It minimizes downtime since the switch to the new version is instantaneous (assuming the propagation time is negligible). The risk of a catastrophic event is reduced since there is a cutoff time after the launch of a new version; in case a problem is detected, we can easily roll back to the stable version, provided we take action before the cutoff time. However, it also comes with challenges related to resource duplication, complexity, and potential operational overhead. Maintaining two different environments at any given time would be expensive. Moreover, keeping the two versions separate and ensuring we are able to roll back or shift to a different version would involve some complexity. Data migration between the two environments can also be challenging. Versioning also needs to be planned, otherwise, we can end up with discrepancies.
Similar to blue/green deployment, the canary release model also offers benefits such as risk mitigation since we can easily roll back to the stable version. The risk is much less in this case since only a small population is exposed to the latest version. The larger group can access it only once the previous stage is successful. Sharing with a smaller group also allows for faster feedback and better performance monitoring. By slowly ramping up the load, we are also able to do capacity testing. This ability to enable real-world testing is what makes a canary release so useful. However, it also introduces complexities and overheads of managing multiple versions and duplicate resource allocation. On top of that, the potential to confuse users is present since different users can see different versions of the same application at the same time. This can also lead to compatibility issues and dependency challenges if the version launch is not planned properly.
This blog has only scratched the surface of a type of deployment strategy. For a detailed discussion on system design, have a look at our Grokking Modern System Design Interview for Engineers & Managers and Grokking the Principles and Practices of Advanced System Design courses.
Free Resources
