Rolling Back or Rolling Forward?

Understanding the scenarios

At this point, we are, more or less, capable of deploying new releases to production as soon as they are ready. However, there will be problems. Something unexpected will happen. A bug will sneak in and put our production cluster at risk. What should we do in such a case? The answer to that question largely depends on the size of the changes and the frequency of Deployments.

If we use a continuous Deployment process, we deploy new releases to production fairly often. Instead of waiting until features accumulate, we deploy small chunks. In such cases, fixing a problem might be just as fast as rolling back.

After all, how much time would it take us to fix a problem caused by only a few hours of work (maybe a day) that was discovered minutes after you committed? Probably not much. The problem was introduced by a very recent change that is still in the engineer’s head. Fixing it should not take long, and we should be able to deploy a new release soon.

We might not have frequent releases, or the number of changes included are more than a couple of hundreds of lines of code. In such a case, rolling forward might not be as fast as it should be. Still, rolling back might not even be possible.

We might not be able to revert the Deployment if the database schema change and make it incompatible with the previous versions of the back-end that uses it. The moment the first transaction enters, we might lose the option to roll back, at least not without losing the data generated since the new release.

Note: Rolling back a release that introduced database changes is often not possible. Even when it is, rolling forward is usually a better option when we’re practicing continuous Deployment with high-frequency releases limited to a small scope of changes.

Rolling back should be considered a last resort. Still, in some cases, that is a better option. In others, it might be the only option. Luckily, rolling back is reasonably straightforward with Kubernetes.

Rolling back with Kubernetes

Let’s suppose we just discovered that the latest release of the vfarcic/go-demo-2 image is faulty and that we should roll back to the previous release. Before going to the command, let’s first create the cluster and deploy go-demo-api.