Provisioning and Deployment Services

Deployment

In the later chapters of this course, we’ll look at how to design services and applications to be deployable. Here, let’s look at the supporting infrastructure to perform the deployments themselves. Deployment may be the most well-trodden area of operations tools. It’s an obvious nexus between development and production. To some organizations, deployment means DevOps. It’s understandable. In many organizations deployment is ridiculously painful, so it’s a good place to start making life better. Consequently, a host of deployment tools represent push and pull methods.

Push-based deployment

A push-style tool uses SSH or another agent so a central server can reach out to run scripts on the target machines. The machines may not know their own roles. The server assigns them.

Pull-based deployment

In contrast, pull-based deployment tools rely more on the machines to know their own roles. Software on the machine reaches out to a configuration service to grab the latest bits for its role.

Pull-based tools work especially well with elastic scaling. Elastically scaled virtual machines or containers have ephemeral identities, so there’s no point in having a push-based tool maintain a mapping from machine identity to role. The machine identity will shortly disappear, never to be seen again! With long-lived virtual machines or even physical hosts, push-based tools can be simpler to set up and administer. That’s because they use commodity software like SSH rather than agents that require their own configuration and authentication techniques.

The deployment tool by itself should be augmented with a package repository. Whether that’s an official artifact repository tool or an S3 bucket is up to you. But it’s important to have a location for blessed binary bits that isn’t populated from a developer’s laptop.

Production builds

Production builds need to be run on a clean build server using libraries with known provenance. The build pipeline should tag the build as it passes various stages, especially verification steps like unit or integration tests.

This isn’t just being pedantic or jumping through hoops to satisfy a security department. Repeatable builds are important so code that works on our machine works in production, too.

Build servers as attack vectors

Any widely used piece of server software will be used for an attack. That includes build servers such as Jenkins, Bamboo, or GoCD. At least one major software vendor was attacked by means of the build environment. The attacker compromised a plugin to the vendor’s continuous integration server. The plugin injected code that targeted a well-known customer of this vendor (relayed in personal communication to the author). This vendor kept its libraries in a controlled artifact repository but had overlooked the plugins to the build system itself. Those were downloaded directly from the Net.

Canary deployments

Canary deployments are an important part of the build tooling. The canary is a small set of instances that get the new build first. For a period of time, the instances running the new build coexist with instances running the old build. (See Chapter Handling Versions). If the canary instances behave oddly, or their metrics go south, then the build is not rolled out to the remaining population. Like every other stage of build and deployment, the purpose of the canary deployment is to reject a bad build before it reaches the users.