How to manage and harden VM images in Azure (6 easy ways)

Assigning and Managing CPU and Memory resources in the Kubernetes can be tricky and easy at the same time. Having done this task for numerous customers, I have decided to create a framework zone. I will show you what Kubernetes’ resources and Kubernetes’ limits are and how to manage them.

Managing VM Images may be a nightmare. Here are 6 simple approaches on how to seamlessly build, share, test, and copy images in Azure. As a bonus, you will also get how to build image notifications based on Event-Driven-Architecture.

Problem

If you are managing 100 Virtual Machines or 1000+ that build and harden VM images, do you manually manage your images? If you do, you know that it is quite expensive and leads to hard-to-detect errors and potential security vulnerabilities.

Here is how I approach this issue for my customers. My procedure of creating image management for Azure is as follows:

1. Building an image using the Azure Image Builder

The Azure Image Builder is a service that allows you to create custom images with Azure CLI. The image creation is based on the JSON template. Take a look at the example below:

{
    "type": "Microsoft.VirtualMachineImages/imageTemplates",
    "apiVersion": "2019-05-01-preview",
    "location": "<region>",
    "dependsOn": [],
    "tags": {
        "imagebuilderTemplate": "ubuntu1804",
        "userIdentity": "enabled"
            },
        "identity": {
            "type": "UserAssigned",
                    "userAssignedIdentities": {
                    "<imgBuilderId>": {}
                        
                }
                },
    "properties": {

        "buildTimeoutInMinutes" : 80,
        
        "vmProfile": 
            {
            "vmSize": "Standard_D1_v2",
            "osDiskSizeGB": 30
            },

        "source": {
            "type": "PlatformImage",
                "publisher": "Canonical",
                "offer": "UbuntuServer",
                "sku": "18.04-LTS",
                "version": "latest"
            
        },
        "customize": [
            {
                "type": "Shell",
                "name": "RunScriptFromSource",
                "scriptUri": "https://raw.githubusercontent.com/danielsollondon/azvmimagebuilder/master/quickquickstarts/customizeScript.sh"
            }          

        ],
        "distribute": 
            [

            ]
        }
    }

I have to cut the original template as it is quite long. You can find the full example here.

The ARM template is quite simple and contains the following properties:

• The identity section is required, you have to create Managed Identity for image builder to have access to create and edit images
• VmProfile is used to set up a VM configuration plan
• Source allows you to specify base image parameters. I use the latest Ubuntu Server 18.04 LTS from Canonical
• The customization section allows you to specify VM hardening scripts

You can see the full list of Image Builder options here

Advantages

Templates and the process itself is easy to understand and can be easily integrated with Azure DevOps.

Disadvantages

The image builder is still in review and is, therefore, not recommended for use in production. Set up can be a bit difficult in comparison to Hashicorp Packer (this is explained in the next section).

2. Building an image using the Hashicorp Packer

Hashicorp Packer is a multi-platform solution that allows users to build custom images based on JSON templates that are well-structured and based on an easy-to-understand object model. The JSON templates have three root objects: Communicators, Builders, Provisioners, and Post-Processors.

You can find the JSON template here .

The JSON template creates the VHD image of Ubuntu with a preinstalled NGINX web server and other updates. You can find a lot of other templates here as well.

To set up packer, you can use Chocolate package manager:

choco install packer -y

You can run the JSON template using the following command:

packer build <path/your/template.json>

Before you run the template, you need to create Management Identity or Service Principal with proper permissions. For example:

az ad sp create-for-rbac -n "ImageContributor" 
 --role contributor `    
 --scopes /subscriptions/<subscription_id> `     
 --sdk-auth > az-principal.auth

This command automatically creates a JSON file with clientId, clientSecret, tenantId, subscriptionId, and other fields.

Advantages

The Packer and JSON templates are simple to understand and allow you to quickly set up the environment and start building images. They also provide a strong community; plus, the Packer supports multiple cloud providers.

Disadvantages

I have not found any serious disadvantages. However, while building an image, the packer always removes the disk that is required in some operations with images (ex: when copying an image).

3. Sharing images

To share images in Azure, you may use the Shared Image Gallery. It can: - create image definition - keep versions of an image - share an image

For example, you can share an image across your Azure subscriptions, resource groups, and tenants.

In the current scenario, you can create a user group, a single user/service principal, or assign contributor rights for this shared image gallery only.

az ad sp create-for-rbac -n "ImageContributor" 
 --role contributor `    
 --scopes /subscriptions/<subscr-id>/resourceGroups/sig-we-rg/providers/Microsoft.Compute/galleries/testsig

As a result, users from the group can create the Virtual Machine based on the Ubuntu image from this Shared Image Gallery in different subscriptions.

az vm create `
  --resource-group myResourceGroup `
  --name UbuntuVM`
  --image "/subscriptions/<subscr-id>/resourceGroups/sig-we-rg/providers/Microsoft.Compute/galleries/testsig/images/ubuntu-server-image-def/versions/1.0.0" \
  --admin-username azureuser \
  --generate-ssh-keys

Advantages

The Azure Shared Image Gallery (SIG) allows you to easiyly build, share, manage, and customize images within your organization. SIG also has Azure CLI, so you can easily automate image distribution.

Disadvantages

The image remains in a shared access gallery; thus, it physically stays there. So, when you share it across subscriptions, you cannot independently change or remove it for each subscription.

4. Copying images

By copying images, you can deliver images from one subscription to another or from one resource group to another. All copied images are independent of each other. You can copy images using Azure Image Copy extension or implement manual copying using Go. Let’s have a look at the examples below.

Copying image with Az Copy Extension

To copy images between subscriptions, I use Az Image Copy extension. It creates a new image (from the source image) in Resource Group A.

Install copy extension:

az extension add --name image-copy-extension

Copy command:

az image copy --source-resource-group Azure-Resource-Group-B `
              --source-object-name image-version-1.0 `
              --target-resource-group Azure-Resource-Group-A `
              --target-location westeurope `
              --target-subscription 111111-2222-2222-0000-0000000 ` # Subscription A
              --cleanup

Important. Packer removes the disk automatically after the managed image is created.

Advantages

The Az Image Copy extension is simple to use and automate.

Disadvantages

An image must contain a managed disc; otherwise, the copy process will fail with a ‘Resource not found’ error message.

Copy image manually

If Az Copy Image Extension does not work for you, you can create a VHD image in the storage account and copy it to another destination storage account.

The process workflow:

1. Create 2 Storage accounts: Source and Destination.
2. Create the VHD image in the Source storage account. You can use the Packer script from the first section.
3. Generate Shared Access Signature for the VHD Source image. You can find an example of how to do it with Azure CLI here.
4. Copy the image. To demonstrate, I used the following Go script. You can also use the AzCopy tool, here.

You can find the source code here.

Advantages

The current image coping workflow is fully custom; so, it can be changed at any time. It can also be useful when the Az Image Copy extension does not work for you. For example, when it removes a managed disk while creating an image.

Disadvantages

You have to implement all workflow steps, including:

- creating VHD
- generating and managing SAS token
- copying an image
- cleaning up
- converting an image

5. Image and Disk Converting

The operations to convert a VHD image to a Managed Disk, or a Managed Disk to a VHD image. This is useful when you need to distribute images across your organization with Azure Shared Image Gallery in the image copy process and when you need to spin up a new VM.

Also, it is useful when you have some legacy VHD and need to install updates, set up automatic backups, use availability sets, and availability zones.

Here is a list of advantages of the Managed Image.

Converting VHD image to the Managed Disk

# Provide the subscription Id where Managed Disks will be created
$subscriptionId = '00000-00000-0000-0000-0000000'

# Provide the name of your resource group where Managed Disks will be created. 
$resourceGroupName ='HBI'

# Provide the name of the Managed Disk
$diskName = 'image-test-disk'

# The Disk It should be greater than the VHD file size.
$diskSize = '130'

# Storage LRS. Options: Premium_LRS, Standard_LRS, etc
$storageType = 'Premium_LRS'

# Set Azure region (e.g. westus, westeurope), Ensure that location of the VHD image (alongside with Storage Account), 
# and future Managed Disk is the same.
$location = 'westeurope'

# Set URI of the VHD file (page blob) in a storage account.
$sourceVHDURI = 'https://imagesstorage.blob.core.windows.net/test-image-disk.vhd'

# Set the Resource Id of the Source storage account where VHD file is stored. You can avoid it if VHD in the same subscription
$storageAccountId = '/subscriptions/subscription-id/resourceGroups/test-rg/providers/Microsoft.Storage/storageAccounts/imagesstorage'

# Set the context to the subscription Id where Managed Disk will be created
Select-AzSubscription -SubscriptionId $SubscriptionId

$diskConfig = New-AzDiskConfig -AccountType $storageType -Location $location -CreateOption Import -StorageAccountId $storageAccountId -SourceUri $sourceVHDURI

New-AzDisk -Disk $diskConfig -ResourceGroupName $resourceGroupName -DiskName $diskName

Convert Managed Disk to Managed Image

$rgName = "HBI"
$location = "westeurope"
$imageName = "boriszn-test-imagefromdisk"


# Get disk 
$disk = Get-AzDisk -ResourceGroupName 'HBI' -DiskName 'image-test-disk' 
$diskId = $disk.Id

# Create Managed Image Config with Managed Disk Info 
# OS Types (Linux, Windows)
$imageConfig = New-AzImageConfig -Location $location
$imageConfig = Set-AzImageOsDisk -Image $imageConfig -OsState Generalized -OsType Linux -ManagedDiskId $diskId

# Create the image.
$image = New-AzImage -ImageName $imageName -ResourceGroupName $rgName -Image $imageConfig

Advantages

With both cmdlets, you can deliver an image to your Azure Shared Image across different subscriptions and resource groups.

Disadvantages

The conversion process can be complicated because some images may be outdated or the conversion process may fail.

6. Testing images

To test images, you can simply spin up the new VM from the image gallery and use an image definition:

az vm create `
   --resource-group container-image-rg `
   --name vm-test `
   --image "/subscriptions/<subscription-id>/resourceGroups/container-image-rg/providers/Microsoft.Compute/galleries/test-gallery/images/image-test-def" `
   --generate-ssh-keys 

Advantages

The command and process itself are quite simple.

Disadvantages

It does not check if specific software and services are installed correctly in the JSON configuration. Therefore, this logic has to be implemented separately.

Conclusion

That’s it. Based on these ways you can easily set up images Hardening and management pipeline in the Azure DevOps.