Need to build event-driven architecture in the Azure? Easy.

Building and supporting infrastructure in a large organization can be challenging due to the automation of all deployment steps (e.g., creating cloud environments for a department with a set of virtual machines that contain preinstalled software, utilities, and services that may have their own dependencies like database and storage, and connection to log analytics and other log services). Utilities and services may have external dependencies as well; therefore, you need to have a mechanism to check and ensure that required connections are established, dependencies are installed, and proper automation runbook is triggered. The mechanism should alert the administrator of error and perform a backup.

Building the mechanism

To build this mechanism, I will use the following Azure resources:

Event Grid

Event Grid is a key component that acts as the main event processor and contains:

• Topics are Azure resources that represent components that generate events.
• Subscriptions/endpoint is an Azure resource that handles the event.

The relation between publisher components, events, topic, and subscriber/endpoint is shown in the diagram below.

Event grid contains:

• Dead Letter Queue and retry policy — if the message is not able to reach the Endpoint, then you should also configure the retry policy
• Event filtering — the rule allows the event grid to deliver specific event types to the endpoint point. For example, when you create a new VM in the topic container (resource group or subscription), the Event Grid will catch the event and deliver it to the endpoint.

Queue Storage account

Queue Storage uses as the main event storage. When the event is generated via Event Grid, the final destination will be a queue.

Azure Function

Functions are used as microservices that contain logic to validate required resources and connections. Each function may also contain a database for storing configuration or state management data.

Now, after a rather high-level architecture description, I will provide more details below.

Architecture

As you have noticed, an event grid is linked with a subscription listening on events related to new virtual machine creation. It is necessary to add filtering here; otherwise, the event grid will generate messages whenever any resource is created in a subscription or the target resource group. All events delivered are delivered in the storage queue. In my project, I’m using three queues:

• The main queue is a destination for all messages from the event grid.
• Retry queue receives all messages that failed during the first steps of validation and were scheduled for future retry.
• Succeeded queue is used for all successfully processed messages. In my project, I used this queue for future statistics and reporting.

A storage account is also linked to Azure Log Analytics to synchronize all logs and alerts. For example, log analytics will log this as an error and alert administrator if there are more messages in the retry queue than expected.

The Azure function app contains several Azure functions with validation, message processing, and logic to trigger the runbook.

The validation function is linked with the main event queue. When the event grid sends a message to the main queue, the function is automatically triggered.

Retry Function is based on a timer trigger and will be run constantly to check for failed messages intended to retry (in retry event queue).

API Function(HTTP trigger) is intended to trigger (re-run) the whole process from runbook, Admin UI, etc.

The whole functions workflow is described below:

Architecture code base

The function app is written on PowerShell (Powershell Class) and uses the principles of OOP. This approach allows us to build modular, well-supported code and add or remove functions at any time.

Why did I opt for PowerShell and not C# or JavaScript? The main reason is that a solution based on PS can be supported by, not only by developers, but by cloud administrators and system engineers as well.

Conclusion

In this article, I described how to build event-driven architecture to manage the virtual machine, related utilities, and components.

You can reuse the presented solution in the following scenarios:

• Key Vaults and SSL certificate management (check certificate expiration time, log and inform, update certificate automatically)
• Create custom logic to build cloud expense reports
• Cloud resources backup, check availability and log (using Log Analytics or other tools)
• Resource clean up management
• Container management solution

You can find the source code here and the complete tutorial is available here.

Bonus

This topic is quite useful and popular in the world of cloud and software architecture; moreover, these approaches solve a lot of business problems. With this in mind, I’ve decided to create a practical and interactive course. It is based on my experience building EDA architecture for enterprise customers.

You can play with the code, test architecture, and run pipelines directly in the course! Click the link below to enroll in the course:

Building Event-Driven and Microservices Architecture in Azure