Monitoring Pipeline Performance and Quality

Learn how to monitor pipeline performance and quality in ADF using Azure Monitor.

Monitoring the performance of pipelines is essential to ensure that they are running smoothly and delivering desired results. In this lesson, we’ll discuss how pipeline performance can be monitored using Azure’s Monitor service.

Azure Monitor service

Azure Monitor is a comprehensive monitoring solution provided by Azure, designed to collect and analyze data from various Azure services in real-time. It acts as a centralized monitoring platform and connects to multiple Azure services to gather logs and metrics, enabling users to gain insights into the health and performance of their cloud resources.

At its core, Azure Monitor relies on a distributed architecture that employs a combination of data collectors, log analytics, and alerting mechanisms.

  • Data collectors, like Azure Diagnostic Logs, collect data from Azure services and transmit it to the Azure Monitor service. This data is then stored in a centralized repository for further analysis and visualization.

  • The log analytics feature enables users to query and visualize this data using Kusto Query Language (KQL), empowering them to uncover patterns, detect anomalies, and troubleshoot issues efficiently.

  • Azure Monitor’s alerting capabilities enable users to set up custom alerts based on defined conditions, allowing for proactive monitoring and quick response to critical events.

Resource alert and health monitoring

As seen earlier, Azure monitor can monitor the health of resources across an Azure subscription. Along with pre-built monitoring metrics, it also allows for the definition of custom monitoring metrics. Below are some commonly used Azure monitor metrics:

  • CPU percentage: Monitors the CPU utilization of virtual machines, providing insights into resource consumption and performance bottlenecks.

  • Memory usage: Tracks the memory consumption of virtual machines, ensuring optimal memory allocation and preventing potential memory-related issues.

  • Network in/out: Measures the incoming and outgoing network traffic for virtual machines, helping to identify network performance issues and bandwidth requirements.

Monitoring and alerting generally go hand in hand. If a monitoring service is used regularly, it can be designed to create custom alerts that will help keep the health of the resources in check. Here are some commonly used alerts that can be set up using Azure monitor.

  • High CPU usage: Sets up an alert to trigger when CPU usage exceeds a defined threshold, enabling proactive investigation of resource-intensive processes or applications.

  • Low memory availability: Creates an alert when memory usage drops below a specified level, allowing prompt action to avoid memory-related performance degradation.

  • Network bandwidth exceedance: Sets up an alert to notify when network traffic surpasses a predefined limit, helping to identify potential network congestion or spikes in data transfer.

Regular monitoring metrics provide real-time insights into the health and performance of Azure resources, enabling proactive identification of potential issues. Custom alerts allow users to define specific conditions based on these metrics and receive timely notifications when critical thresholds are breached. This combination empowers users to promptly address emerging problems, optimize resource utilization, and maintain a highly available and performant Azure environment.

Monitoring Azure Data Factory performance and health

By configuring Azure Monitor with ADF, users can track metrics such as pipeline success/failure rates, data movement latency, activity execution times, and resource utilization. These metrics enable users to proactively identify bottlenecks, optimize data integration workflows, and ensure the smooth functioning of data pipelines.

Monitoring using Azure portal UI

Let’s now explore how this can be established in Azure for the data factories we have created so far. First, let’s look at the metrics on the Azure portal.

  1. Navigate to the Azure Data Factory resource page from the Azure portal.

  2. On the left navigation pane, click “Metrics,” as shown in the image below. This will open a blank chart on the right asking for inputs on what needs to be monitored.

  3. Select the ADF instance (here az-adu-adf) as the scope for the chart. In the metric namespace, ensure that the “Data Factory (V2) standard” is selected.

  4. Now, let’s explore our first metric by selecting the “Succeeded pipeline runs” metric and aggregating it to show the count. This will update the chart to show all successful pipeline runs in the ADF instance.

  5. Next, let’s change this metric and select the “Total factory size (GB units)” metric to see the overall size of the ADF instance, including all pipelines and datasets.

The images below illustrate the use of metrics in the Azure portal UI:

Get hands-on with 1300+ tech skills courses.