Metrics

Introduction

We had taken a detailed look at logging previously. Logs capture detailed textual records of events, errors, and transactions over time. Logs are a way for the system to communicate with its user or maintainer about what it is doing. They are invaluable for post-incident analysis, debugging, compliance, and auditing. In distributed systems analysis, root cause identification and remediation are supposed to be as and when the issue is seen. In such situations, where a real-time overview of a system’s health and performance are needed, logs alone won’t help. This is where metrics can help.

Metrics, in contrast to logs, offer a different perspective: They provide real-time, quantitative measurements of critical system parameters such as CPU utilization, memory usage, response times, and error rates. They excel in delivering immediate insights into the current state of a distributed system, enabling the detection of anomalies and performance issues as they occur.

In short, imagine logs as a plane’s black box that records every action and decision made during a flight. Metrics are like its dashboard, providing real-time quantitative measures that matter at the moment. Metrics are a numeric representation of data measured over intervals of time. We saw examples of how metrics can be useful in the Memory Leak lesson of this course. We'll explore more in this lesson.

Metric anatomy

Metrics are meant to measure and convey critical information about the performance, health, and behavior of software systems. To effectively capture and communicate this information, they are structured with specific components designed to offer clarity and context and help answer crucial questions about system behavior. They are meant to address inquiries like: How often did a particular event occur? What were the variations in performance over time? What factors influenced the outcome of a process? To address these questions comprehensively, metrics consist of three primary components:

• Name: Metrics require clear and descriptive identifiers to convey precisely what they measure. The name serves as a human-readable label that provides context, enabling anyone reviewing the metric to understand its purpose intuitively. For instance, “http_requests_total” explicitly indicates that the metric measures the total count of HTTP requests.

• Value: The heart of a metric is its numerical value, which represents the actual measurement or observation. This value quantifies the aspect being tracked, such as the number of requests, response times, or error occurrences. This component is fundamental for quantitative analysis and comparison.

• Optional labels/tags: While the name and value offer a fundamental understanding of the metric, optional labels or tags add depth and context. Labels are key-value pairs that provide additional information about the metric, allowing for finer granularity and differentiation. They enable users to dissect and categorize data points based on various dimensions. For example, labels like “method” (GET, POST), “status_code” (200, 404), or “service” (frontend, backend) provide crucial context for deeper analysis.

By combining these components, metrics become powerful tools for tracking and conveying system behavior. They offer not only a quantitative measure but also the ability to break down and explore data points, making troubleshooting, debugging, and performance optimization more accessible in complex software systems.

Metric types

In the world of software engineering, no single metric can capture all the relevant information or address every monitoring requirement. As a result, a variety of metric types have emerged to cater to specific aspects of system behavior, performance, and health.

The primary motivation behind the proliferation of metric types is twofold:

• Specialization: Each type of metric is tailored to excel in measuring and conveying specific types of data or characteristics. Whether counting events, tracking resource utilization, analyzing data distributions, or summarizing statistics, these metrics offer a specialized lens through which we can understand our systems more deeply.

• Granularity: Different types of metrics provide varying levels of granularity and detail. Some metrics offer a high-level overview of system health, while others dive deep into specific data points, enabling fine-grained analysis and troubleshooting. The choice of metric type often depends on the level of detail required for monitoring and diagnostic purposes.

By having diverse metric types at their disposal, software engineers, operators, and data analysts can effectively tackle a wide range of monitoring challenges. Whether the goal is to optimize system performance, detect anomalies, troubleshoot issues, or align monitoring with business objectives, the availability of different metric types ensures that the right tool can be used for the right job, ultimately contributing to more effective and informed decision-making in the world of software development and operations.

The table below mentions common types of metrics and when to use them. Here, we should note that not all metrics platforms have all the metrics mentioned below, and even if they don’t, they can be presented differently.