Smart Metrics Strategy

Establishing counter metrics

Metrics-based product development is one of the best things to happen in the software age. Metrics let us define what success looks like and then track our progress until we’ve definitively reached our goals.

Metrics for product development come in a lot of different shapes and sizes, and each one has a different purpose. It is natural to pay the greatest amount of attention to the metrics that show how successful a product is. We use these success metrics or goal metrics to decide which features and improvements to add to our products first. They are quantitative measures of what we think success looks like, based on data. This makes it easy to measure success. They are the main way we know we are going in the right direction, and they are essential in measuring our operational efficiency.

For a given product, we might have a lot of metrics that show how well it’s doing, or we might decide to focus on a single North Star metric for the most clarity and focus all of our efforts on that.

But overemphasizing one or two success metrics can have unintended effects on a business. This is where counter metrics step in, to ensure that we are not blindsided by a focus on a handful of success metrics.

Counter metrics, also called guardrail metrics or health metrics, are measurements of parts of a product or business that may change if we focus on our success metrics. They are an important part of defining our success metrics because they help us see what’s going on and stop things from going horribly wrong.

So, for every success metric, we should have one (or more) counter metrics that serve as a constant reality check. Luckily for us, figuring out our counter metrics is as easy as thinking creatively about what might happen if we focus on our success metrics.

In a developer portal experience, we can create a success metric for the number of accounts created. This metric, although important to track, can be gamed if we start incentivizing users to sign up with different email addresses, in which case we are not truly acquiring new users. This metric can also be a vanity metric if users are signing up and creating an account but are not able to make their first API call. This is why it’s important to look at the number of accounts created in the context of the number of users who make their first call. We can look at this data in the form of cohort analysis, or we can also track the rate of users reaching the first call by channel to understand the quality of users we’re getting on the platform.

Still, it’s important to know that counter metrics need to be weighed against our business’s top priorities. It might be smart to start by focusing on a small group of our possible users before moving on to the rest. So, when we’re choosing which counter metrics to use, we should think about any possible side effects and ask ourselves how they could affect our goals.

Here are some examples of metrics and counter metrics for APIs.