As you already saw, Prometheus Adapter comes with a set of default rules that provide many metrics that we do not need, and not all those that we do. It’s wasting CPU and memory by doing too much, but not enough. We’ll explore how we can customize the adapter with our own rules. Our next goal is to make the adapter retrieve only the nginx_ingress_controller_requests metric since that’s the only one we need. On top of that, it should provide that metric in two forms. First, it should retrieve the rate, grouped by the resource. The second form should be the same as the first but divided with the number of replicas of the Deployment that hosts the Pods where Ingress forwards the resources. That one should give us an average number of requests per replica and will be a good candidate for our first HPA definition based on custom metrics.

Make adapter retrieve nginx_ingress_controller_requests metric #

I already prepared a file with Chart values that might accomplish our current objectives, so let’s take a look at it.

cat mon/prom-adapter-values-ing.yml

The output is as follows.

image:
  tag: v0.5.0
metricsRelistInterval: 90s
prometheus:
  url: http://prometheus-server.metrics.svc
  port: 80
rules:
  default: false
  custom:
  - seriesQuery: 'nginx_ingress_controller_requests'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        ingress: {resource: "ingress"}
    name:
      as: "http_req_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)'
  - seriesQuery: 'nginx_ingress_controller_requests'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        ingress: {resource: "ingress"}
    name:
      as: "http_req_per_second_per_replica"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / sum(label_join(kube_deployment_status_replicas, "ingress", ",", "deployment")) by (<<.GroupBy>>)'

The first few entries in that definition are the same values as the ones we used previously through --set arguments. We’ll skip those, and jump into the rules section.

Within the rules section, we’re setting the default entry to false. That will get rid of the default rules we explored previously and allows us to start with a clean slate. Further on, there are two custom rules.

First custom rule #

The first rule is based on the seriesQuery with nginx_ingress_controller_requests as the value. The overrides entry inside the resources section helps the adapter find out which Kubernetes resources are associated with the metric. We’re setting the value of the namespace label to the namespace resource. There’s a similar entry for ingress. In other words, we’re associating Prometheus labels with Kubernetes resources namespace and ingress.

As you will see soon, the metric itself will be a part of a full query that will be treated as a single metric by HPA. Since we are creating something new, we need a name. So, we specified the name section with a single as entry set to http_req_per_second. That will be the reference in our HPA definitions.

You ...