We confirmed that Prometheus' metrics, fetched by Prometheus Adapter and converted into Kubernetes’ custom metrics, can be used in HPA. So far, we used metrics pulled by Prometheus through exporters (nginx_ingress_controller_requests). Given that the adapter fetches metrics from Prometheus, it shouldn’t matter how they got there. Nevertheless, we’ll confirm that instrumented metrics can be used as well. That will give us an opportunity to cement what we learned so far and, at the same time, maybe learn a few new tricks.

Pull metrics through Instrumented metrics #

cat mon/prom-adapter-values-svc.yml

The output is yet another set of Prometheus Adapter Chart values.

image:
  tag: v0.5.0
metricsRelistInterval: 90s
prometheus:
  url: http://prometheus-server.metrics.svc
  port: 80
rules:
  default: false
  custom:
  - seriesQuery: 'http_server_resp_time_count{kubernetes_namespace!="",kubernetes_name!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_name: {resource: "service"}
    name:
      matches: "^(.*)server_resp_time_count"
      as: "${1}req_per_second_per_replica"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / count(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
  - seriesQuery: 'nginx_ingress_controller_requests'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        ingress: {resource: "ingress"}
    name:
      as: "http_req_per_second_per_replica"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / sum(label_join(kube_deployment_status_replicas, "ingress", ",", "deployment")) by (<<.GroupBy>>)'

Combining both rules #

This time, we’re combining rules containing different metric series. The first rule is based on the http_server_resp_time_count instrumented metric that originates in go-demo-5. We used it in the Debugging Issues Discovered Through Metrics And Alerts chapter and there’s nothing truly extraordinary in its definition. It follows the same logic as the rules we used before. The second rule is a copy of one of the rules we used before.

What is interesting about those rules is that there are two completely different queries producing different results. However, the name is the same (http_req_per_second_per_replica) in both cases.

“Wait a minute”, you might say. The names are not the same. One is called ${1}req_per_second_per_replica while the other is ...