Saturation-related Issues: Measure Memory Usage

Measure memory usage #

Measuring memory consumption is similar to CPU, and yet there are a few differences that we should take into account. But, before we get there, let’s go back to the Prometheus's graph screen and explore our first memory-related metric.

open "http://$PROM_ADDR/graph"

Just as with CPU, first we need to find out how much memory each of our nodes has.

Please type the expression that follows, press the Execute button, and switch to the Graph tab.

node_memory_MemTotal_bytes

Your result is likely to be different than mine. In my case, each node has around 4GB of RAM.

Knowing how much RAM each node has is of no use without knowing how much RAM is currently available. We can get that info through the node_memory_MemAvailable_bytes metric.

Please type the expression that follows, and press the Execute button.

node_memory_MemAvailable_bytes

We can see the available memory on each of the nodes of the cluster. In my case (screenshot below), each has around 3GB of available RAM.

Now that we know how to get total and available memory from each of the nodes, we should combine the queries to get the percentage of the used memory of the whole cluster.

Please type the expression that follows, and press the Execute button.

1 - 
sum(
  node_memory_MemAvailable_bytes
) / 
sum(
  node_memory_MemTotal_bytes
)

Since we are searching for the percentage of used memory, and we have the metric with available memory, we started the expression with 1 - that will invert the result. The rest of the expression is a simple division of available and total memory. In my case (screenshot below), less than thirty percent of memory is used on each of the nodes.

Measure allocatable memory #

Just as with CPU, available and total memory does not paint the whole picture. While that is useful information and a base for a potential alert, we also need to know how much memory is allocatable and how much of it is in use by Pods. We can get the first figure through the kube_node_status_allocatable_memory_bytes metric.

Please type the expression that follows, and press the Execute button.

kube_node_status_allocatable_memory_bytes

Depending on the Kubernetes flavor and the hosting provider you’re using, there might be a very small or a big discrepancy between the total and allocatable memory. I am running the cluster in AKS, and ...