Creating Semaphore Dashboards

Value of dashboards #

If I’m claiming that the value dashboards bring to the table is lower than we think, you might be asking yourself the same question from the beginning of this chapter. Why are we talking about dashboards? Well, I already changed my statement from “dashboards are useless” to “there is some value in dashboards.” They can serve as a registry for queries. Through dashboards, we do not need to memorize expressions that we would need to write in Prometheus. They might be a good starting point of our search for the cause of an issue before we jump into Prometheus for some deeper digging into metrics. But, there is another reason I am including dashboards into the solution.

I love big displays. It’s very satisfying to enter into a room with large screens showing stuff that seem to be important. There is usually a room where operators sit surrounded with monitors on all four walls. That’s usually an impressive sight. However, there is a problem with many such situations. A bunch of monitors displaying a lot of graphs might not amount to much more than a pretty sight. After the initial few days, nobody will stare at graphs. If that’s not true, you can just as well fire that person knowing that he was faking his work.

Let me repeat it one more time.

🔍 Dashboards are not designed for us to stare at them, especially not when they are on big screens where everyone can see them.

Semaphores for big screens #

So, if it’s a good idea to have big screens, but graphs are not a good candidate to decorate them, what should we do instead? The answer lies in semaphores. They are similar to alerts, and they should provide a clear indication of the status of the system. If everything on the screen is green, there is no reason for us to do anything. One of them turning red is a cue that we should do something to correct the problem. Therefore, it is imperative that we try to avoid false positives. If something turns red, and that does not require any action, we are likely to start ignoring it in the future. When that happens, we are risking the situation in which we ignore a real issue, thinking that it is just another false positive. Hence, every appearance of an alarm should be followed by an action. That can be either a fix that will correct the system or a change in the conditions that turned one of the semaphores red. In either case, we should not ignore it.

The main problem with semaphores is that they are not as appealing to CTOs and other decision-makers. They are not colorful, nor do they show a lot of boxes, lines, and numbers. People often confuse usefulness with how pleasing something is to look at. Nevertheless, we are not building something that should be sold to CTOs, but something that can be helpful in our day-to-day work.

🔍 Semaphores are much more useful than graphs as a way to see the status of the system, even though they do not look as colorful and eye-pleasing as graphs.

Creating our first Semaphore #

Please click the Add panel icon from the top-right part of the screen, followed by *Choose Visualization". Select Singlestat. Click the arrow icon next to the Panel Title, and select Edit.