Detailed Design of Monitoring System

We will discuss the core components of our monitoring system, identify the shortcomings of our design, and improve the design to fulfill our requirements.

Storage

We will use time-series databases to save the data locally on the server where our monitoring service is running and then integrate it with a separate storage node. We will use blob storage to store our metrics.

We need to store metrics and know which action to perform if a metric has reached a particular value. For example, if CPU usage exceeds 90%, generate an alert to the end-user so the alert receiver can do the needful like allocate more resources to scale. For this purpose, we need another storage, let’s call it a rules database, that will contain the rules and actions. Upon violation of rules, we can take appropriate action.

Here we have identified two more components in our design, i.e., rules and action DB and storage node (a Blob store).

Data collector

We need a monitoring system to update us about our several data centers. We can stay updated if the information about our processes reaches us, which is possible through logging. We will choose a pull strategy. We will extract our relevant metrics from the logs of the application. As discussed in our logging design, we used a ...