Logging, Alerting, Monitoring, and Auditing

Overview

In this lesson, we will go through the structure of logs within a system. We will discuss time management, minimizing bugs, and completing health checks.

Log messages structure

Logs are entirely distinct aspects of knowing the health of a system. Errors are logged. Services and major subsystems starting and stopping should be logged. If you create a log, find means of reading that log (either as a dashboard or an alert). Document the mappings between the log and the feature that reads it. Log messages should be structured for ease of searching. Include a correlation identifier in the log messages, for example, order id. Logs should be structured in the form:

INFO, starting xxx.

INFO, finished xxx.

ERROR, failed xxx.

Don’t use errors for informational logs. Keep in mind that there might be other log levels used. For example, a log that used the level CRITICAL might not be spotted by a filter that's looking for ERROR. Consider using a log aggregation service that sorts the messages by their timestamp. We can’t always trust code to send log messages in the correct order, but with a properly synced timestamp, we can reconstruct most of the correct sequence. Alerts help detect unusual behavior in the system.