Gain insights into using SQL for anomaly detection. Delve into applying mean, standard deviation, and z-score to identify and investigate data anomalies, enhancing your analytical skills.

ruby.tar.gz

Learn how to identify anomalies in your data using SQL. Understand how to apply mathematical concepts such as mean, standard deviation, and z-score to detect when an anomaly occurred in your data and how to investigate past data to improve and refine the model.

Simple Anomaly Detection using SQL

In the last lesson, we detected a large number of "anomalies" with just one entry. This is very common in errors that do not happen very often. In our case, every once in a while, we get a `400` status code. However, the standard deviation is very low because it does not happen very often so even a single error can be considered way above the acceptable value.

We do not really want to receive an alert in the middle of the night just because of one `400` status code. We cannot have every curious developer fiddling with the dev tools in his browser wake us up in the middle of the night.

To eliminate rows with only a few entries, we set a threshold:

Learn the intuition behind, and the application of, thresholds in finding anomalies.

Detecting Anomalies

Analyzing A Server Log

Backtesting

Improving Accuracy

Adding Thresholds