Simpson's paradox: the paradox of aggregation

It’s often desirable to mix chemicals to get better results in chemistry. The same is true for combining statistical data. However, just like in chemistry, it is not always the case that the results will be better. Sometimes they are, sometimes they aren’t.

An observation is that a relationship between variables appears (or disappears) when conducting experiments, but the relationship might disappear (or appear) when we divide the variables into subgroups. For example, a relationship exists between salary and age, i.e., salary increases with age. However, the relationship disappears when we divide the age into subpopulations (young and old).

Yule-Simpson effect#

Let $A,$$B$, and $C$be events with $A',$ $B’$ & $C’$ as their complement events, respectively. Let’s consider a situation where event $A$ is more likely to occur whenever $B$ does not occur.

This could lead to a different result when we consider event $C$. The likelihood of $A$ given $B$ and $C$ might be more than the case where $B$ does not occur and $C$ occurs.

The likelihood of $A$ given that $B$ occurs and $C$ does not occur might be more than the case where $A$ occurs given both $B$ and $C$ don’t occur.

Example#

Suppose a total of 50 cases opted for surgery and 30 for medicine. Looking at the following data, it's clear that neither surgery nor medicine shows clear success or failure.

We can see that:

#

Now, let’s look at the data when broken down by sex:

Let's look at the following probabilities:

Here, it's clear that the aggregated results differ altogether from the individual results. The aggregated result (disregarding sex) is indifferent to any procedure. While here we see that surgery has a higher chance of success for male patients and a higher chance of failure for female patients. At the same time, medical treatment is almost equally successful for male and female patients.

Conclusion#

The paradox of aggregation, also commonly known as Simpson’s paradox, is not really a paradox. It’s just that patterns are obfuscated when data features are aggregated or separated.

Why is this important for data scientists? Data scientists base their inferences and design models on data features. Simpson’s paradox provides an important insight that data scientists should look at different dataset features and perform a thorough analysis. This could help reduce bad decisions made by their models.