Hypothesis Testing

Hypothesis testing

During our analysis of the different datasets, we are often concerned with questions like whether males default more than females? Do self-driving cars crash more than normal cars? Does drug X help prevent/treat disease Y? To answer these questions, we can use another statistical technique known as Hypothesis Testing.

During data exploration, we discovered interesting patterns hidden in the data. Hypothesis testing enables us to confirm whether these patterns were present in the data by luck or by some real phenomena.

Null and Alternate hypothesis

The aim of the hypothesis test is to determine whether the null hypothesis can be rejected or not. The null hypothesis is a statement that assumes that nothing interesting is going on, or no relationship is present between two variables, or that there is no difference between a sample and a population.

For instance, if we suspect that males default more than females, the null hypothesis would be that males do not default more than females. If there is little or no evidence against the null hypothesis, we accept the null hypothesis. Otherwise, we reject the null hypothesis in favor of the alternate hypothesis, which states that something interesting is going on, or there is a relationship between two variables, or that the sample is different from the population.

To reiterate, the null hypothesis is assumed true and statistical evidence is required to reject it in favor of the alternative hypothesis.