Confidence Intervals,Margins of Error,and Confidence Levels in UX

Collecting metrics is a significant practice in a usability study. There are different ways of collecting metrics, some of which are mentioned below:

Confidence interval and margins of error

A confidence interval is the probable range for the true score of the entire population, e.g., if we say that 57%+/-2% of the Pakistani population has blood pressure, we do not know whether the actual number is 56% or 58% or somewhere in-between. However, we do know that it falls somewhere between 56% and 58%. This range is known as confidence interval.

Margin of error and confidence interval are quite similar to each other. Usually, only one of them is reported. The confidence interval can be computed using the margin of error and observed score. The confidence interval’s width is twice that of the margin of error.

As shown in the image above:

• lower bound: observed score minus the margin of error.
• upper bound: observed score plus the margin of error.
• width: twice the margin of error.

There are various methods to calculate the confidence interval based on the type of metric being calculated that are out of the scope of this article.

A narrow confidence interval is better than a wider one, e.g., if the success rate of a UI is 40% and the confidence interval is between 10% to 70%, that is not helpful as the values can fall anywhere from as low as 10% to as high as 70%. This interpretation does not help decide further steps required.

Narrower confidence intervals are the most helpful ones and give more useful data. The following factors usually impact the confidence interval size:

1. Sample size - the larger the sample size, the smaller the confidence interval.
2. Sample variability - the more the variability, the wider the confidence interval.
3. The confidence level - the extent to which we can trust the confidence-interval calculation.

The confidence level conveys the certainty that the true score will fall within your calculation of the confidence interval. For example, if the confidence level is 95%, there is a 95% chance that your true score falls within the confidence interval.

In UX, lower confidence levels are acceptable instead of scientific publications, where 95% is the acceptable level. Depending on what is being considered, a low confidence level may be acceptable, e.g., predicting the completion rate of an insignificant task.

A higher confidence level results in a wider confidence interval as compared to that which corresponds to a lower confidence level.

Lower confidence levels lead to a narrower confidence interval. Still, there is a higher chance that the true score may not fall in the corresponding confidence interval in the general population.

Conclusion

Narrow confidence intervals carry more information but require a larger sample. A higher confidence level leads to leads to a more correct range but is costly. Choosing the right values depends on researchers.