Intro to Model Explainability

As businesses across sectors implement ML and AI, the need for transparent decision-making grows increasingly important. The problem with black-box models (neural networks, large language models, etc.) is that their decision process is entirely opaque and unauditable. Model explainability has evolved as a subfield to combat this problem.

Explainability vs. interpretability

Simply put, explainability attempts to provide some clarity into how an ML algorithm makes its decision. Interpretability is the ability to have clarity into why an ML algorithm made a decision. The difference is subtle but has significant consequences, notably that explainability is just one piece of interpretability. This is best illustrated in an example.

Consider again a lending algorithm that attempts to classify applicants as either able or unable to repay a loan. Let’s assume we run two different models: a logistic regression and a random forest.

Logistic regressions

With a logistic regression model, it’s possible to retrieve the exact parameters that went into making the prediction. Because logistic regressions are essentially linear models and have a set formula, anyone can instantly understand properties, such as which features were the most relevant and (much more difficult) the effect of changes to variables on the output.