Ways to Categorize Prediction Models

There are several ways to categorize prediction models developed under classification and regression techniques. One approach is to categorize them through two dimensions: parametric/nonparametric and discriminative/generative.

Dimensions of categorizing the models

In general, parametric and generative models are structurally constrained, while nonparametric and discriminative models place less emphasis on capturing the data’s structures and more on outcome accuracy. Another approach is to categorize them according to how human interpretable the model as a black box or white box. White-box models are those easily transformed into human-readable rules, while black-box models are not.

The figure below shows the different classifiers discussed in this chapter along with the three categorization approaches. As we can observe from the figure below, most classification methods are black-box methods. Only tree-based models, such as DTs and RFs are white-box methods and thus interpretable. We’ll learn more about these methods and their limitations later. However, it is important to note that there have been several approaches developed to interpret some of the black-box methods through explainable artificial intelligence methods. Interested learners should refer to articles within this area for more information on these methods and their limitations.

Parametric vs. non-parametric models

Parametric models refer to models that assume that the data follows a predefined structure characterized by a finite set of parameters (for example, fitting a linear equation of a line). Such models assume that our data can fit a predefined structure, and the modeling process is then focused on setting the parameters for the equation that best explains the data given the assumed structure or function (for example, linear or quadratic). In contrast, nonparametric models do not have any assumptions about the structure of the data. As a result, they are regarded as models with a potentially infinite number of parameters, contrary to a fixed number in the case of parametric counterparts.