Generative Learning

Discriminative vs generative models

Machine Learning models are often categorized into discriminative and generative models. This distinction arises from the probabilistic formulation that we use to build and train those models.

Discriminative models learn the probability of a label $y$ based on a data point $x$. In mathematical terms, this is denoted as $p( y | x)$. In order to categorize a data point into a class, we need to learn a mapping between the data and the classes. This mapping can be described as a probability distribution. Each label will “compete” with the other ones for probability density over a specific data point.

Generative models, on the other hand, learn a probability distribution over the data points without external labels. Mathematically, this is formulated as $p(x)$. In this case, we have the data themselves “compete” for probability density.

Conditional Generative models are another category of models that try to learn the probability distribution of the data $x$ conditioned on the labels $y$. As you can probably tell, this is denoted as $p(x|y)$. Here, we again have the data “compete” for density, but for each possible label.

It is important to clarify the notion of competition here. The probability density function $p$ is a normalized function whose integral overall value is equal to 1.

$$\int_{X} p(x) dx =1$$

It is evident that each data point ...