Probabilistic Classification

We have already discussed some classification methods such as support vector machines and decision trees. We’ll now discuss how classification can be seen as an important special case of probabilistic regression. In classification, features are mapped to a finite number of possible categories. We’ll now discuss binary classification again where the two target classes are usually denoted as $0$ and $1$. Later, we can easily generalize the ideas to more classes. We’ll start by describing a binary random variable.

Maximum likelihood estimation (MLE) of the Bernoulli model

An important probabilistic model of a binary random variable is the Bernoulli model. In this model, a random number takes the value of $1$ with probability $\phi$ and the value $0$ with probability $1$ $−$ $\phi$ (the probability of being either of the two choices has to be $1$). This can be nicely combined into the following formula: