Adaboost

Boosting

Boosting is a very popular machine learning technique, and it is widely used for online ML competitions and industries. Boosting is the technique of combining weak classifiers to make a strong classifier. Weak learners can come from any other classifier like logistic regression, decision trees, etc. When we apply weak classifiers, like decision trees with a depth of 2, we get a very low variance error, but the bias error is high. To handle this, we increase the complexity or increase the depth. This training error is reduced, but the testing error starts increasing. Boosting helps with these scenarios.

We combine the weak learners with weights. First, we start by training the first weak classifier. In the next classifier, we focus on the examples where the first classifier was wrong and increase their weights. The idea behind boosting is to focus on the examples in which it is hard to get the predictions right. We keep changing the weight of the data example after the result of each classifier. Finally, we combine these classifiers in a weighted manner to get the result.

Ensemble classifiers

We have discussed different classifiers for predicting the output. Consider three decision tree classifiers that have one depth and predict the output.

Now for each example, we can pass through each of the trees and get the answer. We can assign a weight for each of the tree answers and get the final answer by weighted voting. Let’s say we have $C_1$,$C_2$,…$C_n$ classifiers and $W_1$,$W_2$,…$W_n$ weights. Their respective weights on the final prediction will be:

$$Y= g(W_1C_1 + W_2C_2 + ... W_nC_n)$$ ...