Nonlinear Support Vector Machines (SVMs)

Soft margin classifier

Thus far, we have only discussed the linearly separable case, but what about the case when there are overlapping classes? It is possible to extend the optimization problem by allowing some data points to be in the margin, while penalizing these points somewhat. We, therefore, include some slack variables $\xi_{i}$ that reduce the effective margin for each data point, but we add a penalty term to the optimization that penalizes if the sum of these slack variables is large, as shown below: