Refining Neural Networks with Class Weights

Loss function

A rare event problem has very few positively labeled samples. Due to this, even if the classifier misclassifies the positive labels, their effect on the loss function is minuscule.

$$\mathcal{L(\theta)}=−\frac{1}{n}\sum_{i=1}^{n}\big( y_{i}log(p_{i})+(1−y_{i})log(1−p_{i})\big)$$

where

• $y_{i} ∈ 0,1$ are the true labels.
• $p_{i} = \text{Pr}[y_{i} = 1]$ is the predicted probability for $y_{i} = 1$ .

Remember the loss function in the equation above; it gives equal importance to the positive and negative samples. We can overweight the positives and underweight the negative samples to overcome this. A binary cross-entropy loss function will then be:

$$\mathcal{L(\theta)}=−\frac{1}{n} \sum_{i=1}^{n} \big(w_{1}y_{i}log(p_{i})+w_{0}(1−y_{i})\log(1−p_{i})\big)$$

where

• $w_1 > w_0$.
• $y_{i} ∈ 0,1$ are the true labels.
• $p_{i} = \text{Pr}[y_{i} = 1]$ is the predicted probability for $y_{i} = 1$.

Class-weighting approach

The class-weighting approach works as follows:

• The model estimation objective is to minimize the loss. In a perfect case, if the model could predict all the labels perfectly, that is, $p_i = 1|y_i = 1$ and $p_i = 0|y_i = 0$, the loss will be zero. Therefore, the best model estimate is the one with the loss closest to zero.

• With the class weights, $w_1 > w_0$, if the model misclassifies the positive samples, that is, $p_i → 0|y_i = 1$, the loss goes farther away from zero as compared to if the negative samples are misclassified. In other words, the model training penalizes the misclassification of positives more than negatives.

• Therefore, the model estimation strives to classify the minority positive samples correctly.

In principle, any arbitrary weights such that $w_1 > w_0$ can be used. But a rule-of-thumb is:

• $w_1$:

  $$\text{positive class weight} = \frac{\text{number of negative samples}} {\text{total samples}}$$

• $w_0$:

  $$\text{negative class weight} = \frac{\text{number of positive samples}} {\text{total samples}}$$