Deep Learning with PyTorch Step-by-Step: Part I - Fundamentals/

...

/

Trade-offs and Curves

Trade-offs and Curves

Learn about receiver operating characteristic (ROC) and precision-recall (PR) curves. And take a look at how lowering or increasing thresholds affects the results.

We'll cover the following...

Introduction to ROC and PR curves

We already know there are trade-offs between true and false-positive rates as well as between precision and recall. We also know that there are many confusion matrices, one for each threshold. What if we were to combine these two pieces of information?

We present to you the receiver operating characteristic (ROC) and precision-recall (PR) curves! Well, they are not curves yet, but they will be soon enough.

We have already computed the TPR/recall (91%), FPR (22%), and precision (83%) for our model using the threshold of 50%. If we plot them, we will get the figure above.

Time to try different thresholds.

Low threshold

What about 30%? If the predicted probability is greater than or equal to 30%, we classify the data point as positive and as negative otherwise. That is a very loose threshold since we do not require the model to be very confident to consider a data point to be positive. What can we expect from it? More false positives, less false negatives.

You can see in the figure above that lowering the threshold (moving it to the left on the probability line) turned one false negative into a true positive (blue point close to 0.4), but it also turned one true negative into a false positive (red point close to 0.4).

Let us double-check it with Scikit-learn’s confusion matrix:

Press + to interact
cm = confusion_matrix(y_val, (probabilities_val >= 0.3))
print(cm)

Okay, now let us plot the corresponding metrics one more time:

Still not a curve, I know. But we can already learn something from having these two points.


Lowering the threshold moves you to the right along both curves.

Let us move to the other side now.

High threshold

What about 70%? If the predicted probability is greater than or equal to 70%, we classify the data point as positive and as negative otherwise. That is a very strict threshold since we require the model to be very confident to consider a data point to be positive. What can we expect from it? Less false positives, more false negatives.

You can see in the figure above that raising the threshold (moving it to the right on the probability line) turned two false positives into true negatives (red points close to 0.6), but it also turned one true positive into a false negative (blue point close to 0.6).

Let us double-check it with Scikit-learn’s confusion matrix:

Press + to interact
cm = confusion_matrix(y_val, (probabilities_val >= 0.7))
print(cm)

Okay, now let us plot the corresponding metrics again:

I guess we earned the right to call it a curve now.


Raising the threshold moves you to the left along both curves.

Can we just connect the dots and call it a curve for real? Actually, no, not yet.

ROC and PR curves

We need to try out more thresholds to actually build a curve. Let us try multiples of 10%:

Press + to interact
import numpy as np
threshs = np.linspace(0,1,11)
print(threshs)

If we try plotting this, we get the following graph:

Cool! We finally have proper curves! We have some questions for you:

  • In each plot, which point corresponds to a threshold of zero (every prediction is positive)?

  • In each plot, which point corresponds to a threshold of one (every prediction is negative)?

  • What does the right-most point in the PR curve represent?

  • If we raise the threshold, how do we move along the curve?

You should be able to answer all of those questions by referring to the Metrics lesson. But, if you are eager to get the answers, here they are:

  • The threshold of zero corresponds to the right-most point in both curves.

  • The threshold of one corresponds to the left-most point in both curves.

  • The right-most point in the PR curve represents the proportion of ...