Next Sentence Prediction
Learn how to pre-train the BERT model for the next sentence prediction task.
What is NSP task?
Next sentence prediction (NSP) is another interesting strategy used for training the BERT model. NSP is a binary classification task. In the NSP task, we feed two sentences to BERT and it has to predict whether the second sentence is the follow-up (next sentence) of the first sentence. Let's understand the NSP task with the help of an example.
The isNext label
Consider the following two sentences:
In the preceding pair of sentences, sentence B is a follow-up, that is, the sentence following on from sentence A. So, we label this sentence pair as isNext, indicating that sentence B follows on from sentence A.
The notNext label
Now, consider the following two sentences:
In the preceding sentence pair, sentence B is not a follow-up. That is, it does not follow on from sentence A. So, we label this sentence pair as notNext, indicating that sentence B does not follow on from sentence A.
The NSP as a binary classification task
In the NSP task, the goal of our model is to predict whether the sentence pair belongs to the isNext or notNext category. We feed the sentence pair (sentences A and B) to BERT and train it to predict whether sentence B follows on from sentence A. The model returns isNext if sentence B follows on from sentence A, otherwise, it will return notNext as an output. Thus, NSP is essentially a binary classification task.