Evaluating M-BERT on the NLI task
Learn how to evaluate M-BERT by fine-tuning it on the natural language inference (NLI) task.
In the NLI task, the goal of our model is to determine whether a hypothesis is an entailment (true), contradiction (false), or undetermined (neutral) given a premise. Thus, we feed a sentence pair (premise-hypothesis pair) to the model, and it has to classify whether the sentence pair (premise-hypothesis pair) belongs to entailment, contradiction, or is an undetermined class.
Dataset for NLI task
What dataset can we use for this task? For the NLI task, we generally use the Stanford Natural Language Inference (SNLI) dataset. But since we are evaluating M-BERT in this instance, we use a different dataset called cross-lingual NLI (XNLI). The XNLI dataset is based on a MultiNLI dataset. So, first, let's take a look at the MultiNLI dataset.
MNLI dataset
Multi-Genre Natural Language Inference (MultiNLI) is a corpus similar to SNLI. It consists of premise-hypothesis pairs across various genres. A sample of the MultiNLI dataset is shown in the following table:
MNLI dataset
Genre | Premise | Hypothesis | Label |
Letters | Will you add your dreams to ours? | Will you help us build the best school in the nation? | Neutral |
911 | For the rescue efforts, see FDNY report, Report from the Chief of Department Anthony L. Fusco, in Manning,ed | Anthony L. Fusco, the Chief of Department, wrote a report on the rescue efforts. | Entailment |
Fiction | That was in Bridgetown on the night of the Spanish raid. | There was never a raid on Bridgetown. | Contradiction |
Travel | Just a few blocks in back of the malecen are a growing collection of unique clubs with a more urban edge. | The collection is growing, but not as fast as it did last year. | Neutral |
Verbatim | Serious crime down but murders increase. | There has been a rise in murders. | Entailment |
As we can observe, we have a genre, along with the premise-hypothesis pair and the corresponding label.
XNLI dataset
Now let's see how the XNLI dataset is based on the MultiNLI dataset. The XNLI dataset is an extension of the MultiNLI dataset. The XNLI training set consists of 433K English sentence pairs (premise-hypothesis pairs) from the MultiNLI corpus. ...