Natural Language Processing with TensorFlow/

...

/

Training and Evaluating Model and Answering Questions with BERT

Training and Evaluating Model and Answering Questions with BERT

Learn train and evaluate the BERT model.

We'll cover the following...

Training the model

We already have the data prepared and the model defined. Training the model is quite easy and is just a one-liner:

model_v2.fit(
    train_dataset,
    validation_data=valid_dataset,
epochs=3 )

We should see an output as follows:

Epoch 1/2
19400/19400 [==============================] - 7175s 369ms/step
398 Transformers
- loss: 2.7193 - tf_bert_for_question_answering_loss: 1.4153 - tf_
bert_for_question_answering_1_loss: 1.3040 - tf_bert_for_question_
answering_sparse_categorical_accuracy: 0.5975 - tf_bert_for_question_
answering_1_sparse_categorical_accuracy: 0.6376 - val_loss: 2.1615
- val_tf_bert_for_question_answering_loss: 1.0898 - val_tf_bert_for_
question_answering_1_loss: 1.0717 - val_tf_bert_for_question_answering_
sparse_categorical_accuracy: 0.7120 - val_tf_bert_for_question_
answering_1_sparse_categorical_accuracy: 0.7350
Epoch 2/2
19400/19400 [==============================] - 7192s 370ms/step
- loss: 1.6691 - tf_bert_for_question_answering_loss: 0.8865 - tf_
bert_for_question_answering_1_loss: 0.7826 - tf_bert_for_question_
answering_sparse_categorical_accuracy: 0.7245 - tf_bert_for_question_
answering_1_sparse_categorical_accuracy: 0.7646 - val_loss: 2.1836
- val_tf_bert_for_question_answering_loss: 1.0988 - val_tf_bert_for_
question_answering_1_loss: 1.0847 - val_tf_bert_for_question_answering_
sparse_categorical_accuracy: 0.7289 - val_tf_bert_for_question_
answering_1_sparse_categorical_accuracy: 0.7504
It took 14366.591783046722 seconds to complete the training

Evaluating the model

We should see the accuracy on the validation set reaching an accuracy between 73% and 75%. This is quite high, given we only trained the model for two epochs. This performance can be attributed to the high level of language understanding the pretrained model already had when we downloaded it. Let’s evaluate the model on our test data:

 model_v2.evaluate(test_dataset)

It should output the following:

1322/1322 [======================] - 345s 261ms/step - loss: 2.2205
- tf_bert_for_question_answering_loss: 1.1325 - tf_bert_for_question_
answering_1_loss: 1.0881 - tf_bert_for_question_answering_sparse_
categorical_accuracy: 0.6968 - tf_bert_for_question_answering_1_sparse_
categorical_accuracy: 0.7250

We see that it ...