Back To Course Home

Getting Started with Google BERT

0% completed

Before We Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer Understanding the Encoder of the Transformer Self-Attention Mechanism Understanding the Self-Attention Mechanism Multi-Head Attention Mechanism Learning Position with Positional Encoding Encoder: Feedforward and Add and Norm Component Putting All the Encoder Components Together Understanding the Decoder of the Transformer Masked Multi-Head Attention Multi-Head Attention Decoder: Feedforward and Add and Norm Component Putting All the Decoder Components Together Putting the Encoder and Decoder Together Summary: A Primer on Transformers Exercise: A Primer on Transformers Exercise: Decoder Architecture Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to the BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-Level Byte Pair Encoding and WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model

Extracting Embeddings From All Encoder Layers of BERT

Fine-Tuning BERT for Downstream Tasks

Sentiment Analysis

Natural Language Interface

Question-Answering

Performing Question-Answering with the Fine-Tuned BERT

Named Entity Recognition

Summary: Getting Hands-On with BERT

Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants—Based on Knowledge Distillation Quiz: BERT Variants—Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT with a Siamese Network Sentence-BERT with a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Summary: Getting Hands-On with BERT

Let’s summarize what we have learned so far.

We'll cover the following

Key highlights

Key highlights

Summarized below are the main highlights of what learned in this chapter.

We looked at different configurations of the pre-trained BERT model provided by Google.
We learned that we can use the pre-trained BERT model in two ways: as a feature extractor by extracting embeddings, and by fine-tuning the pre-trained BERT model for downstream tasks such as text classification, question-answering, and more.
We learned how to extract embeddings from the pre-trained BERT model in detail.

Get hands-on with 1400+ tech skills courses.