Back To Course Home

Getting Started with Google BERT

0% completed

Before We Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer Understanding the Encoder of the Transformer Self-Attention Mechanism Understanding the Self-Attention Mechanism Multi-Head Attention Mechanism Learning Position with Positional Encoding Encoder: Feedforward and Add and Norm Component Putting All the Encoder Components Together Understanding the Decoder of the Transformer Masked Multi-Head Attention Multi-Head Attention Decoder: Feedforward and Add and Norm Component Putting All the Decoder Components Together Putting the Encoder and Decoder Together Summary: A Primer on Transformers Exercise: A Primer on Transformers Exercise: Decoder Architecture Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to the BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-Level Byte Pair Encoding and WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model Transformers Extracting Embeddings From All Encoder Layers of BERT Fine-Tuning BERT for Downstream Tasks Sentiment Analysis Natural Language Interface Question-Answering Performing Question-Answering with the Fine-Tuned BERT Named Entity Recognition Summary: Getting Hands-On with BERT Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants—Based on Knowledge Distillation Quiz: BERT Variants—Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT

Evaluating M-BERT on the NLI task

How Multilingual is Multilingual BERT?

M-BERT Generalization

Effect of Code-Switching and Transliteration

The Cross-Lingual Language Model (XLM)

The XLM-R Model

Language-Specific BERT

BETO for Spanish

BERTje for Dutch

FinBERT for Finnish

BERT Models for Italian and Portuguese

RuBERT for Russian

Summary: Applying BERT to Other Languages

Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT with a Siamese Network Sentence-BERT with a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Quiz: Applying BERT to Other Languages

Take a short quiz to test your understanding of the multilingual BERT model.

1

What is the primary motivation behind oversampling and undersampling in multilingual BERT?

A)

To prioritize high-resource languages for training.

B)

To exclude the low-resource languages from training.

C)

To maintain a balanced data distribution in each language.

D)

To randomly shuffle the training data to achieve balance.

Question 1 of 80 attempted

Get hands-on with 1400+ tech skills courses.