Back To Course Home

Getting Started with Google BERT

0% completed

Before We Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer Understanding the Encoder of the Transformer Self-Attention Mechanism Understanding the Self-Attention Mechanism Multi-Head Attention Mechanism Learning Position with Positional Encoding Encoder: Feedforward and Add and Norm Component Putting All the Encoder Components Together Understanding the Decoder of the Transformer Masked Multi-Head Attention Multi-Head Attention Decoder: Feedforward and Add and Norm Component Putting All the Decoder Components Together Putting the Encoder and Decoder Together Summary: A Primer on Transformers Exercise: A Primer on Transformers Exercise: Decoder Architecture Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to the BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-Level Byte Pair Encoding and WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model Transformers Extracting Embeddings From All Encoder Layers of BERT Fine-Tuning BERT for Downstream Tasks Sentiment Analysis Natural Language Interface Question-Answering Performing Question-Answering with the Fine-Tuned BERT Named Entity Recognition Summary: Getting Hands-On with BERT Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants—Based on Knowledge Distillation Quiz: BERT Variants—Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT with a Siamese Network

Sentence-BERT with a Triplet Network

Use Cases of Sentence-BERT Model

Learning Multilingual Embeddings Through Knowledge Distillation

Multilingual Sentence-BERT Model

Domain-Specific BERT

Summary: Exploring Sentence and Domain-Specific BERT

Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Quiz: Exploring Sentence and Domain-Specific BERT

Take a short quiz to test your understanding of Sentence-BERT and domain-specific BERT.

1

Which method obtains a sentence representation that hold the meaning of important words (tokens)?

A)

The representation of the [CLS] token

B)

Min pooling

C)

Mean pooling

D)

Max pooling

Question 1 of 70 attempted

Get hands-on with 1400+ tech skills courses.