0% completed

All LessonsFree Lessons (5)

Before We Start

About the Course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer

Understanding the Encoder of the Transformer

Self-Attention Mechanism

Understanding the Self-Attention Mechanism

Multi-Head Attention Mechanism

Learning Position with Positional Encoding

Encoder: Feedforward and Add and Norm Component

Putting All the Encoder Components Together

Understanding the Decoder of the Transformer

Masked Multi-Head Attention

Multi-Head Attention

Decoder: Feedforward and Add and Norm Component

Putting All the Decoder Components Together

Putting the Encoder and Decoder Together

Summary: A Primer on Transformers

Exercise: A Primer on Transformers

Exercise: Decoder Architecture

Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to the BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-Level Byte Pair Encoding and WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model

Extracting Embeddings From All Encoder Layers of BERT

Fine-Tuning BERT for Downstream Tasks

Sentiment Analysis

Natural Language Inference

Question-Answering

Performing Question-Answering with the Fine-Tuned BERT

Named Entity Recognition

Summary: Getting Hands-On with BERT

Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants—Based on Knowledge Distillation Quiz: BERT Variants—Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Project

Semantic Search with Transformers

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT with a Siamese Network Sentence-BERT with a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Project

Similarity Detection in English Language Using RoBERTa

Getting Started with Google BERT/

...

/

Summary: A Primer on Transformers

Summary: A Primer on Transformers

Let’s summarize what we have learned so far.

We'll cover the following...

Key highlights

Key highlights

Summarized below are the main highlights of what we have learned in this chapter.

We learned what the transformer model is and how ...