Back To Course Home

Transformers for Natural Language Processing

0% completed

Course Introduction

Getting Started

Basics of Transformers

Introduction: Basics of Transformers Transformers and Industry 4.0 The History of Transformers Resources to Use Summary: Basics of Transformers Quiz: Basics of Transformers

Architecture of the Transformer Model

Introduction: Architecture of the Transformer Model

The Rise of the Transformer: Attention Is All You Need

Input Embedding

Positional Encoding

Sublayer 1: Multi-Head Attention

Building Sublayer 1: Multi-Head Attention

Post-Layer Normalization and Sublayer 2: Feedforward Network

The Decoder Stack and Hugging Face

Summary: Architecture of the Transformer Model

Quiz: Architecture of the Transformer Model

Challenge 1: Build a Multi-Head Attention Sublayer

Solution: Build a Multi-Head Attention Sublayer

Challenge 2: Evaluate Positional Encodings of a Sentence

Solution: Evaluate Positional Encodings of a Sentence

Fine-Tuning BERT Models

Introduction: Fine-Tuning BERT Models Architecture of BERT Preparing the Pretraining Input Environment Pretraining and Setting Up a BERT Model Configuring and Training a BERT Model Evaluating a BERT Model Summary: Fine-Tuning BERT Models Quiz: Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Introduction: Pretraining a RoBERTa Model from Scratch Preliminary Steps Involved in Building KantaiBERT Main Steps Involved in Building KantaiBERT Summary: Pretraining a RoBERTa Model from Scratch Quiz: Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Introduction: Downstream NLP Tasks with Transformers Transduction and the Inductive Inheritance of Transformers Evaluating Models with Metrics Benchmark Tasks and Datasets SuperGLUE Benchmark Tasks Running Downstream Tasks Summary: Downstream NLP Tasks with Transformers Quiz: Downstream NLP Tasks with Transformers Challenge: Run Downstream Tasks Solution: Run Downstream Tasks

Machine Translation with the Transformer

Introduction: Machine Translation with the Transformer Types of Transductions and Translations Preprocessing Datasets Finalizing Data Preprocessing Evaluating Machine Translation with BLEU Implementing Translations Summary: Machine Translation with the Transformer Quiz: Machine Translation with the Transformer Challenge: Preprocess and Clean a Czech-English Dataset Solution: Preprocess and Clean a Czech-English Dataset

The Rise of Transformers with GPT-3 Engines

Introduction: The Rise of Transformers with GPT-3 Engines Architecture of OpenAI GPT Transformer Models Evolving From Fine-Tuning to Zero-Shot Models Generic Text Completion with GPT-2 Running OpenAI GPT-3 Tasks NLP Tasks and Examples Fine-Tuning GPT-3 The Role of an Industry 4.0 AI Specialist Summary: The Rise of Transformers with GPT-3 Engines Quiz: The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Introduction: Applying Transformers for AI Text Summarization Text to Text Transformer Models The T5 Model Text Summarization with T5 Summarizing Documents with T5-Large Summary: Applying Transformers for AI Text Summarization Quiz: Applying Transformers for AI Text Summarization Challenge: Summarize Efficient Solar Panel Breakthroughs Solution: Summarize Efficient Solar Panel Breakthroughs

Matching Tokenizers and Datasets

Introduction: Matching Tokenizers and Datasets Best Practices Word2Vec Tokenization Standard NLP Tasks with Specific Vocabulary Summary: Matching Tokenizers and Datasets Quiz: Matching Tokenizers and Datasets

Course Assessment

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Introduction: Semantic Role Labeling with BERT-Based Transformers Getting Started with SRL Basic Samples Difficult Samples Questioning the Scope of SRL Summary: Semantic Role Labeling with BERT-Based Transformers Quiz: Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Introduction: Your Data Speaks: Story, Questions, and Answers Method 0: Trial and Error Method 1: NER First Method 2: SRL First Next Steps Summary: Your Data Speaks: Story, Questions and Answers Quiz: Your Data Speaks: Story, Questions, and Answers Challenge: Q&A with Transformers Solution: Q&A with Transformers

Detecting Customer Emotions to Make Predictions

Introduction: Detecting Customer Emotions to Make Predictions Sentiment Analysis Transformers Predicting Customer Behavior with Sentiment Analysis Sentiment Analysis with Hugging Face Models Sentiment Analysis with GPT-3 Summary: Detecting Customer Emotions to Make Predictions Quiz: Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Introduction: Analyzing Fake News with Transformers Emotional Reaction to Fake News Behavioral Representation of Fake News A Rational Approach to Fake News A Political Approach to Fake News Summary: Analyzing Fake News with Transformers Quiz: Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

Introduction: Interpreting Black Box Transformer Models Transformer Visualization with BertViz LIT and PCA Transformer Visualization via Dictionary Learning Summary: Interpreting Black Box Transformer Models Quiz: Interpreting Black Box Transformer Models Challenge: Visualize and Complete Sentences Solution: Visualize and Complete Sentences

From NLP to Task-Agnostic Transfomer Models

Introduction: From NLP to Task-Agnostic Transformer Models Choosing a Model and an Ecosystem Vision Transformers The CLIP Model The DALL·E Model Summary: From NLP to Task-Agnostic Transformer Models Quiz: From NLP to Task-Agnostic Transformer Models

The Emergence of Transformer-Driven Copilots

Introduction: The Emergence of Transformer-Driven Copilots Prompt Engineering Training in Prompt Engineering Domain-Specific GPT-3 Engines Transformer-Based Recommender Systems Transformers Applied to Computer Vision Summary: The Emergence of Transformer-Driven Copilots Quiz: The Emergence of Transformer-Driven Copilots

Course Assessment

Transformers for NLP—Exam 2

Conclusion

Appendix

Terminology of Transformer Models Hardware Constraints for Transformer Models Generic Text Completion with GPT-2 Attaining OpenAI API Keys

Input Embedding

Learn about input embeddings and how they are used in transformers in this lesson.

We'll cover the following

Input Embedding

This global view of the encoder shows the highly optimized architecture of the transformer. Now, let's zoom into the sublayers and mechanisms—we'll begin with the embedding sublayer.

Input Embedding

The input embedding sublayer converts the input tokens to vectors of dimension $d_{model} \text{ = 512}$ using learned embeddings in the original transformer model. The structure of the input embedding is classical:

Get hands-on with 1200+ tech skills courses.