Back To Course Home

Transformers for Natural Language Processing

0% completed

Course Introduction

Getting Started

Basics of Transformers

Introduction: Basics of Transformers Transformers and Industry 4.0 The History of Transformers Resources to Use Summary: Basics of Transformers Quiz: Basics of Transformers

Architecture of the Transformer Model

Introduction: Architecture of the Transformer Model The Rise of the Transformer: Attention Is All You Need Input Embedding Positional Encoding Sublayer 1: Multi-Head Attention Building Sublayer 1: Multi-Head Attention Post-Layer Normalization and Sublayer 2: Feedforward Network The Decoder Stack and Hugging Face Summary: Architecture of the Transformer Model Quiz: Architecture of the Transformer Model Challenge 1: Build a Multi-Head Attention Sublayer Solution: Build a Multi-Head Attention Sublayer Challenge 2: Evaluate Positional Encodings of a Sentence Solution: Evaluate Positional Encodings of a Sentence

Fine-Tuning BERT Models

Introduction: Fine-Tuning BERT Models Architecture of BERT Preparing the Pretraining Input Environment Pretraining and Setting Up a BERT Model Configuring and Training a BERT Model Evaluating a BERT Model Summary: Fine-Tuning BERT Models Quiz: Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Introduction: Pretraining a RoBERTa Model from Scratch Preliminary Steps Involved in Building KantaiBERT Main Steps Involved in Building KantaiBERT Summary: Pretraining a RoBERTa Model from Scratch Quiz: Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Introduction: Downstream NLP Tasks with Transformers Transduction and the Inductive Inheritance of Transformers Evaluating Models with Metrics Benchmark Tasks and Datasets SuperGLUE Benchmark Tasks Running Downstream Tasks Summary: Downstream NLP Tasks with Transformers Quiz: Downstream NLP Tasks with Transformers Challenge: Run Downstream Tasks Solution: Run Downstream Tasks

Machine Translation with the Transformer

Introduction: Machine Translation with the Transformer Types of Transductions and Translations Preprocessing Datasets Finalizing Data Preprocessing Evaluating Machine Translation with BLEU Implementing Translations Summary: Machine Translation with the Transformer Quiz: Machine Translation with the Transformer Challenge: Preprocess and Clean a Czech-English Dataset Solution: Preprocess and Clean a Czech-English Dataset

The Rise of Transformers with GPT-3 Engines

Introduction: The Rise of Transformers with GPT-3 Engines Architecture of OpenAI GPT Transformer Models Evolving From Fine-Tuning to Zero-Shot Models Generic Text Completion with GPT-2 Running OpenAI GPT-3 Tasks NLP Tasks and Examples Fine-Tuning GPT-3 The Role of an Industry 4.0 AI Specialist Summary: The Rise of Transformers with GPT-3 Engines Quiz: The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Introduction: Applying Transformers for AI Text Summarization

Text to Text Transformer Models

Text Summarization with T5

Summarizing Documents with T5-Large

Summary: Applying Transformers for AI Text Summarization

Quiz: Applying Transformers for AI Text Summarization

Challenge: Summarize Efficient Solar Panel Breakthroughs

Solution: Summarize Efficient Solar Panel Breakthroughs

Matching Tokenizers and Datasets

Introduction: Matching Tokenizers and Datasets Best Practices Word2Vec Tokenization Standard NLP Tasks with Specific Vocabulary Summary: Matching Tokenizers and Datasets Quiz: Matching Tokenizers and Datasets

Course Assessment

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Introduction: Semantic Role Labeling with BERT-Based Transformers Getting Started with SRL Basic Samples Difficult Samples Questioning the Scope of SRL Summary: Semantic Role Labeling with BERT-Based Transformers Quiz: Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Introduction: Your Data Speaks: Story, Questions, and Answers Method 0: Trial and Error Method 1: NER First Method 2: SRL First Next Steps Summary: Your Data Speaks: Story, Questions and Answers Quiz: Your Data Speaks: Story, Questions, and Answers Challenge: Q&A with Transformers Solution: Q&A with Transformers

Detecting Customer Emotions to Make Predictions

Introduction: Detecting Customer Emotions to Make Predictions Sentiment Analysis Transformers Predicting Customer Behavior with Sentiment Analysis Sentiment Analysis with Hugging Face Models Sentiment Analysis with GPT-3 Summary: Detecting Customer Emotions to Make Predictions Quiz: Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Introduction: Analyzing Fake News with Transformers Emotional Reaction to Fake News Behavioral Representation of Fake News A Rational Approach to Fake News A Political Approach to Fake News Summary: Analyzing Fake News with Transformers Quiz: Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

Introduction: Interpreting Black Box Transformer Models Transformer Visualization with BertViz LIT and PCA Transformer Visualization via Dictionary Learning Summary: Interpreting Black Box Transformer Models Quiz: Interpreting Black Box Transformer Models Challenge: Visualize and Complete Sentences Solution: Visualize and Complete Sentences

From NLP to Task-Agnostic Transfomer Models

Introduction: From NLP to Task-Agnostic Transformer Models Choosing a Model and an Ecosystem Vision Transformers The CLIP Model The DALL·E Model Summary: From NLP to Task-Agnostic Transformer Models Quiz: From NLP to Task-Agnostic Transformer Models

The Emergence of Transformer-Driven Copilots

Introduction: The Emergence of Transformer-Driven Copilots Prompt Engineering Training in Prompt Engineering Domain-Specific GPT-3 Engines Transformer-Based Recommender Systems Transformers Applied to Computer Vision Summary: The Emergence of Transformer-Driven Copilots Quiz: The Emergence of Transformer-Driven Copilots

Course Assessment

Transformers for NLP—Exam 2

Conclusion

Appendix

Terminology of Transformer Models Hardware Constraints for Transformer Models Generic Text Completion with GPT-2 Attaining OpenAI API Keys

The T5 Model

Learn about the structure and the features of the T5 model.

We'll cover the following

Architecture
Aspects

Now let’s look at the architecture of the T5 transformer model.

Architecture

Raffel et al. (2019) focused on designing a standard input format to obtain text output. The Google T5 team did not want to try new architectures derived from the original transformer, such as BERT-like encoder-only layers or GPT-like decoder-only layers. Instead, the team focused on defining NLP tasks in a standard format.

Get hands-on with 1200+ tech skills courses.