Back To Course Home

Transformers for Natural Language Processing

0% completed

Course Introduction

Getting Started

Basics of Transformers

Introduction: Basics of Transformers Transformers and Industry 4.0 The History of Transformers Resources to Use Summary: Basics of Transformers Quiz: Basics of Transformers

Architecture of the Transformer Model

Introduction: Architecture of the Transformer Model The Rise of the Transformer: Attention Is All You Need Input Embedding Positional Encoding Sublayer 1: Multi-Head Attention Building Sublayer 1: Multi-Head Attention Post-Layer Normalization and Sublayer 2: Feedforward Network The Decoder Stack and Hugging Face Summary: Architecture of the Transformer Model Quiz: Architecture of the Transformer Model Challenge 1: Build a Multi-Head Attention Sublayer Solution: Build a Multi-Head Attention Sublayer Challenge 2: Evaluate Positional Encodings of a Sentence Solution: Evaluate Positional Encodings of a Sentence

Fine-Tuning BERT Models

Introduction: Fine-Tuning BERT Models Architecture of BERT Preparing the Pretraining Input Environment Pretraining and Setting Up a BERT Model Configuring and Training a BERT Model Evaluating a BERT Model Summary: Fine-Tuning BERT Models Quiz: Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Introduction: Pretraining a RoBERTa Model from Scratch Preliminary Steps Involved in Building KantaiBERT Main Steps Involved in Building KantaiBERT Summary: Pretraining a RoBERTa Model from Scratch Quiz: Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Introduction: Downstream NLP Tasks with Transformers Transduction and the Inductive Inheritance of Transformers Evaluating Models with Metrics Benchmark Tasks and Datasets SuperGLUE Benchmark Tasks Running Downstream Tasks Summary: Downstream NLP Tasks with Transformers Quiz: Downstream NLP Tasks with Transformers Challenge: Run Downstream Tasks Solution: Run Downstream Tasks

Machine Translation with the Transformer

Introduction: Machine Translation with the Transformer Types of Transductions and Translations Preprocessing Datasets Finalizing Data Preprocessing Evaluating Machine Translation with BLEU Implementing Translations Summary: Machine Translation with the Transformer Quiz: Machine Translation with the Transformer Challenge: Preprocess and Clean a Czech-English Dataset Solution: Preprocess and Clean a Czech-English Dataset

The Rise of Transformers with GPT-3 Engines

Introduction: The Rise of Transformers with GPT-3 Engines Architecture of OpenAI GPT Transformer Models Evolving From Fine-Tuning to Zero-Shot Models Generic Text Completion with GPT-2 Running OpenAI GPT-3 Tasks NLP Tasks and Examples Fine-Tuning GPT-3 The Role of an Industry 4.0 AI Specialist Summary: The Rise of Transformers with GPT-3 Engines Quiz: The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Introduction: Applying Transformers for AI Text Summarization Text to Text Transformer Models The T5 Model Text Summarization with T5 Summarizing Documents with T5-Large Summary: Applying Transformers for AI Text Summarization Quiz: Applying Transformers for AI Text Summarization Challenge: Summarize Efficient Solar Panel Breakthroughs Solution: Summarize Efficient Solar Panel Breakthroughs

Matching Tokenizers and Datasets

Introduction: Matching Tokenizers and Datasets Best Practices Word2Vec Tokenization Standard NLP Tasks with Specific Vocabulary Summary: Matching Tokenizers and Datasets Quiz: Matching Tokenizers and Datasets

Course Assessment

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Introduction: Semantic Role Labeling with BERT-Based Transformers

Getting Started with SRL

Difficult Samples

Questioning the Scope of SRL

Summary: Semantic Role Labeling with BERT-Based Transformers

Quiz: Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Introduction: Your Data Speaks: Story, Questions, and Answers Method 0: Trial and Error Method 1: NER First Method 2: SRL First Next Steps Summary: Your Data Speaks: Story, Questions and Answers Quiz: Your Data Speaks: Story, Questions, and Answers Challenge: Q&A with Transformers Solution: Q&A with Transformers

Detecting Customer Emotions to Make Predictions

Introduction: Detecting Customer Emotions to Make Predictions Sentiment Analysis Transformers Predicting Customer Behavior with Sentiment Analysis Sentiment Analysis with Hugging Face Models Sentiment Analysis with GPT-3 Summary: Detecting Customer Emotions to Make Predictions Quiz: Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Introduction: Analyzing Fake News with Transformers Emotional Reaction to Fake News Behavioral Representation of Fake News A Rational Approach to Fake News A Political Approach to Fake News Summary: Analyzing Fake News with Transformers Quiz: Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

Introduction: Interpreting Black Box Transformer Models Transformer Visualization with BertViz LIT and PCA Transformer Visualization via Dictionary Learning Summary: Interpreting Black Box Transformer Models Quiz: Interpreting Black Box Transformer Models Challenge: Visualize and Complete Sentences Solution: Visualize and Complete Sentences

From NLP to Task-Agnostic Transfomer Models

Introduction: From NLP to Task-Agnostic Transformer Models Choosing a Model and an Ecosystem Vision Transformers The CLIP Model The DALL·E Model Summary: From NLP to Task-Agnostic Transformer Models Quiz: From NLP to Task-Agnostic Transformer Models

The Emergence of Transformer-Driven Copilots

Introduction: The Emergence of Transformer-Driven Copilots Prompt Engineering Training in Prompt Engineering Domain-Specific GPT-3 Engines Transformer-Based Recommender Systems Transformers Applied to Computer Vision Summary: The Emergence of Transformer-Driven Copilots Quiz: The Emergence of Transformer-Driven Copilots

Course Assessment

Transformers for NLP—Exam 2

Conclusion

Appendix

Terminology of Transformer Models Hardware Constraints for Transformer Models Generic Text Completion with GPT-2 Attaining OpenAI API Keys

Difficult Samples

Analyze some difficult samples with and GPT-3 and BERT SRL.

We'll cover the following

Sample 1
Sample 2
Sample 3
Code playground

Now let’s try some more difficult samples.

This lesson will run samples that contain problems that the BERT-based transformer will solve. We will end with an intractable sample.

Let’s start with a complex sample that the BERT-based transformer can analyze.

Sample 1

Sample 1 takes us into more tricky SRL territory. The sample separates Alice from the verb liked, creating a long-term dependency that has to jump over whose husband went jogging every Sunday.

The sentence is:

Get hands-on with 1200+ tech skills courses.