Back To Course Home

Transformers for Natural Language Processing

0% completed

Course Introduction

Getting Started

Basics of Transformers

Introduction: Basics of Transformers Transformers and Industry 4.0 The History of Transformers Resources to Use Summary: Basics of Transformers Quiz: Basics of Transformers

Architecture of the Transformer Model

Introduction: Architecture of the Transformer Model The Rise of the Transformer: Attention Is All You Need Input Embedding Positional Encoding Sublayer 1: Multi-Head Attention Building Sublayer 1: Multi-Head Attention Post-Layer Normalization and Sublayer 2: Feedforward Network The Decoder Stack and Hugging Face Summary: Architecture of the Transformer Model Quiz: Architecture of the Transformer Model Challenge 1: Build a Multi-Head Attention Sublayer Solution: Build a Multi-Head Attention Sublayer Challenge 2: Evaluate Positional Encodings of a Sentence Solution: Evaluate Positional Encodings of a Sentence

Fine-Tuning BERT Models

Introduction: Fine-Tuning BERT Models Architecture of BERT Preparing the Pretraining Input Environment Pretraining and Setting Up a BERT Model Configuring and Training a BERT Model Evaluating a BERT Model Summary: Fine-Tuning BERT Models Quiz: Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Introduction: Pretraining a RoBERTa Model from Scratch Preliminary Steps Involved in Building KantaiBERT Main Steps Involved in Building KantaiBERT Summary: Pretraining a RoBERTa Model from Scratch Quiz: Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Introduction: Downstream NLP Tasks with Transformers Transduction and the Inductive Inheritance of Transformers Evaluating Models with Metrics Benchmark Tasks and Datasets SuperGLUE Benchmark Tasks Running Downstream Tasks Summary: Downstream NLP Tasks with Transformers Quiz: Downstream NLP Tasks with Transformers Challenge: Run Downstream Tasks Solution: Run Downstream Tasks

Machine Translation with the Transformer

Introduction: Machine Translation with the Transformer Types of Transductions and Translations Preprocessing Datasets Finalizing Data Preprocessing Evaluating Machine Translation with BLEU Implementing Translations Summary: Machine Translation with the Transformer Quiz: Machine Translation with the Transformer Challenge: Preprocess and Clean a Czech-English Dataset Solution: Preprocess and Clean a Czech-English Dataset

The Rise of Transformers with GPT-3 Engines

Introduction: The Rise of Transformers with GPT-3 Engines Architecture of OpenAI GPT Transformer Models Evolving From Fine-Tuning to Zero-Shot Models Generic Text Completion with GPT-2 Running OpenAI GPT-3 Tasks NLP Tasks and Examples Fine-Tuning GPT-3 The Role of an Industry 4.0 AI Specialist Summary: The Rise of Transformers with GPT-3 Engines Quiz: The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Introduction: Applying Transformers for AI Text Summarization Text to Text Transformer Models The T5 Model Text Summarization with T5 Summarizing Documents with T5-Large Summary: Applying Transformers for AI Text Summarization Quiz: Applying Transformers for AI Text Summarization Challenge: Summarize Efficient Solar Panel Breakthroughs Solution: Summarize Efficient Solar Panel Breakthroughs

Matching Tokenizers and Datasets

Introduction: Matching Tokenizers and Datasets

Word2Vec Tokenization

Standard NLP Tasks with Specific Vocabulary

Summary: Matching Tokenizers and Datasets

Quiz: Matching Tokenizers and Datasets

Course Assessment

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Introduction: Semantic Role Labeling with BERT-Based Transformers Getting Started with SRL Basic Samples Difficult Samples Questioning the Scope of SRL Summary: Semantic Role Labeling with BERT-Based Transformers Quiz: Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Introduction: Your Data Speaks: Story, Questions, and Answers Method 0: Trial and Error Method 1: NER First Method 2: SRL First Next Steps Summary: Your Data Speaks: Story, Questions and Answers Quiz: Your Data Speaks: Story, Questions, and Answers Challenge: Q&A with Transformers Solution: Q&A with Transformers

Detecting Customer Emotions to Make Predictions

Introduction: Detecting Customer Emotions to Make Predictions Sentiment Analysis Transformers Predicting Customer Behavior with Sentiment Analysis Sentiment Analysis with Hugging Face Models Sentiment Analysis with GPT-3 Summary: Detecting Customer Emotions to Make Predictions Quiz: Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Introduction: Analyzing Fake News with Transformers Emotional Reaction to Fake News Behavioral Representation of Fake News A Rational Approach to Fake News A Political Approach to Fake News Summary: Analyzing Fake News with Transformers Quiz: Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

Introduction: Interpreting Black Box Transformer Models Transformer Visualization with BertViz LIT and PCA Transformer Visualization via Dictionary Learning Summary: Interpreting Black Box Transformer Models Quiz: Interpreting Black Box Transformer Models Challenge: Visualize and Complete Sentences Solution: Visualize and Complete Sentences

From NLP to Task-Agnostic Transfomer Models

Introduction: From NLP to Task-Agnostic Transformer Models Choosing a Model and an Ecosystem Vision Transformers The CLIP Model The DALL·E Model Summary: From NLP to Task-Agnostic Transformer Models Quiz: From NLP to Task-Agnostic Transformer Models

The Emergence of Transformer-Driven Copilots

Introduction: The Emergence of Transformer-Driven Copilots Prompt Engineering Training in Prompt Engineering Domain-Specific GPT-3 Engines Transformer-Based Recommender Systems Transformers Applied to Computer Vision Summary: The Emergence of Transformer-Driven Copilots Quiz: The Emergence of Transformer-Driven Copilots

Course Assessment

Transformers for NLP—Exam 2

Conclusion

Appendix

Terminology of Transformer Models Hardware Constraints for Transformer Models Generic Text Completion with GPT-2 Attaining OpenAI API Keys

Summary: Matching Tokenizers and Datasets

Get a quick recap of what we covered in this chapter.

In this chapter, we measured the impact of the tokenization and subsequent data encoding process on transformer models. A transformer model can only attend to tokens from the embedding and positional encoding sublayers of a stack. It does not matter if the model is an encoder-decoder, encoder-only, or decoder-only model. It does not matter if the dataset seems good enough to train.

Get hands-on with 1400+ tech skills courses.