Back To Course Home

Transformers for Natural Language Processing

0% completed

Course Introduction

Getting Started

Basics of Transformers

Introduction: Basics of Transformers Transformers and Industry 4.0 The History of Transformers Resources to Use Summary: Basics of Transformers Quiz: Basics of Transformers

Architecture of the Transformer Model

Introduction: Architecture of the Transformer Model The Rise of the Transformer: Attention Is All You Need Input Embedding Positional Encoding Sublayer 1: Multi-Head Attention Building Sublayer 1: Multi-Head Attention Post-Layer Normalization and Sublayer 2: Feedforward Network The Decoder Stack and Hugging Face Summary: Architecture of the Transformer Model Quiz: Architecture of the Transformer Model Challenge 1: Build a Multi-Head Attention Sublayer Solution: Build a Multi-Head Attention Sublayer Challenge 2: Evaluate Positional Encodings of a Sentence Solution: Evaluate Positional Encodings of a Sentence

Fine-Tuning BERT Models

Introduction: Fine-Tuning BERT Models Architecture of BERT Preparing the Pretraining Input Environment Pretraining and Setting Up a BERT Model Configuring and Training a BERT Model Evaluating a BERT Model Summary: Fine-Tuning BERT Models Quiz: Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Introduction: Pretraining a RoBERTa Model from Scratch Preliminary Steps Involved in Building KantaiBERT Main Steps Involved in Building KantaiBERT Summary: Pretraining a RoBERTa Model from Scratch Quiz: Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Introduction: Downstream NLP Tasks with Transformers Transduction and the Inductive Inheritance of Transformers Evaluating Models with Metrics Benchmark Tasks and Datasets SuperGLUE Benchmark Tasks Running Downstream Tasks Summary: Downstream NLP Tasks with Transformers Quiz: Downstream NLP Tasks with Transformers Challenge: Run Downstream Tasks Solution: Run Downstream Tasks

Machine Translation with the Transformer

Introduction: Machine Translation with the Transformer Types of Transductions and Translations Preprocessing Datasets Finalizing Data Preprocessing Evaluating Machine Translation with BLEU Implementing Translations Summary: Machine Translation with the Transformer Quiz: Machine Translation with the Transformer Challenge: Preprocess and Clean a Czech-English Dataset Solution: Preprocess and Clean a Czech-English Dataset

The Rise of Transformers with GPT-3 Engines

Introduction: The Rise of Transformers with GPT-3 Engines Architecture of OpenAI GPT Transformer Models Evolving From Fine-Tuning to Zero-Shot Models Generic Text Completion with GPT-2 Running OpenAI GPT-3 Tasks NLP Tasks and Examples Fine-Tuning GPT-3 The Role of an Industry 4.0 AI Specialist Summary: The Rise of Transformers with GPT-3 Engines Quiz: The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Introduction: Applying Transformers for AI Text Summarization Text to Text Transformer Models The T5 Model Text Summarization with T5 Summarizing Documents with T5-Large Summary: Applying Transformers for AI Text Summarization Quiz: Applying Transformers for AI Text Summarization Challenge: Summarize Efficient Solar Panel Breakthroughs Solution: Summarize Efficient Solar Panel Breakthroughs

Matching Tokenizers and Datasets

Introduction: Matching Tokenizers and Datasets Best Practices Word2Vec Tokenization Standard NLP Tasks with Specific Vocabulary Summary: Matching Tokenizers and Datasets Quiz: Matching Tokenizers and Datasets

Course Assessment

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Introduction: Semantic Role Labeling with BERT-Based Transformers

Getting Started with SRL

Difficult Samples

Questioning the Scope of SRL

Summary: Semantic Role Labeling with BERT-Based Transformers

Quiz: Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Introduction: Your Data Speaks: Story, Questions, and Answers Method 0: Trial and Error Method 1: NER First Method 2: SRL First Next Steps Summary: Your Data Speaks: Story, Questions and Answers Quiz: Your Data Speaks: Story, Questions, and Answers Challenge: Q&A with Transformers Solution: Q&A with Transformers

Detecting Customer Emotions to Make Predictions

Introduction: Detecting Customer Emotions to Make Predictions Sentiment Analysis Transformers Predicting Customer Behavior with Sentiment Analysis Sentiment Analysis with Hugging Face Models Sentiment Analysis with GPT-3 Summary: Detecting Customer Emotions to Make Predictions Quiz: Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Introduction: Analyzing Fake News with Transformers Emotional Reaction to Fake News Behavioral Representation of Fake News A Rational Approach to Fake News A Political Approach to Fake News Summary: Analyzing Fake News with Transformers Quiz: Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

Introduction: Interpreting Black Box Transformer Models Transformer Visualization with BertViz LIT and PCA Transformer Visualization via Dictionary Learning Summary: Interpreting Black Box Transformer Models Quiz: Interpreting Black Box Transformer Models Challenge: Visualize and Complete Sentences Solution: Visualize and Complete Sentences

From NLP to Task-Agnostic Transfomer Models

Introduction: From NLP to Task-Agnostic Transformer Models Choosing a Model and an Ecosystem Vision Transformers The CLIP Model The DALL·E Model Summary: From NLP to Task-Agnostic Transformer Models Quiz: From NLP to Task-Agnostic Transformer Models

The Emergence of Transformer-Driven Copilots

Introduction: The Emergence of Transformer-Driven Copilots Prompt Engineering Training in Prompt Engineering Domain-Specific GPT-3 Engines Transformer-Based Recommender Systems Transformers Applied to Computer Vision Summary: The Emergence of Transformer-Driven Copilots Quiz: The Emergence of Transformer-Driven Copilots

Course Assessment

Transformers for NLP—Exam 2

Conclusion

Appendix

Terminology of Transformer Models Hardware Constraints for Transformer Models Generic Text Completion with GPT-2 Attaining OpenAI API Keys

Basic Samples

Analyze some basic samples with GPT-3 and BERT SRL.

We'll cover the following

Sample 1
- Generating SRL labels
Sample 2
Sample 3
Code playground

We are now ready to warm up with some basic samples.

Basic samples seem intuitively simple but can be tricky to analyze. Compound sentences, adjectives, adverbs, and modals can be difficult to identify.

Sample 1

The first sample is long but relatively easy for the transformer:

Get hands-on with 1400+ tech skills courses.