Transformer Visualization with BertViz

Learn about how layers of a transformer model are visualized with BertViz.

We'll cover the following

Running BertViz
Code playground

Jesse Vig’s article, "A Multiscale Visualization of Attention in the Transformer ModelThe paper can be accessed at: https://aclanthology.org/P19-3007.pdf" (2019), recognizes the effectiveness of transformer models. However, Jesse Vig explains that deciphering the attention mechanism is challenging. The paper describes the process of BertViz, a visualization tool.

BertViz can visualize attention head activity and interpret a transformer model’s behavior.

BertViz was first designed to visualize BERT and GPT-3 models. In this lesson, we will visualize the activity of a BERT model.

Let’s now run BertViz.

Running BertViz

It only takes five steps to visualize transformer attention heads and interact with them.

Step 1: Importing BertViz and other modules

The notebook BertViz.ipynb imports bertViz, Hugging Face transformers, and the other basic requirements to implement the program:

Get hands-on with 1400+ tech skills courses.

Course Introduction

Basics of Transformers

Architecture of the Transformer Model

Fine-Tuning BERT Models

Pretraining a RoBERTa Model from Scratch

Downstream NLP Tasks with Transformers

Machine Translation with the Transformer

The Rise of Transformers with GPT-3 Engines

Applying Transformers for AI Text Summarization

Matching Tokenizers and Datasets

Transformers for NLP—Exam 1

Semantic Role Labeling with BERT-Based Transformers

Your Data Speaks: Story, Questions, and Answers

Detecting Customer Emotions to Make Predictions

Analyzing Fake News with Transformers

Interpreting Black Box Transformer Models

From NLP to Task-Agnostic Transfomer Models

The Emergence of Transformer-Driven Copilots

Transformers for NLP—Exam 2

Conclusion

Appendix

Transformer Visualization with BertViz

Running BertViz

Step 1: Importing BertViz and other modules