Bidirectional Transformers for Language Understanding

In the last lesson, we discussed how self-attention allowed every word in a sentence to interact directly with every other word, breaking free from the old sequential constraints of RNNs and LSTMs. Yet, as powerful as that was, something was still missing. Imagine transformers as a brilliant highlighter that points out the most important parts of a text—but it couldn’t capture the subtle interplay of context in both directions.

Researchers have already begun improving language models using deep contextual embeddings. ELMo (2018) introduced context-aware word vectors via LSTMs but still relied on sequential processing. Around the same time, OpenAI explored a unidirectional transformer for text generation (GPT), which excelled at producing fluent text but didn’t capture context from both sides of a sentence in a single pass.

In 2018, Google AI introduced BERT (Bidirectional encoder representations from transformers), sparking a revolution in how machines understand language. Instead of reading text only from left to right or right to left, BERT reads in both directions simultaneously. A picture can look at a sentence as a whole, understanding every word about all its neighbors at once. That is the magic of BERT: it sees the complete picture rather than a one-sided snapshot. Thanks to that bidirectional view, BERT became one of the first large language models to deeply grasp the nuanced meaning in the text.

What is BERT?

So, what exactly is BERT? At its core, BERT is an architecture built on the same transformer encoder we’ve discussed—but with one crucial twist: bidirectional. Like some early language models, traditional models would process text in one direction (imagine reading a sentence word by word from left to right). BERT, however, takes in the entire sentence at once, considering both the words that come before and after any given word. This approach allows it to capture subtleties and context that unidirectional models might miss.

In simple terms, BERT is like a super attentive reader. When it sees a sentence, it doesn’t just think, what’s the next word? Instead, it asks, what does every word in this sentence ...