The evolution of GPTs

This blog is an attempt to unfold the historical developments within computational linguistics that helped give birth to ChatGPT. Like us humans, do computers also need to know what words mean?

How machines understand words#

Fluent speakers possess vast knowledge, primarily reflected in their vocabulary. This knowledge includes the grammatical function, meaning, real-world reference, and pragmatic use of words. While estimates of adult vocabulary size vary, it is agreed that most words used by mature speakers are acquired early in life through spoken interactions. This active vocabulary remains limited compared to the adult vocabulary, leaving many words to be acquired through other means. Children achieve remarkable vocabulary growth rates by learning approximately 7 to 10 words daily, with reading playing a significant role in this process. A renowned principle within linguistics, known as the distributional hypothesis, suggests that word meanings can be learned from text alone, based on the associations between words and their co-occurring words. This happens because synonymous words often appear in similar contexts or alongside similar words in written text. A class of machine learning models called language models has proven effective in capturing this type of knowledge from vast amounts of text. These language models can be utilized for various natural language processing tasks, including sentiment analysis, language translations, text summarization, chatbot Q&A, text generation, etc.

This blog will center around the basics of language models that use a specific architecture and provide a foundation for understanding the much-hyped GPT (Generative Pre-trained Transformer). Subsequently, we will explore the evolution of GPT models as it happened over the years.

Language models#

In the context of natural language processing (NLP), a language model is a computational model designed to understand and generate human language. It is trained on a large corpus of text data and learns the language’s statistical patterns, relationships, and structures. The main goal of a language model is to predict the probability of the next word in a sequence of words, given the previous words. In particular, if $w_1,w_2,\dots,w_n$ is a sequence of $n$, words, then probability of the next world, $w_{n+1}$, using Bayes rule, can be defined as follows:

$$P(w_{n+1}|w_n,w_{n-1},\dots,w_1)=\frac{P(w_{n+1},w_n,w_{n-1},\dots,w_1)}{P(w_n,w_{n-1},\dots,w_1)}$$

Text generation#

Once the language model is trained in this way on a text corpus, it gains the ability to sample from the learned probability distribution. This sampling process enables the language model to generate coherent and contextually relevant text, as it can draw upon its learned knowledge of language patterns and structures.

But wait a minute!#

Before delving further into the details of how language models generate text, some questions need to be asked:

• How is a word represented in a language model?
• How is the estimation of probabilities conducted using prepared training data?
• Which models are used that are insensitive to the length of the sequence?
• How is the generation of the next word accomplished, and what does the sampling process entail?

Let’s address these questions individually and provide a comprehensive explanation for each one.

Word embeddings#

In computational models, words are typically represented using numerical vectors or embeddings, instead of strings or characters. These embeddings capture the semantic and syntactic properties of words, allowing them to be processed mathematically.

Learning the optimal word embedding representation can be treated as a standalone task, but more commonly, it is learned concurrently with the estimation of the probability model for the problem being solved.

Fun fact: Word2Vec is a popular word embedding technique developed by Google.

Buy one, get one free!#

Let’s consider a word $w_i$ that is represented by a vector of $d$ numbers:

$$w_i = (v_1,v_2,\dots,v_d)$$

In this representation, $v_1, v_2, \dots, v_d$ are additional learnable parameters that are updated alongside the parameters of the probability model. These embedding parameters are adjusted to enhance the accuracy of predictions. Typically, the parameters are initialized randomly unless a more sophisticated initialization method is employed.

The probability model#

The probability model used for predicting the next word needs to possess two essential capabilities:

• It should take into account all the words in the sequence that have been processed thus far as input.
• It should generate the probability of each potential next word as output.

While various models can be employed in theory, the most commonly used approach is to utilize recurrent functions. These functions process one word at a time, incorporating information from all previously processed words (history) as input. In doing so, they provide information about all the words processed so far and a probability distribution for the next word.

The initial history for the first word can be arbitrarily assigned or set randomly.

Example#

Let’s look at the following code that essentially builds a simple language model that predicts the next word in a sequence of words based on the history of words. This code implements a basic language model for predicting the next word in a sentence. It employs a small vocabulary of words such as “yes,” “no,” and “maybe,” each represented by numerical embeddings. The model uses two crucial matrices, $W_h$ and $W_x$, as learnable parameters. These matrices are used to assign weights to the historical context and the current word when making predictions. The probability_model function calculates the probabilities of the next word being “yes”, “no”, or “maybe” based on the input history and the current word. It then demonstrates the model’s usage by iteratively updating the history as it predicts the next word in a given sequence, essentially simulating a simple language prediction task.

The beauty of recurrent functions is that they can seamlessly process sequences of words of arbitrary lengths without any hindrance. Thus, we can set the length of the word sequence as desired, and the recurrent function will handle it smoothly. Feel free to experiment with the seq list in the provided code. However, it is important to note that the simplicity of the above recurrent function may not adequately capture the complex statistical patterns found in real-world language. To enhance the model’s representational capacity, one approach is to model it as a neural network. This is where the true strength of recurrent neural networks begins to emerge.

When a recurrent function is implemented using a neural network, it is referred to as a recurrent neural network (RNN).

The following code is a template of a recurrent neural network in the context of language modeling. Notice the similarity with the function defined earlier.

def probability_model(history, current_word):
    h = history
    x = embeddings[current_word]

    history_next, probabilities = neural_network(h,x)

    return history_next, probabilities

Generation using RNNs#

The model’s parameters are optimized to make accurate predictions, enabling the generation of next-word sequences. This process involves substituting the current word in each subsequent function call with the word that was predicted to be the best choice in the previous call. The figure below provides a visual representation and further explanation of this concept.

Sampling the next word#

The generation of the next word involves sampling from the learned probability distribution. The language model assigns probabilities to each potential next word given the preceding words in the sequence. Sampling can be done using various methods, such as greedy sampling (selecting the word with the highest probability), random selection of the next word, or advanced techniques like temperature-based sampling or beam search, which help control the diversity or quality of the generated text.

When using random or temperature-based sampling, it is possible to obtain different next words for the same history and current word on multiple occasions. Have you had the opportunity to try out ChatGPT yourself?

Transformers#

Despite being well-suited for language models, RNNs face two primary challenges. First, they rely on sequential processing through time, and second, they struggle with managing long-term dependencies in historical data.

Transformers, conversely, can process input sequences in parallel, whereas RNNs operate sequentially. This parallelism enables transformers to handle long-range dependencies more effectively and significantly speeds up computation, making them more efficient for training and inference.

While transformers offer significant advantages in practice, particularly in language modeling, it’s worth noting that RNNs still excel in certain scenarios. The choice between transformers and RNNs depends on the specific requirements and characteristics of the problem at hand.

Transformers are designed to transform sequences of input embeddings $(\bold{x_1, ..., x_n})$ into sequences of transformed embeddings $(\bold{y_1, ..., y_n})$ with the same length.

Self-attention#

The essence of the transformer architecture lies in its utilization of self-attention, which plays a crucial role in the entire process. The concept revolves around modeling the history of the current word in a more advanced manner. Specifically, certain words within the current word’s history may carry greater significance than others when generating the next word. Self-attention achieves this by assigning weights to all the words in the history, ensuring that important words receive higher weights. It is typical to rescale the weights to sum up to 1, and all weights are non-negative.

How self-attention works#

Consider a sequence of input embeddings $(\bold x_1, \bold x_2, \dots, \bold x_n)$ where each embedding $\bold x_i$ has $d$ components, i.e., $\bold x_i \in \mathbb{R}^d$ for all $i$. Let’s assume that the current word’s embedding is $\bold x_i$, and we aim to transform it to obtain $\bold y_i$. Using parameter matrices $W_q, W_k,$ and $W_v$, we transform all the input embeddings up to the current word’s embedding as follows:

$$\begin{align*} \bold q_j &= W_q\bold x_j \\ \bold k_j &= W_k\bold x_j \\ \bold v_j &= W_v\bold x_j \end{align*}$$

The weight $a_{ij}$ can then be defined as follows:

$$a_{ij}=\bold q_j^T\bold k_i$$

Finally, $\bold y_i$ can be estimated as a linear combination of the transformed embeddings:

$$\bold y_i = \sum_{j=1}^ia_{ij}\bold v_j$$

The parameter matrices, $W_q, W_k,$ and $W_v$, are learned during the training process, allowing the weight $a_{ij}$ to be learned as well. Here, $a_{ij}$ represents the importance or attention given to the $j^{th}$ element when processing the $i^{th}$ element in the sequence. It quantifies the relevance or contribution of other elements to the current element’s representation, making self-attention a powerful mechanism for capturing dependencies and relationships within a sequence of data, such as in natural language processing tasks.

Positional encoding#

It is important to note that shuffling the input embeddings would yield the same transformations in self-attention due to the properties of linear combinations. However, this shuffling disregards the sequential nature of the words and fails to utilize their inherent order. To preserve and incorporate sequential information, it is essential to include positional information within each word’s corresponding input embedding. By incorporating positional encoding, the model becomes aware of the relative positions of words in the sequence, enabling it to leverage and utilize the sequential relationships for more accurate processing and understanding of the input.

Training a transformer#

The training process of a transformer is fundamentally similar to that of RNNs. However, with the inclusion of self-attention, the key distinction lies in the parallel nature of computations. Unlike RNNs, which process sequential input sequentially, transformers can perform computations in parallel, allowing for more efficient training.

Once trained, generating the next word in a transformer follows a similar approach as in RNNs. The process involves sampling the next word and using it as the current word for the subsequent time step. By iteratively generating words in this manner, the transformer produces a sequence of words that can extend beyond the training data.

Generative Pre-trained Transformer (GPT)#

In the context of transformers, generative refers to the ability of the model to generate new content, such as text, based on its understanding of the patterns and structure in the training data. Generative models aim to produce outputs that resemble and extend beyond the data they were trained on.

In the case of transformer models like GPT (Generative Pre-trained Transformer), the term “generative” indicates that the model is capable of generating coherent and contextually relevant text. By leveraging its learned knowledge of language patterns and relationships, a generative transformer can generate sequences of words that are meaningful and resemble human-like language.

Generative transformers have shown impressive capabilities in various natural language processing tasks, including text completion, text generation, machine translation, and more. They have the ability to generate novel and contextually appropriate responses, making them valuable tools in applications such as chatbots, content generation, and creative writing assistance.