Text-to-Text Generation Products

Now that we have gone through the basics of the Transformer model, let’s dive deep into some amazing real-world applications of generative AI empowered by the transformers. There has been a tremendous increase in interest in building text-to-text-driven products. Imagine a language model that can be fine-tuned in a customized context and be able to work for questions and response-based tasks. Similarly, we can take a powerful language model with a given input sentence and generate a paragraph or even write an entire book. Such fine-tuned models have applications in writing product descriptions, code generation, sentiment analysis, personalized recommendations, and text summarization, to name just a few.

In this section, we will take a look at two commonly used model architectures in academia and the industry: DistilBERT for query-response tasks and GPT-2 for text generation-based tasks. As addressed before, the same models can also be fine-tuned for several other applications. The latest LLM products, such as ChatGPT by OpenAI, Claude by Anthropic, and Bard by Google, are mostly extensions of the same GPT-based model architecture with bigger parameters and huge data exposure.

DistilBERT

Bidirectional Encoder Representations from Transformers (BERT) is a commonly used model for natural language processing with more than 300 million parameters. Launched in 2018 by Google, it serves applications for several of the most common language tasks, such as sentiment analysis and named entity recognition. DistilBERT is a compressed version of BERT. It also uses the transformer architecture for efficient language understanding but only has around 60 million parameters. The BERT model relies on self-attention mechanisms to capture contextual word relationships that can mathematically be expressed through attention scores computed using softmax normalization: