Summary: Applying Transformers for AI Text Summarization

In this chapter, we saw how the T5 transformer models standardized the input of the encoder and decoder stacks of the original transformer. The original transformer architecture has an identical structure for each block (or layer) of the encoder and decoder stacks. However, the original transformer did not have a standardized input format for NLP tasks.

We then implemented a T5 model that could summarize any text. We tested the model on texts that were not part of ready-to-use training datasets. We tested the model on constitutional and corporate samples. The results were interesting, but we also discovered some of the limits of transformer models, as predicted by Raffel et al. (2018).