The Machine Learning Pipeline for Image Caption Generation

Here, we’ll look at the image caption generation pipeline at a very high level and then discuss it piece by piece until we have the full model. The image caption generation framework consists of two main components:

• A pretrained vision transformer model to produce an image representation.

• A text-based decoder model that can decode the image representation to a series of token IDs. This uses a text tokenizer to convert tokens to token IDs and vice versa.

Though the transformer models were initially used for text-based NLP problems, they have outgrown the domain of text data and have been used in other areas, such as image data and audio data.

Here, we’ll be using one transformer model that can process image data and another that can process text data.

Vision transformer (ViT)

First, let’s look ...