What are multimodal generative models?

Multimodal generative models are a specific type of machine learning model that generates a variety of outputs from several modalities. The goal of these models is to capture the underlying connections and dependencies between several modalities in training data. Therefore, they might generate samples with features from other domains, such as images, text, or sound.

Note: Please check out this Answer for more information on multimodal learning and modality.

Working mechanism

Multimodal generative models work by learning joint representations of data from different modalities and using these representations to generate diverse and coherent outputs across modalities. Here’s a general overview of how multimodal generative models work:

Data representation

Input modalities: The model needs to encode the inputs into a shared representation space if the input modalities are, for instance, text and images. This is frequently accomplished by employing different encoders for each modality.

Shared representation learning

Latent space: The model aims to learn a shared latent space where representations from different modalities are close to each other. This enables the model to represent fundamental connections and dependencies between modalities.

Generative process

Decoders: After learning the common representation, the model generates outputs using decoders specific to each modality. These decoders produce outputs in the appropriate modalities using samples from the common latent space.

Training:
- Adversarial training: In order to train the model to generate outputs that are identical to real data in each modality, several multimodal generative models utilize adversarial training. This helps the model to provide realistic samples throughout modalities.
- Cycle-consistency: Certain models, such as MUNIT, use cycle consistency loss to make sure that the outputs that are produced can be traced back to the original input space, which aids in maintaining the style and content.
Inference:
- Sampling: Using suitable decoders and sampling from the learned latent space, the model can provide coherent and varied samples in other modalities given input in one modality during inference.
Application:
- Cross-modal tasks: The trained model may be utilized for image captioning, text-to-image synthesis, style transfer, and any other application that requires several modalities.

Famous multimodal generative models

Multimodal generative models come in a variety of forms, each with a special method for generating data. Some well-known multimodal generative models are as follows:

CLIP (Contrastive Language-Image Pre-training): OpenAI created the transformer-based model CLIP to help people learn combined representations of images and text. It can be used for a number of tasks like image classification, identifying objects, and creating textual descriptions for images.
VQ-VAE-2 (Vector Quantized Variational Autoencoder 2): The multimodal capabilities of the VQ-VAE-2 can be used to create diversified and high-quality material.
MUNIT (Multimodal Unsupervised Image-to-Image Translation): MUNIT is an unsupervised image-to-image translation model that works across several modalities. To demonstrate its capacity to modify images across multiple visual domains, it can, for example, transform images of horses into zebras without the need for associated data.
UNIT (Unsupervised Image-to-Image Translation Networks): UNIT is an image-to-image translation mechanism that works similarly to MUNIT. For example, it might transform satellite images of urban areas to look like they were taken at night without depending on associated simulations.
DALL·E: OpenAI created a multimodal generative model called DALL·E which is capable of generating images from textual descriptions.

Code example

The code below uses numpy to construct a basic MUNIT-like Multimodal Generative Model (a generator) with randomly initialized encoder and decoder weights. It prints the size of the generated image after feeding a randomly generated input vector through the generator to generate a synthetic image:

import numpy as np
import matplotlib.pyplot as plt
# MUNIT-like Multimodal Generative Model
class MUNITGenerator:
    def __init__(self, input_dim, output_dim):
        self.encoder_weights = np.random.rand(512, input_dim)
        self.decoder_weights = np.random.rand(output_dim, 512)
    def forward(self, x):
        x = np.dot(self.encoder_weights, x)
        x = np.maximum(0, x)
        x = np.dot(self.decoder_weights, x)
        return x
input_dim = 100
output_dim = 3 * 64 * 64
generator = MUNITGenerator(input_dim, output_dim)
input_vector = np.random.randn(input_dim, 1)
output_image = generator.forward(input_vector)
print("Generated Image Size:", output_image.shape)
generated_image = output_image.reshape(3, 64, 64)
flattened_image = generated_image.flatten()
# Create a histogram of pixel values
plt.hist(flattened_image, bins=50, color='blue', alpha=0.7)
plt.title("Histogram of Pixel Values")
plt.xlabel("Pixel Value")
plt.ylabel("Frequency")
plt.savefig("output/syn.png", dpi=300)

Code explanation

Here is the explanation of the above code:

Lines 1–2: Imports necessary libraries.
Lines 4–13: Defines the MUNITGenerator class:
- The generator has an encoder and a decoder, each represented by randomly initialized weights.
- The forward method performs a forward pass through the generator, transforming an input vector x into an output vector.
Lines 15–16: Sets the hyperparameters: input_dim and output_dim.
Line 17: Instantiates the MUNIT-like MUNITGenerator.
Line 18–19: Generate a random input_vector and use the generator to produce an output_image.
Line 21: Displays the generated image size.
Line 22: Reshape the output_image to match the image dimensions (3 channels, 64x64 pixels).
Line 23: Flattens the pixel values of the generated image.
Lines 26–30: Creates a histogram of pixel values using Matplotlib and saves the histogram plot as .png file.
Expected output: This code generates a synthetic image using a MUNIT-like generator and then creates a histogram of pixel values in the generated image.

Note: The shape of the histogram is likely to vary each time we run the code. The randomness in the MUNIT-like generator comes from the initialization of the encoder and decoder weights with random values. These weights influence the transformation applied to the input vector during the forward pass, so the output image will differ for each run.

Applications

Multimodal generative models have applications in a wide range of disciplines where input from several modalities may be used to improve data interpretation, creation, or manipulation. The following applications are important to know:

Image captioning: To generate useful captions for images, multimodal generative models might be utilized. The model takes an image as input and integrates visual and textual data to build an adequate and coherent caption.
Audio-visual scene perception: These models, which combine both audio and visual modalities, can be used for tasks like scene comprehension.
Medical imaging: Data from multiple imaging modalities (e.g., MRI, CT, PET) can be fused to increase diagnosis accuracy or provide more meaningful representations using multimodal models.
Robotics and autonomous systems: By processing input from multiple sensors such as cameras, microphones, and other sensors, multimodal generative models can help robots and autonomous systems comprehend and interact with their environment.
Social media analysis: Analyzing and generating material for social media platforms requires using numerous modalities, such as images, text, and audio, to give a more comprehensive understanding of user-generated content.

Test your understanding

Assess your comprehension of multimodal generative models in this engaging quiz.

Free Resources

Learn in-demand tech skills in half the time

PRODUCTS

Mock Interview

New

Courses

Skill Paths

Projects

Assessments