Contrastive Learning: The SimCLR Algorithm

What is contrastive learning?

The objective of contrastive learning is to learn neural network embeddings such that embeddings from related images should be closer than embeddings from unrelated or dissimilar images. So, given an image $X_a$, we can call it an “anchor image” and define two terms:

• Positives: Images that are closely related to the anchor image, $X_a$. Let’s represent them by $X_p$

• Negatives: Images that are unrelated or dissimilar to $X_a$. Let’s call them $X_n$.

The contrastive learning objective thus learns a neural network $f(.)$ such that:

Here, $\text{sim}(. \ ,\ .)$ is any similarity measure (e.g., cosine similarity). Therefore, we need to maximize the similarity between positive pairs and minimize the similarity between negative pairs. This is illustrated in the figure below.

Contrastive loss

To enforce the contrastive property, we can define a contrastive loss function as follows:

Here, $f_a = f(X_a)$, $f_p=f(X_p)$ ...