Predicting Relative Position of Patches

Relative positioning of patches

Given an image $X_i$, this pretext task involves sampling a random pair of patches $(X_i^{p_1}, X_i^{p_2})$ in one of the eight spatial configurations (shown in figure below) and assigning a pseudo label $P_i$ (shown in figure below) that denotes the position of patch $X_i^{p_2}$ relative to patch $X_i^{p_1}$. The neural network, $f(.)$ thus learns to predict $P_i$ given $(X_i^{p_1}, X_i^{p_2})$ as input (i.e, $f((X_i^{p_1}, X_i^{p_2})) = P_i$). This ensures that the model extracts and understands the relative spatial arrangement of relevant objects in an image useful for recognition.

The figure below illustrates how a training example is generated. We first sample a center patch $X_i^{p_1}$ (shown in blue) uniformly within the image bounds. After that, the second patch $X_i^{p_2}$ is sampled randomly in one of the eight spatial configurations (shown in dotted red) relative to the center patch $X_i^{p_1}$.