CycleGAN: Image-to-Image Translation from Unpaired Collections

We may have noticed that when training pix2pix, we need to determine a direction (A to B or B to A) that the images are translated to. Does this mean that if we want to freely translate from image set A to image set B and vice versa, we need to train two models separately?
Not with CycleGAN, we say!

CycleGANZhu, Jun-Yan, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired image-to-image translation using cycle-consistent adversarial networks." In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017. is a bidirectional generative model based on unpaired image collections. The core idea of CycleGAN is built on the assumption of cycle consistency, which means that if we have two generative models, $G$ and $F$, which translates between two sets of images, $X$ and $Y$, in which $Y=G(X)$ and $X=F(Y)$, we can naturally assume that $F(G(X))$ should be very similar to $X$ and $G(F(Y))$ should be very similar to $Y$. This means that we can train two sets of generative models at the same time that can freely translate between two sets of images.

CycleGAN is specifically designed for unpaired image collections, which means that the training samples are not necessarily strictly paired like they were in pix2pix and pix2pixHD (for example, semantic segmentation maps vs. street views from the same perspective or regular maps vs. satellite photos of the same location). This makes CycleGAN more than just an image-to-image translation tool. It unlocks the potential to transfer style from any kind of image to our own images, for example, turning apples into oranges, horses into zebras, photos into oil paintings, and vice versa. Here, we will perform image-to-image translation on landscape photos and Vincent van Gogh’s paintings as an example to show us how CycleGAN is designed and trained.