Paired Style Transfer Using pix2pix GAN

Style transfer is an intriguing research area that pushes the boundaries of creativity and deep learning together. In their work, “Image-to-Image Translation with Conditional Adversarial NetworksIsola, P., Zhu, J-Y., Zhou, T., & Efros, A.A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5967-5976. https://ieeexplore.ieee.org/document/8100115,” Isola and Zhu et al. present a conditional GAN network that is able to learn task-specific loss functions and work across datasets. As the name suggests, this GAN architecture takes a specific type of image as input and transforms it into a different domain.

It is called pair-wise style transfer, as the training set needs to have matching samples from both source and target domains. This generic approach is shown to effectively synthesize high-quality images from label maps and edge maps, and even colorize images. The authors highlight the importance of developing an architecture capable of understanding the dataset at hand and learning mapping functions without the need for hand-engineering (which has been the case typically).

The U-Net generator

CNNs are optimized for computer vision tasks, using them for the generator as well as discriminator architectures has a number of advantages. This work focuses on two related architectures for the generator setup. The two choices are the vanilla encoder-decoder architecture and the encoder-decoder architecture with skip connections. The architecture with skip connections has more in common with the U-Net modelRonneberger, O., Fisher, P., & Brox, T. (2015). U-net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 2015. https://arxiv.org/abs/1505.04597 than the encoder-decoder setup. Hence, the generator in the pix2pix GAN is termed a U-Net generator (see the figure below for reference).