Qualitative Methods

The evaluation of GANs with qualitative methods focuses on exploratory data analysis. In such methods, the researcher evaluates the fake samples by visual inspection. This can be done independently from other samples or with respect to real samples. Qualitative methods are useful as they can quickly provide information about issues with your current GAN experiment related to image quality, image variety, and the violation of specifications.

In GAN literature, the visual inspection of samples is a very common practice, and authors use it to quickly confirm that they have not observed mode collapse or that their framework is robust enough to avoid mode collapse if some criteria are met (Arjovsky et al., 2017; Gulrajaniet al., 2017; Mao et al., 2016; and Radford et al., 2015).

Qualitative methods for evaluation are very useful to quickly detect problems with fake data. This allows us to quickly modify our experiment to achieve better results. This qualitative evaluation, however, comes at the price of being superficial and not systematic. For a deeper and more systematic evaluation of fake samples, it is necessary to devise a systematic, quantitative evaluation.

In the following sections, we provide some qualitative measures for GAN evaluation, as elaborated by Ali Borji in his paper, “Pros and Cons of GAN Evaluation Measures.”

$k$-nearest neighbors

Generally speaking, the $k$-nearest neighbors algorithm is a non-parametric technique that, given samples $n$ in a dataset $(\mathcal{N})$, reference samples $(r)$, and some similarity measures $(S)$, $k$-nearest neighbors finds the $k$ samples in $\mathcal{N}$ most similar to $r$, given $S$. Assuming that samples are represented as points in some space, the closest point to a reference point can be thought of as its neighbor.

Mathematically speaking, $k$-nearest neighbors can be defined as follows: