Generator and Discriminator of the ELECTRA Model

The generator model

First, let's have a look at the generator. The generator performs the MLM task. We randomly mask a few tokens with a 15% mask rate and train the generator to predict the masked token. Let's represent our input tokens as $X = [x_1, x_2, ..., x_n]$. We randomly mask some tokens and feed them as input to the generator, which returns the representation of each of the tokens. Let $h_G (X) = [h_1, h_2, ..., h_n]$denote the representation of each token returned by the generator. That is, $h_1$ denotes the representation of the first token $x_1$, $h_2$ denotes the representation of the second token $x_2$, and so on.