Training the ELECTRA Model

The generator is trained using the MLM task.

Selecting the masking position

So, for a given input, $X = [x_1, x_2, ..., x_n]$, we randomly select a few positions for masking. Let $M = [m_1, m_2, ..., m_n]$ denote the selected positions for masking.

Replacing the selected positions with the [MASK] token

Then, we replace the tokens in the selected positions with the [MASK] token, which can be expressed as follows: