Training with an EMA (Exponential Moving Average)

The PyTorch Image Model framework supports an exponential moving average (EMA), which maintains moving averages of the trained variables by employing an exponential decay.

The implementation of an EMA is as follows:

1. Add shadow copies of trained weights during initialization.
2. Compute a moving average of the trained weights at each training step. It uses exponential decay for the computation.

$$weights = decay * weights + (1 - decay) * weights_{new}$$ ...