Distillation of Embedding and Prediction Layer

Embedding layer distillation

In embedding layer distillation, we transfer knowledge from the embedding layer of the teacher to the embedding layer of the student. Let $E^S$ denote the embedding of the student and $E^T$ denote the embedding of the teacher, then we train the network to perform embedding layer distillation by minimizing the mean squared error between the embedding of student $E^S$ and teacher $E^T$, as shown in the following: