BYOL Training

Student and teacher architectures

The student and teacher network in BYOL follows the same backbone architecture. However, the student network uses an additional MLP prediction head, $p(.)$, to ensure asymmetry in the overall student-teacher architecture. In other words, in $f^{\text{teacher}} = g \circ h$ and $f^{\text{student}} = p \circ g \circ h$, $h(.)$ is the feature extractor, $g(.)$ is the linear projection layer, and $p(.)$ is the prediction head.