DistilBERT
Learn about the DistilBERT variant of BERT.
We'll cover the following...
The pre-trained BERT model has a large number of parameters and also high inference time, which makes it harder to use on edge devices such as mobile phones. To solve this issue, we use DistilBERT—the distilled version of BERT—which was introduced by researchers at Hugging Face. DistilBERT is a smaller, faster, cheaper, and lighter version of BERT.
Press + to interact
As the name suggests, DistilBERT uses knowledge distillation. The ultimate idea of DistilBERT is that we take a large pre-trained BERT model and transfer its knowledge to a small BERT through knowledge distillation. The large pre-trained BERT is called a teacher BERT ...