DistilBERT

The pre-trained BERT model has a large number of parameters and also high inference time, which makes it harder to use on edge devices such as mobile phones. To solve this issue, we use DistilBERT—the distilled version of BERT—which was introduced by researchers at Hugging Face. DistilBERT is a smaller, faster, cheaper, and lighter version of BERT.