BERT Models for Italian and Portuguese
Learn about the architecture and different variants of the UmBERTo and BERTimbau.
We'll cover the following...
UmBERTo for Italian
UmBERTo is the pre-trained BERT model for the Italian language by Musixmatch research. The UmBERTo model inherits the RoBERTa model architecture. The RoBERTa is essentially BERT with the following changes in pre-training:
Dynamic masking is used instead of static masking in the MLM task.
The NSP task is removed and trained using only ...