Unsupervised Cross-lingual Representation Learning at Scale
This paper shows that pretraining multilingual
language models at scale leads to significant
performance gains for a wide range of cross-
lingual transfer tasks. We train a Transformer-
based masked language model on one hundred
languages, using more than two terabytes of fil-
tered CommonCrawl data. Our model, dubbed
XLM-R, significantly outperforms multilingual
BERT (mBERT) on a variety of cross-lingual
benchmarks, including +14.6% average accu-
racy on XNLI, +13% average F1 score on
MLQA, and +2.4% F1 score on NER. XLM-R
performs particularly well on low-resource lan-
guages, improving 15.7% in XNLI accuracy
for Swahili and 11.4% for Urdu over previ-
ous XLM models. We also present a detailed
empirical analysis of the key factors that are
required to achieve these gains, including the
trade-offs between (1) positive transfer and ca-
pacity dilution and (2) the performance of high
and low resource languages at scale. Finally,
we show, for the first time, the possibility of
multilingual modeling without sacrificing per-
language performance; XLM-R is very compet-
itive with strong monolingual models on the
GLUE and XNLI benchmarks. We will make
our code, data and models publicly available
https://metatext.io/datasets/cc100-nepali
Common Crawl
A library, museum, or archival institute
United States of America
Alexis Conneau and Guillaume Wenzek
 info@commoncrawl.org
Yes - it has a direct download link or links
https://metatext.io/datasets/cc100-nepali
Yes
https://commoncrawl.org/terms-of-use/
public domain open license research use
other: Other license