Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: ALBETO and DistilBETO: Lightweight Spanish Language Models   In recent years there have been considerable advances in pre-trained language
models, where non-English language versions have also been made available. Due
to their increasing use, many lightweight versions of these models (with
reduced parameters) have also been released to speed up training and inference
times. However, versions of these lighter models (e.g., ALBERT, DistilBERT) for
languages other than English are still scarce. In this paper we present ALBETO
and DistilBETO, which are versions of ALBERT and DistilBERT pre-trained
exclusively on Spanish corpora. We train several versions of ALBETO ranging
from 5M to 223M parameters and one of DistilBETO with 67M parameters. We
evaluate our models in the GLUES benchmark that includes various natural
language understanding tasks in Spanish. The results show that our lightweight
models achieve competitive results to those of BETO (Spanish-BERT) despite
having fewer parameters. More specifically, our larger ALBETO model outperforms
all other models on the MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets.
However, BETO remains unbeaten for POS and NER. As a further contribution, all
models are publicly available to the community for future research.

2
The categories of this paper are: math
17
The categories of this paper are: q-fin
4
The categories of this paper are: cond-mat
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
33
The categories of this paper are: plasm-ph
25
The categories of this paper are: patt-sol
16
The categories of this paper are: stat
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph