Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Finnish Language Modeling with Deep Transformer Models   Transformers have recently taken the center stage in language modeling after
LSTM's were considered the dominant model architecture for a long time. In this
project, we investigate the performance of the Transformer architectures-BERT
and Transformer-XL for the language modeling task. We use a sub-word model
setting with the Finnish language and compare it to the previous State of the
art (SOTA) LSTM model. BERT achieves a pseudo-perplexity score of 14.5, which
is the first such measure achieved as far as we know. Transformer-XL improves
upon the perplexity score to 73.58 which is 27\% better than the LSTM model.

12
The categories of this paper are: nucl-th
2
The categories of this paper are: math
10
The categories of this paper are: q-bio
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
6
The categories of this paper are: astro-ph
17
The categories of this paper are: q-fin
15
The categories of this paper are: nucl-ex
5
The categories of this paper are: gr-qc
34
The categories of this paper are: chem-ph
21
The categories of this paper are: adap-org
28
The categories of this paper are: alg-geom
29
The categories of this paper are: funct-an
37
The categories of this paper are: supr-con
13
The categories of this paper are: quant-ph
32
The categories of this paper are: atom-ph