Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Adding Recurrence to Pretrained Transformers for Improved Efficiency and
  Context Size   Fine-tuning a pretrained transformer for a downstream task has become a
standard method in NLP in the last few years. While the results from these
models are impressive, applying them can be extremely computationally
expensive, as is pretraining new models with the latest architectures. We
present a novel method for applying pretrained transformer language models
which lowers their memory requirement both at training and inference time. An
additional benefit is that our method removes the fixed context size constraint
that most transformer models have, allowing for more flexible use. When applied
to the GPT-2 language model, we find that our method attains better perplexity
than an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora, for a
given amount of computation or memory.

11
The categories of this paper are: hep-lat
12
The categories of this paper are: nucl-th
16
The categories of this paper are: stat
7
The categories of this paper are: hep-th
5
The categories of this paper are: gr-qc
4
The categories of this paper are: cond-mat
15
The categories of this paper are: nucl-ex
21
The categories of this paper are: adap-org
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
23
The categories of this paper are: solv-int
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
28
The categories of this paper are: alg-geom