Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: When Attention Meets Fast Recurrence: Training Language Models with
  Reduced Compute   Large language models have become increasingly difficult to train because of
the growing computation time and cost. In this work, we present SRU++, a
highly-efficient architecture that combines fast recurrence and attention for
sequence modeling. SRU++ exhibits strong modeling capacity and training
efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and
Billion Word datasets, our model obtains better bits-per-character and
perplexity while using 3x-10x less training cost compared to top-performing
Transformer models. For instance, our model achieves a state-of-the-art result
on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We
further demonstrate that SRU++ requires minimal attention for near
state-of-the-art performance. Our results suggest jointly leveraging fast
recurrence with little attention as a promising direction for accelerating
model training and inference.

10
The categories of this paper are: q-bio
12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci
26
The categories of this paper are: bayes-an
28
The categories of this paper are: alg-geom
35
The categories of this paper are: mtrl-th
11
The categories of this paper are: hep-lat
18
The categories of this paper are: econ
3
The categories of this paper are: physics
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin
23
The categories of this paper are: solv-int
37
The categories of this paper are: supr-con
19
The categories of this paper are: eess