Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: LaMemo: Language Modeling with Look-Ahead Memory   Although Transformers with fully connected self-attentions are powerful to
model long-term dependencies, they are struggling to scale to long texts with
thousands of words in language modeling. One of the solutions is to equip the
model with a recurrence memory. However, existing approaches directly reuse
hidden states from the previous segment that encodes contexts in a
uni-directional way. As a result, this prohibits the memory to dynamically
interact with the current context that provides up-to-date information for
token prediction. To remedy this issue, we propose Look-Ahead Memory (LaMemo)
that enhances the recurrence memory by incrementally attending to the
right-side tokens, and interpolating with the old memory states to maintain
long-term information in the history. LaMemo embraces bi-directional attention
and segment recurrence with an additional computation overhead only linearly
proportional to the memory length. Experiments on widely used language modeling
benchmarks demonstrate its superiority over the baselines equipped with
different types of memory.

33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
13
The categories of this paper are: quant-ph
32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
16
The categories of this paper are: stat
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
4
The categories of this paper are: cond-mat
30
The categories of this paper are: q-alg
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph