Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Area Attention   Existing attention mechanisms are trained to attend to individual items in a
collection (the memory) with a predefined, fixed granularity, e.g., a word
token or an image grid. We propose area attention: a way to attend to areas in
the memory, where each area contains a group of items that are structurally
adjacent, e.g., spatially for a 2D memory such as images, or temporally for a
1D memory such as natural language sentences. Importantly, the shape and the
size of an area are dynamically determined via learning, which enables a model
to attend to information with varying granularity. Area attention can easily
work with existing model architectures such as multi-head attention for
simultaneously attending to multiple areas in the memory. We evaluate area
attention on two tasks: neural machine translation (both character and
token-level) and image captioning, and improve upon strong (state-of-the-art)
baselines in all the cases. These improvements are obtainable with a basic form
of area attention that is parameter free.

33
The categories of this paper are: plasm-ph
10
The categories of this paper are: q-bio
15
The categories of this paper are: nucl-ex
28
The categories of this paper are: alg-geom
4
The categories of this paper are: cond-mat
22
The categories of this paper are: chao-dyn
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph
24
The categories of this paper are: dg-ga
7
The categories of this paper are: hep-th
23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
2
The categories of this paper are: math
31
The categories of this paper are: ao-sci