Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Attention is Not Only a Weight: Analyzing Transformers with Vector Norms   Attention is a key component of Transformers, which have recently achieved
considerable success in natural language processing. Hence, attention is being
extensively studied to investigate various linguistic capabilities of
Transformers, focusing on analyzing the parallels between attention weights and
specific linguistic phenomena. This paper shows that attention weights alone
are only one of the two factors that determine the output of attention and
proposes a norm-based analysis that incorporates the second factor, the norm of
the transformed input vectors. The findings of our norm-based analyses of BERT
and a Transformer-based neural machine translation system include the
following: (i) contrary to previous studies, BERT pays poor attention to
special tokens, and (ii) reasonable word alignment can be extracted from
attention mechanisms of Transformer. These findings provide insights into the
inner workings of Transformers.

27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph
9
The categories of this paper are: nlin
34
The categories of this paper are: chem-ph
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
19
The categories of this paper are: eess
28
The categories of this paper are: alg-geom
18
The categories of this paper are: econ
11
The categories of this paper are: hep-lat
0
The categories of this paper are: hep-ph
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph