Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Eigen Analysis of Self-Attention and its Reconstruction from Partial
  Computation   State-of-the-art transformer models use pairwise dot-product based
self-attention, which comes at a computational cost quadratic in the input
sequence length. In this paper, we investigate the global structure of
attention scores computed using this dot product mechanism on a typical
distribution of inputs, and study the principal components of their variation.
Through eigen analysis of full attention score matrices, as well as of their
individual rows, we find that most of the variation among attention scores lie
in a low-dimensional eigenspace. Moreover, we find significant overlap between
these eigenspaces for different layers and even different transformer models.
Based on this, we propose to compute scores only for a partial subset of token
pairs, and use them to estimate scores for the remaining pairs. Beyond
investigating the accuracy of reconstructing attention scores themselves, we
investigate training transformer models that employ these approximations, and
analyze the effect on overall accuracy. Our analysis and the proposed method
provide insights into how to balance the benefits of exact pair-wise attention
and its significant computational expense.

14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
31
The categories of this paper are: ao-sci
28
The categories of this paper are: alg-geom
12
The categories of this paper are: nucl-th
34
The categories of this paper are: chem-ph
27
The categories of this paper are: comp-gas
8
The categories of this paper are: hep-ex
36
The categories of this paper are: cmp-lg
6
The categories of this paper are: astro-ph
30
The categories of this paper are: q-alg
2
The categories of this paper are: math
17
The categories of this paper are: q-fin
13
The categories of this paper are: quant-ph
24
The categories of this paper are: dg-ga