Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Mathematical Theory of Attention   Attention is a powerful component of modern neural networks across a wide
variety of domains. However, despite its ubiquity in machine learning, there is
a gap in our understanding of attention from a theoretical point of view. We
propose a framework to fill this gap by building a mathematically equivalent
model of attention using measure theory. With this model, we are able to
interpret self-attention as a system of self-interacting particles, we shed
light on self-attention from a maximum entropy perspective, and we show that
attention is actually Lipschitz-continuous (with an appropriate metric) under
suitable assumptions. We then apply these insights to the problem of
mis-specified input data; infinitely-deep, weight-sharing self-attention
networks; and more general Lipschitz estimates for a specific type of attention
studied in concurrent work.

11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
7
The categories of this paper are: hep-th
5
The categories of this paper are: gr-qc
20
The categories of this paper are: acc-phys
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
27
The categories of this paper are: comp-gas
8
The categories of this paper are: hep-ex
36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
6
The categories of this paper are: astro-ph
34
The categories of this paper are: chem-ph
18
The categories of this paper are: econ
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin