Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Learning the Transformer Kernel   In this work we introduce KERNELIZED TRANSFORMER, a generic, scalable, data
driven framework for learning the kernel function in Transformers. Our
framework approximates the Transformer kernel as a dot product between spectral
feature maps and learns the kernel by learning the spectral distribution. This
not only helps in learning a generic kernel end-to-end, but also reduces the
time and space complexity of Transformers from quadratic to linear. We show
that KERNELIZED TRANSFORMERS achieve performance comparable to existing
efficient Transformer architectures, both in terms of accuracy as well as
computational efficiency. Our study also demonstrates that the choice of the
kernel has a substantial impact on performance, and kernel learning variants
are competitive alternatives to fixed kernel Transformers, both in long as well
as short sequence tasks.

25
The categories of this paper are: patt-sol
23
The categories of this paper are: solv-int
36
The categories of this paper are: cmp-lg
12
The categories of this paper are: nucl-th
8
The categories of this paper are: hep-ex
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math
20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
16
The categories of this paper are: stat
10
The categories of this paper are: q-bio