Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Can Vision Transformers Perform Convolution?   Several recent studies have demonstrated that attention-based networks, such
as Vision Transformer (ViT), can outperform Convolutional Neural Networks
(CNNs) on several computer vision tasks without using convolutional layers.
This naturally leads to the following questions: Can a self-attention layer of
ViT express any convolution operation? In this work, we prove that a single ViT
layer with image patches as the input can perform any convolution operation
constructively, where the multi-head attention mechanism and the relative
positional encoding play essential roles. We further provide a lower bound on
the number of heads for Vision Transformers to express CNNs. Corresponding with
our analysis, experimental results show that the construction in our proof can
help inject convolutional bias into Transformers and significantly improve the
performance of ViT in low data regimes.

16
The categories of this paper are: stat
15
The categories of this paper are: nucl-ex
30
The categories of this paper are: q-alg
37
The categories of this paper are: supr-con
7
The categories of this paper are: hep-th
20
The categories of this paper are: acc-phys
27
The categories of this paper are: comp-gas
9
The categories of this paper are: nlin
14
The categories of this paper are: math-ph
10
The categories of this paper are: q-bio
2
The categories of this paper are: math
35
The categories of this paper are: mtrl-th
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
26
The categories of this paper are: bayes-an
23
The categories of this paper are: solv-int